I am trying to setup a site-to-site VPN to a large telco.
We are running VyOS 1.1.7 in AWS; they are using a Cisco ASA 5520.
Since the telco is large, and we are small, they have dictated all the required settings to us, and are unlikely to change anything on our behalf. Additionally, getting log files or more configuration information from them can be tedious.
From their perspective, they want to whitelist a single static IP of ours for all application traffic (http). So any connections we initiate to them must appear as if it came from our static IP, no matter what the rest of our internal networking architecture may look like. So we use NAT to satisfy this requirement.
Anyway, we can successfully establish both an IKEv1 connection as well as an ESP connection with them, and both ICMP and TCP traffic can flow in both directions. We can ping them, they can ping us, and inbound http traffic to our application server also works.
To me, this means our AWS security policy is configured properly, our firewall rules are sane, our VPN settings match up, and routing our subnets is working too.
The problem is that after a short amount of time, say ~10 minutes, we are no longer reachable from the telco's perspective.
On our side, the output from show vpn ipsec sa indicates that the tunnels are all still up:
vyos@VPN-FW01:~$ show vpn ipsec sa
Peer ID / IP Local ID / IP
------------ -------------
<TELCO IP> <MY IP>
Tunnel State Bytes Out/In Encrypt Hash NAT-T A-Time L-Time Proto
------ ----- ------------- ------- ---- ----- ------ ------ -----
1 up 0.0/0.0 aes128 sha1 no 2153 28800 all
2 up 0.0/0.0 aes128 sha1 no 1964 28800 all
3 up 0.0/0.0 aes128 sha1 no 1906 28800 all
4 up 0.0/0.0 aes128 sha1 no 1864 28800 all
But as you can see, no traffic has passed on the tunnels.
There doesn't seem to be anything informative in the log files either. The output of show log has many entries that look like this:
Jun 17 11:46:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #347: sent QI2, IPsec SA established {ESP=>0x1e15be1f <0xc9355ae4}
Jun 17 11:54:26 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #338 in 10 seconds
Jun 17 11:54:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-2" #348: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #338 {using isakmp#14}
Jun 17 11:54:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-2" #348: sent QI2, IPsec SA established {ESP=>0xa48e62c1 <0xc67eaa07}
Jun 17 12:02:26 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #332 in 10 seconds
Jun 17 12:02:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-3" #349: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #332 {using isakmp#14}
Jun 17 12:02:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-3" #349: sent QI2, IPsec SA established {ESP=>0xe0d44968 <0xccc1945f}
Jun 17 12:03:56 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #333 in 10 seconds
Jun 17 12:04:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-4" #350: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #333 {using isakmp#14}
Jun 17 12:04:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-4" #350: sent QI2, IPsec SA established {ESP=>0xad009d57 <0xc8b2287d}
But no other errors or anything.
Using tcpdump on our end doesn't show anything informative either. Many entries that look like the below, along with typical ARP traffic, NTP, etc.
TELCO-IP.isakmp > ip-MY-IP.ec2.internal.isakmp: isakmp 1.0 msgid dd22ed6d: phase 2/others ? inf[E]: [encrypted hash]
12:06:59.148180 IP (tos 0x0, ttl 64, id 2672, offset 0, flags [DF], proto UDP (17), length 120)
ip-MY-IP.ec2.internal.isakmp > TELCO-IP.isakmp: isakmp 1.0 msgid f8f1d9ba: phase 2/others ? inf[E]: [encrypted hash]
12:07:19.147638 IP (tos 0x0, ttl 234, id 31559, offset 0, flags [none], proto UDP (17), length 120)
But we don't see the incoming ping nor http traffic anymore.
The interesting bit is that if we ping the telco's subnet from our side, then incoming traffic, including http, works again for another ~10 minutes, before dropping.
Any clues?
My VyOS config is here:
firewall {
all-ping enable
broadcast-ping disable
config-trap disable
group {
address-group TELCO-HOSTS {
address 192.xx.yy.38
address 192.xx.yy.39
address 192.xx.yy.40
address 192.xx.yy.41
}
}
ipv6-receive-redirects disable
ipv6-src-route disable
ip-src-route disable
log-martians enable
name eth0in {
default-action reject
rule 20 {
action accept
description "accept ICMP pings"
icmp {
type-name echo-request
}
protocol icmp
}
rule 30 {
action accept
destination {
port 22
}
protocol tcp
}
rule 40 {
action accept
description "accept all internal traffic"
source {
address 10.113.0.0/16
}
}
rule 50 {
action accept
description "accept expected tunneled TCP traffic from TELCO"
destination {
port 5101,8310,8443,8080,9101,9107,9109
}
protocol tcp
source {
group {
address-group TELCO-HOSTS
}
}
}
rule 200 {
action drop
}
}
name eth0out {
default-action accept
}
receive-redirects disable
send-redirects enable
source-validation disable
state-policy {
established {
action accept
}
invalid {
action drop
}
related {
action accept
}
}
syn-cookies enable
twa-hazards-protection disable
}
interfaces {
ethernet eth0 {
address dhcp
duplex auto
firewall {
in {
name eth0in
}
out {
name eth0out
}
}
hw-id 0a:d2:b0:8e:53:f3
smp_affinity auto
speed auto
}
loopback lo {
}
}
nat {
source {
rule 10 {
description "US to TELCO"
destination {
address 192.xx.yy.0/24
}
outbound-interface eth0
translation {
address <MY-APP-SERVER>
}
}
rule 500 {
description "US to anywhere else"
outbound-interface eth0
source {
address 10.113.0.0/16
}
translation {
address masquerade
}
}
}
}
service {
ssh {
disable-password-authentication
port 22
}
}
vpn {
ipsec {
esp-group ESP {
compression disable
lifetime 28800
mode tunnel
pfs enable
proposal 1 {
encryption aes128
hash sha1
}
}
ike-group IKE {
key-exchange ikev1
lifetime 86400
proposal 1 {
dh-group 5
encryption aes256
hash sha1
}
}
ipsec-interfaces {
interface eth0
}
site-to-site {
peer <TELCO-STATIC-IP> {
authentication {
mode pre-shared-secret
pre-shared-secret ****************
}
connection-type initiate
ike-group IKE
local-address <MY-IP>
tunnel 1 {
allow-nat-networks disable
allow-public-networks disable
esp-group ESP
local {
prefix 10.113.0.0/24
}
remote {
prefix 192.xx.yy.38/32
}
}
tunnel 2 {
allow-nat-networks disable
allow-public-networks disable
esp-group ESP
local {
prefix 10.113.0.0/24
}
remote {
prefix 192.xx.yy.39/32
}
}
tunnel 3 {
allow-nat-networks disable
allow-public-networks disable
esp-group ESP
local {
prefix 10.113.0.0/24
}
remote {
prefix 192.xx.yy.40/32
}
}
tunnel 4 {
allow-nat-networks disable
allow-public-networks disable
esp-group ESP
local {
prefix 10.113.0.0/24
}
remote {
prefix 192.xx.yy.41/32
}
}
}
}
}
}
