2

I am trying to setup a site-to-site VPN to a large telco.

We are running VyOS 1.1.7 in AWS; they are using a Cisco ASA 5520.

Since the telco is large, and we are small, they have dictated all the required settings to us, and are unlikely to change anything on our behalf. Additionally, getting log files or more configuration information from them can be tedious.

From their perspective, they want to whitelist a single static IP of ours for all application traffic (http). So any connections we initiate to them must appear as if it came from our static IP, no matter what the rest of our internal networking architecture may look like. So we use NAT to satisfy this requirement.

Anyway, we can successfully establish both an IKEv1 connection as well as an ESP connection with them, and both ICMP and TCP traffic can flow in both directions. We can ping them, they can ping us, and inbound http traffic to our application server also works.

To me, this means our AWS security policy is configured properly, our firewall rules are sane, our VPN settings match up, and routing our subnets is working too.

The problem is that after a short amount of time, say ~10 minutes, we are no longer reachable from the telco's perspective.

On our side, the output from show vpn ipsec sa indicates that the tunnels are all still up:

vyos@VPN-FW01:~$ show vpn ipsec sa
Peer ID / IP                        Local ID / IP               
------------                        -------------
<TELCO IP>                          <MY IP>                         

Tunnel  State  Bytes Out/In   Encrypt  Hash    NAT-T  A-Time  L-Time  Proto
------  -----  -------------  -------  ----    -----  ------  ------  -----
1       up     0.0/0.0        aes128   sha1    no     2153    28800   all
2       up     0.0/0.0        aes128   sha1    no     1964    28800   all
3       up     0.0/0.0        aes128   sha1    no     1906    28800   all
4       up     0.0/0.0        aes128   sha1    no     1864    28800   all

But as you can see, no traffic has passed on the tunnels.

There doesn't seem to be anything informative in the log files either. The output of show log has many entries that look like this:

Jun 17 11:46:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #347: sent QI2, IPsec SA established {ESP=>0x1e15be1f <0xc9355ae4}
Jun 17 11:54:26 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #338 in 10 seconds
Jun 17 11:54:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-2" #348: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #338 {using isakmp#14}
Jun 17 11:54:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-2" #348: sent QI2, IPsec SA established {ESP=>0xa48e62c1 <0xc67eaa07}
Jun 17 12:02:26 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #332 in 10 seconds
Jun 17 12:02:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-3" #349: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #332 {using isakmp#14}
Jun 17 12:02:36 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-3" #349: sent QI2, IPsec SA established {ESP=>0xe0d44968 <0xccc1945f}
Jun 17 12:03:56 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-1" #14: received Delete SA payload: replace IPSEC State #333 in 10 seconds
Jun 17 12:04:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-4" #350: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP to replace #333 {using isakmp#14}
Jun 17 12:04:06 VPN-FW01 pluto[18897]: "peer-TELCO-IP-tunnel-4" #350: sent QI2, IPsec SA established {ESP=>0xad009d57 <0xc8b2287d}

But no other errors or anything.

Using tcpdump on our end doesn't show anything informative either. Many entries that look like the below, along with typical ARP traffic, NTP, etc.

    TELCO-IP.isakmp > ip-MY-IP.ec2.internal.isakmp: isakmp 1.0 msgid dd22ed6d: phase 2/others ? inf[E]: [encrypted hash]
12:06:59.148180 IP (tos 0x0, ttl 64, id 2672, offset 0, flags [DF], proto UDP (17), length 120)
    ip-MY-IP.ec2.internal.isakmp > TELCO-IP.isakmp: isakmp 1.0 msgid f8f1d9ba: phase 2/others ? inf[E]: [encrypted hash]
12:07:19.147638 IP (tos 0x0, ttl 234, id 31559, offset 0, flags [none], proto UDP (17), length 120)

But we don't see the incoming ping nor http traffic anymore.

The interesting bit is that if we ping the telco's subnet from our side, then incoming traffic, including http, works again for another ~10 minutes, before dropping.

Any clues?

My VyOS config is here:

firewall {
    all-ping enable
    broadcast-ping disable
    config-trap disable
    group {
        address-group TELCO-HOSTS {
            address 192.xx.yy.38
            address 192.xx.yy.39
            address 192.xx.yy.40
            address 192.xx.yy.41
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name eth0in {
        default-action reject
        rule 20 {
            action accept
            description "accept ICMP pings"
            icmp {
                type-name echo-request
            }
            protocol icmp
        }
        rule 30 {
            action accept
            destination {
                port 22
            }
            protocol tcp
        }
        rule 40 {
            action accept
            description "accept all internal traffic"
            source {
                address 10.113.0.0/16
            }
        }
        rule 50 {
            action accept
            description "accept expected tunneled TCP traffic from TELCO"
            destination {
                port 5101,8310,8443,8080,9101,9107,9109
            }
            protocol tcp
            source {
                group {
                    address-group TELCO-HOSTS
                }
            }
        }
        rule 200 {
            action drop
        }
    }
    name eth0out {
        default-action accept
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    state-policy {
        established {
            action accept
        }
        invalid {
            action drop
        }
        related {
            action accept
        }
    }
    syn-cookies enable
    twa-hazards-protection disable
}
interfaces {
    ethernet eth0 {
        address dhcp
        duplex auto
        firewall {
            in {
                name eth0in
            }
            out {
                name eth0out
            }
        }
        hw-id 0a:d2:b0:8e:53:f3
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
}
nat {
    source {
        rule 10 {
            description "US to TELCO"
            destination {
                address 192.xx.yy.0/24
            }
            outbound-interface eth0
            translation {
                address <MY-APP-SERVER>
            }
        }
        rule 500 {
            description "US to anywhere else"
            outbound-interface eth0
            source {
                address 10.113.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
service {
    ssh {
        disable-password-authentication
        port 22
    }
}    
vpn {
    ipsec {
        esp-group ESP {
            compression disable
            lifetime 28800
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        }
        ike-group IKE {
            key-exchange ikev1
            lifetime 86400
            proposal 1 {
                dh-group 5
                encryption aes256
                hash sha1
            }
        }
        ipsec-interfaces {
            interface eth0
        }
        site-to-site {
            peer <TELCO-STATIC-IP> {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
                connection-type initiate
                ike-group IKE
                local-address <MY-IP>
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group ESP
                    local {
                        prefix 10.113.0.0/24
                    }
                    remote {
                        prefix 192.xx.yy.38/32
                    }
                }
                tunnel 2 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group ESP
                    local {
                        prefix 10.113.0.0/24
                    }
                    remote {
                        prefix 192.xx.yy.39/32
                    }
                }
                tunnel 3 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group ESP
                    local {
                        prefix 10.113.0.0/24
                    }
                    remote {
                        prefix 192.xx.yy.40/32
                    }
                }
                tunnel 4 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group ESP
                    local {
                        prefix 10.113.0.0/24
                    }
                    remote {
                        prefix 192.xx.yy.41/32
                    }
                }
            }
        }
    }
}
0

Following up to myself, we discovered that the AWS security policy had an effect here.

When we were seeing the weird "disconnect after 10 minutes" issue, the security policy was set to only allow inbound UDP on port 500 (whitelisted to the peer).

We changed the policy to allow all inbound traffic (on all ports) from the peer and the problem seems to have gone away.

  • You should accept your answer so that the question doesn't keep popping up, looking for an answer. – Ron Maupin Jun 21 '16 at 17:59
0

As VyOS is inside aws it will always have nat from your device to internet. Then you must allow udp port 4500 because all IPsec connection will happen on udp 4500 when the device is behind a nat. For this connection you need protocols 50, 51 (ah and esp) and udp 500 and 4500. That’s why it worked when you opened everything. How is it going? The VyOS is stable ?

  • ESP/AH is not required for IPSEC NAT-T to work, all is encapsulated into UDP. – Andrey Prokhorov Jan 14 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.