2

We have two IPsec VPN tunnels (over the public network) to a VPC in AWS. Each tunnel has one BGP session. Our router is a Cisco 3925E.

Under load (not necessarily excessive), the BGP sessions are often flapping (hold time expired). I've already tried to play with the keepalive / hold time parameters but without any success... What's the best approach to solve this issue?

Current BGP config:

router bgp 65000
 neighbor 169.254.26.225 remote-as 10124
 neighbor 169.254.34.93  timers 5 30 30

Logs:

Apr 25 10:30:09.897: %BGP-3-NOTIFICATION: sent to neighbor 169.254.34.93 4/0 (hold time expired) 0 bytes 
Apr 25 10:30:09.899: %BGP-5-NBR_RESET: Neighbor 169.254.34.93 reset (BGP Notification sent)
Apr 25 10:30:09.899: %BGP-5-ADJCHANGE: neighbor 169.254.34.93 Down BGP Notification sent
Apr 25 10:30:09.899: %BGP_SESSION-5-ADJCHANGE: neighbor 169.254.34.93 IPv4 Unicast topology base removed from session  BGP Notification sent
Apr 25 10:30:13.023: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel21, changed state to down
Improve this question
4
  • 1
    It sounds like the load is high enough that keepalives can't get through. You may need more bandwidth. Playing with timers is not the solution.
    – Ron Trunk
    Commented Apr 25, 2017 at 11:37
  • The problem is that we don't control the amount of bandwidth available (public network). Therefore, sometimes the BGP sessions flap at around 80Mbps and sometimes around 800Mbps... While the fluctuation of the bandwidth available is acceptable to us, the BGP session flapping sometimes cause a total loss of connectivity (which is not acceptable).
    – Pierre
    Commented Apr 25, 2017 at 12:01
  • what is yours mtu? have you tried to decrease them?
    – Datagram.Network
    Commented Apr 25, 2017 at 12:02
  • What is the bandwidth of the link connected to your router?
    – Ron Trunk
    Commented Apr 25, 2017 at 12:27

2 Answers 2

Reset to default
2

As mentioned by others in the comments section, you are losing enough BGP keep-alive packets to cause the session to fail. When the session drops then all traffic will likely cease until BGP re-establishes its session (which is a minimum of 60 seconds and usually a fair amount longer).

So - you've got two options: either find a mechanism to prioritize BGP traffic or make BGP less sensitive to packet loss.

The former (prioritization) likely isn't going to be doable given that, as you point out, you're relying on a public network and the other side of the session is outside your control. You can certainly prioritize and protect the packets you actually transmit, of course, but that has no guarantee of being honored a hop beyond the router you manage.

The latter approach (BGP less sensitive) may be a more pragmatic option. In this case I would suggest making your dead timer longer. At the moment you have packets being sent every 5 seconds, with the session timing out if a response isn't received in 30. Try increasing the latter value to 60 or 90 (also consider increasing the third number, as it is the minimum you will allow negotiated and we don't know the peer's info). This gives you a better shot of getting a keep-alive through during a bursty period.

Finding an optimum value for keep-alives is always a compromise between stability and detection speed. I would add, though, that if this is the only path to AWS for you - or if the alternate path is substantially less desirable - that erring on the side of slow failure detection actually makes a lot of sense.

Improve this answer
1

Disable path-mtu-discovery on the neighbors.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.