Some VPNs have 1-5% packet loss

Question

I am currently in this situation and I'll try to be as specific as I can. My firewall has 20 VPNs (over the internet) with different vendors. Everything was fine til a month ago.

I usually use ping to monitor the remote networks and some of my VPNs have 1-5% packet loss. The VPN does not go down, but the ping does. The firewall log shows nothing wrong, not any blocked packets everything seems all right. Overall, it is a 0% packet loss / day, but in some intervals it goes to 5% (at about 1pm, where traffic peaks) Some other VPNs are fine with 100% reply.

What I noticed:

-Most loss occours when we're at peak traffic (again, only for the "damaged" VPNs. The other VPNs are fine)

-From my WAN i have 0% loss with 8.8.8.8 but there's 1-5% packet loss with the remote WANs in question.

-Restarting the firewall stabilizes the situation for about a day, but then everything starts over again.

-The VPN tunnel is UP all the time.

any help from your experience would be truly appreciated

Answer 1

same boat here

what we have determined is carriers are using adavnced boxes (like giant bluecoats) to rate limit udp and esp traffic.

try getting one of these on cisco vpn ezvpn if they are cisco asa 5505's and use ipsec/tcp to the main site with network extension mode. this "chap" has a nice quick write up for you to follow

http://www.jump.net.uk/blog-cisco-easy-vpn-on-asa

the loss stopped when we did ipsec/tcp

but smokeping had 5% loss for weeks.

nasty times, huh?

Answer 2

Nearly every piece of hardware will drop ICMP (ping) traffic in favour of TCP, UDP, and other "real" traffic. Seeing 1-5% packet loss during peak utilization is not remarkable. I would expect, however, to see similar loss across all tunnels terminating at the same firewall were this the case, but 1-5% is still pretty much nothing.

One way for a provider to limp older hardware along is to traffic shape or rate-limit ICMP, which means it gets dropped even more aggressively. I would also expect similar loss on all tunnels going through the same provider hardware, so you may want to Traceroute/MTR (there's a WinMTR for Windows machines) these tunnel endpoints to see if there are any similarities in routes. But, again, 1-5% is usually nothing.

You may want to look into TCPing, which simulates ICMP but using TCP connections. If you're getting 1-5% packet loss with TCP traffic too, then that's more remarkable (but still not terrible).

Check CPU utilization on the firewalls on both sides, memory, interface stats like errors, discards, overruns, etc.

Answer 3

If it was me I wouldn't measure up time of a connection using ICMP. During these peak times are the sites that are showing this 1-5% ping packet loss showing any other signs of link degradation? Like an increase in TCP round-trip time or dropped sessions/services?

I work as part of a team that does network analysis of situations such as these. What I would do is put detailed network traffic monitoring on the main link (behind your firewall) to capture traffic patterns so you can do full analysis of the data and see how things like round trip times are affected by peak data usage. And as another user suggested put some traffic shaping policies in place to rate limit some of the data streams to see if you can improve performance. There are a number of free and paid solutions to do this.