4

I am currently in this situation and I'll try to be as specific as I can. My firewall has 20 VPNs (over the internet) with different vendors. Everything was fine til a month ago.

I usually use ping to monitor the remote networks and some of my VPNs have 1-5% packet loss. The VPN does not go down, but the ping does. The firewall log shows nothing wrong, not any blocked packets everything seems all right. Overall, it is a 0% packet loss / day, but in some intervals it goes to 5% (at about 1pm, where traffic peaks) Some other VPNs are fine with 100% reply.

What I noticed:

-Most loss occours when we're at peak traffic (again, only for the "damaged" VPNs. The other VPNs are fine)

-From my WAN i have 0% loss with 8.8.8.8 but there's 1-5% packet loss with the remote WANs in question.

-Restarting the firewall stabilizes the situation for about a day, but then everything starts over again.

-The VPN tunnel is UP all the time.

any help from your experience would be truly appreciated

  • What kind of firewall is it? Have you monitored CPU and memory during the outages? – Ron Trunk Oct 7 '14 at 12:10
  • What are you pinging across the VPN? If its a Router/Firewall pinging a Router/Firewall, then its common to see packet loss. This is due to Routers/Firewalls giving more precedence to transit traffic rather than traffic destined to the Router/Firewall itself. So it might handling through traffic with 100% reliability, but might not care as much about responding to ICMP messages directed to its own address. This is why a sequence of Traceroutes to a 'known up' address often results in various routers at various times not sending back a response. – Eddie Jan 3 '15 at 21:36
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer. – Ron Maupin Aug 10 '17 at 16:00
1

same boat here

what we have determined is carriers are using adavnced boxes (like giant bluecoats) to rate limit udp and esp traffic.

try getting one of these on cisco vpn ezvpn if they are cisco asa 5505's and use ipsec/tcp to the main site with network extension mode. this "chap" has a nice quick write up for you to follow

http://www.jump.net.uk/blog-cisco-easy-vpn-on-asa

the loss stopped when we did ipsec/tcp

but smokeping had 5% loss for weeks.

nasty times, huh?

| improve this answer | |
  • 3
    Are you sure the packet loss actually stopped rather than it now being obfuscated by TCP? If you're tunneling ICMP inside TCP, then ICMP won't see the packet loss since TCP will retransmit the dropped packets. You could still be having the same packet loss, but smokeping wouldn't know it since packets are being retransmitted at the VPN/TCP level. – Jeremy Feb 5 '15 at 14:42
0

Nearly every piece of hardware will drop ICMP (ping) traffic in favour of TCP, UDP, and other "real" traffic. Seeing 1-5% packet loss during peak utilization is not remarkable. I would expect, however, to see similar loss across all tunnels terminating at the same firewall were this the case, but 1-5% is still pretty much nothing.

One way for a provider to limp older hardware along is to traffic shape or rate-limit ICMP, which means it gets dropped even more aggressively. I would also expect similar loss on all tunnels going through the same provider hardware, so you may want to Traceroute/MTR (there's a WinMTR for Windows machines) these tunnel endpoints to see if there are any similarities in routes. But, again, 1-5% is usually nothing.

You may want to look into TCPing, which simulates ICMP but using TCP connections. If you're getting 1-5% packet loss with TCP traffic too, then that's more remarkable (but still not terrible).

Check CPU utilization on the firewalls on both sides, memory, interface stats like errors, discards, overruns, etc.

| improve this answer | |
0

If it was me I wouldn't measure up time of a connection using ICMP. During these peak times are the sites that are showing this 1-5% ping packet loss showing any other signs of link degradation? Like an increase in TCP round-trip time or dropped sessions/services?

I work as part of a team that does network analysis of situations such as these. What I would do is put detailed network traffic monitoring on the main link (behind your firewall) to capture traffic patterns so you can do full analysis of the data and see how things like round trip times are affected by peak data usage. And as another user suggested put some traffic shaping policies in place to rate limit some of the data streams to see if you can improve performance. There are a number of free and paid solutions to do this.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.