4

The context: an Internet facing web site, streaming near real-time market prices. Paying customers can be anywhere on the Internet.

The issue: Customers can experience total packet loss for 10s, 30s and more. This brings the TCP connection delivering the prices to a halt, thus rendering the website useless. All connectivity from the data center to the peering points have been sized correctly, and have capacity to spare.

It has been asserted that this is commonplace in public Internet. My gut however says no, and all of my experience as a consumer is to the contrary, however I have no hard facts to draw upon.

Presuming that packet loss is caused by congestion, when a route becomes congested surely the router holding that route will choose an alternate, and furthermore will do so more rapidly that 10s?

Finally, is there any feedback available when a route congests?

M.

  • 2
    It's true that there can and will be packet loss on the internet. TCP can cope with that. This being said, a total loss of connectivity for 10-30s is not something that is normal. Can you provide a traceroute/mtr when the problem is occuring? This might show where the problem is. – Sebastian Wiesinger Jul 31 '14 at 10:58
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer. – Ron Maupin Aug 8 '17 at 21:35
9

The issue: Customers can experience total packet loss for 10s, 30s and more. This brings the TCP connection delivering the prices to a halt, thus rendering the website useless. All connectivity from the data center to the peering points have been sized correctly, and have capacity to spare.

It has been asserted that this is commonplace in public Internet. My gut however says no, and all of my experience as a consumer is to the contrary, however I have no hard facts to draw upon.

Your gut instinct is correct. Speaking as a guy who has built networks for the last 20 years, you should never blindly accept significant ongoing packet loss on a wired network, and do not let any equipment vendor or ISP tell you otherwise. Regular 10 to 30 second outages are unacceptable; if you have a choice, perhaps you could let your upstream know that you're going to start considering your options unless they take the problem seriously. The sad reality is that bandwidth isn't free, ISPs profit margins are usually quite low, and some providers will do everything they can to save money even if it means unhappy customers (crazy as that may sound).

Assuming I'm sending basic TCP traffic, the most packet loss I will accept for a public internet service (where you expect some loss from occasional over-subscription) is 0.10% over a 24 hour period, and no more than 0.50% over any 5 minute period Note 1.

I tolerate 0.0% packet loss inside our wired corporate network Note 2. As long as the packet drops are not expected, corporate network engineers should upgrade their circuits (or LAN) if they start seeing packet loss.

Presuming that packet loss is caused by congestion, when a route becomes congested surely the router holding that route will choose an alternate, and furthermore will do so more rapidly that 10s?

By default, routing protocols like OSPF or BGP won't re-route based on congestion unless the congestion makes them drop their routing protocol packets long enough to declare the neighbor down (on the order of 40 seconds to 120 seconds with default timers); the tricky issue here is that routing protocol packets are prioritized higher than other traffic in many vendors equipment. That means the circuit has to be really congested for that whole interval before the routing protocols will drop.

Finally, is there any feedback available when a route congests?

There are technical tools for managing which upstream provider you choose for a given route (assuming you have multiple upstreams); consider Cisco's PfR as just one possible way to work around this problem.


Note 1: While I consider 0.5% packet loss over 5 minutes enough to notice, I wouldn't make a big deal out of it... this threshold is just a number where you can concede that there might start to be a noticable TCP performance problem; 0.5% loss probably isn't going to get your attention. However, if the packet loss is both regular and bad enough to notice (such as your case), insist that your provider fix the problem (as long as the loss is in their network).

Note 2: In my experience, some switches simply cannot operate with zero packet drops when multiple users are on this switch. At this point you have choices: A) Forklift all the existing switches (if the packet drop problem is big enough), B) Wait for a patch from the vendor to fix the issue, or C) Accept the packet drops as a "feature" of the switch. By way of example, I have faced this choice with the Cisco 3850's shared-buffer architecture.

5

Total Packet loss for 10-30 seconds is surely caused by other factors than lost packets somewhere on Internet. A lost tcp packet would normally generate a retransmit within ~10-100ms. Congested links would not block all traffic for such a long time period.

A congested link will normally not make a router choose an alternate route unless link state is lost, or the router looses connectivity to adjacent routers on the other end of the link, forcing the router to rebuild its routing table.

Does this happen to all your customers simultaneously? If so, you might have other types of routing issues. I have experienced ospf issues where ip-routes are "hijacked" by mis-configured equipment, disrupting traffic flow in a similar way.

I do not think you as an "end user" would receive any form of feedback when a router link congests. You would have to debug the packet flow and interpret tcp retransmits, changes in tcp window sizes etc. However, routers and network protocols have numerous methods for handling and mitigating network congestions. For details on this: http://en.wikipedia.org/wiki/Network_congestion

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.