6

We need a dual gigabit connection to one of our customers and for the time being one of the lines has been brought online. The connection is between our DC and the customer's HQ, it's a L2 connection. The cities are roughly 420km apart.

The initial tests showed a RTT of 26ms. Since this seemed a bit high so we complained to the carrier (T-Systems in Germany). They did some tests and "optimizations" and suddenly the RTT went down to 15ms.

I know they can't break the speed of light, I also know that the speed of light in fiber is quite a bit lower than in a vacuum but still, 15 ms seems a bit high for 420km. I also assume there must be several hops along the way but nevertheless, is this what we can reasonably expect ? Should we insist that they do some more "optimizations" ?

  • I believe latency should really be as low as any provider can make it. In this day and age latency really should be a thing of the past. Your maybe better off having a word with each individual provider to discuss getting a quote and finding out what their latency is – user28373 Aug 3 '16 at 19:52
4

Each piece of equipment you run through will add latency through it. If you leased dark fiber between the two locations you should see something just under 3ms one way. Just under 6ms RTT.

How much latency a device will add can vary wildly depending on how efficient it is at processing the packet, how efficient it is at converting to/from light, if any buffering must be done due to over subscription, etc.

I don't know how your service is delivered but more than likely you're looking at something like this:
CE -> CPE -> PE -> P -> P -> PE -> CPE ->CE

  • CE = Client Edge
  • CPE = Customer Premesis Device
  • PE = Provider Edge
  • P = Provider Router (Similar to a Core device)

If each device adds 1ms of latency you've gone from 6ms RTT on dark fiber to 22ms. Each device will probably handle the packet for <1ms but you can see how this adds up.

More than likely the "optimizations" your SP did was to use MPLS-TE to use a more efficient route. I can't confirm this but there's no special voodoo you can throw at a box to make it all of a sudden decrease latency...at least none that I've found yet...still looking.

To sum up, I don't think 15ms is too outrageous. If you need better numbers from a latency perspective then you might want to consider other options.

Note: These times do not include any optical devices these lines might (more than likely) be going through. Things like multiplexers and amplifiers will add small amounts of latency as well. Over long distances the likelihood of going through a multiplexer increases greatly.

  • I am not sure that this connection went across their MPLS backbone. Of course, I have no way of knowing that. But if it didn't go over MPLS, then there wouldn't be so many L3 hops along the way, would it ? That was the reason for my question, I don't know if 15ms is bad or just normal :) The server guys complain they get low speeds when they transfer lots of small files over the line (windows shares mostly) but considering the latency and the lack of WAN optimization devices, they just have to live with it for the time being because that's normal behaviour. – Stefan Radovanovici May 28 '13 at 16:22
  • 1
    When we turn up a brand new customer there is typically a waiting period while we build out new fiber, but they always connect back to our MPLS network. It's the only way to scale. If this company has plans on being around a while you're running through their MPLS network. How much you're paying for the service might shed some light on it too. If the cost is very large then it might be a leased wavelength. If it's fairly regularly prices fro a 1Gb link then it's more than likely switched. – bigmstone May 28 '13 at 16:24
  • That makes sense. – Stefan Radovanovici May 28 '13 at 16:27
8

Light travels down fiber at roughly 2/3 the speed of light.

Therefore 420km should result in about 4-5 ms round trip.

Also, keep in mind that 420km of aerial distance between two points does not necessarily mean that the fiber distance is 420km, it could be quite different.

Depending on how many hand-offs/hops are on the line I would say that 15 ms round trip is pretty good.

  • Yes, I also came up with 4-5ms. That's why 15ms (3-4 times as much) seemed a bit high. 420km is what google maps shows (relatively straight roads), the aerial distance is shorter. But with the hops, I assumed that 420km sounds about right. – Stefan Radovanovici May 28 '13 at 16:06
  • 1
    That's if there's a direct point to point connection, you might be channeled through multiplexers and regenerators as well. – nos May 28 '13 at 16:12
  • I think 10ms would be the lowest you would see, so 15ms ain't so bad. Don't forget that you might no be on a single VLAN for example that spans a load of switches between point A and point B. Any or all off this L2 connection could be a pseudowire or multiple pseudowires which can add in additional latency. Rather than being directly over L1 all the way there. – jwbensley May 28 '13 at 16:18
  • 2
    Unless you know for a fact that the fiber route is more or less point-to-point then I would hesitate to say Google Maps is a good approximation of fiber mileage. There is good chance that it is twice the point-to-point distance, hence 3-4x the aerial distance delay doesn't sound so bad. Depending on if your carrier is the sole carrier on the circuit and your relationship with them, they may be willing to share a limited detail view of the fiber path. Doesn't hurt to ask. – Mike Marotta May 28 '13 at 16:18
  • @MikeMarotta Good point, I'll make sure they are asked. – Stefan Radovanovici May 28 '13 at 16:31
0

Ultimately it will depend on the carrier, as well as the region. In my experience 15ms for that distance may be a little high, but not really that much. I would expect around 10-15ms, all told.

If I were in your shoes, I'd be interested in hearing about these optimizations - I would ask the carrier what was done to "optimize" this.

  • I am not sure that would bring up any useful information, except some vague explanation. I am pretty sure they won't admit they had some faulty connection somewhere :) – Stefan Radovanovici May 28 '13 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.