7

I am looking for a technology to achieve TCP connection fault tolerance with the help of two links between hosts and without time delays for route failure detection. Something like this:

                       link1   packet1copy1->
                     --------------------------
      packet1->     /                          \    packet1copy1/packet1copy2->
host1--------router1                            router2 ------------------------host2
                    \  link2   packet1copy2->  /
                     --------------------------

host1 and host2 are connected via router1 and router2 with two links between them. Each router duplicates every packet coming from hosts before forwarding them into both links simultaneously. Then either the peer router or the destination host IP stack take care of redundant packet elimination.

Edit: This is in fact a search for a general purpose fault-tolerance-by-replication solution for TCP (IP) transport. The solution should be of no-need-to-recover type as opposed to reasonably-fast-to-recover approaches like BGP / OSPF / Cisco IP SLA, etc. Some proprietary packet redundancy solutions are known already, though insufficiently universal. In particular, Engage Communication offers IP Tube Protector for VoIP. Unfortunately this solution 1) is more of equipment than of standard technology and 2) is confined to VoIP domain only. It may be also worth noting Juniper Packet Redundancy technology, though it looks like limited to single link only, and not to redundant links.

I wonder why I can't find anything similar from Cisco... Does any standard or at least general purpose technology address this?

1
0

With Mikrotik routers, you can using bonding in broadcast mode, see bonding. I made some tests accross a 4G link connection, it does reduce packet loss going from 1 to 2 and I benefit from TCP speed improvements. Packet losses are not completely eliminated but going to 3 links does not further improve. I would investigate next in network coded TCP.

2
  • Product or resource recommendations are explicitly off-topic here, as are consumer-grade devices, e.g. MikroTik. – Ron Maupin Feb 26 '17 at 17:38
  • @Netflow Thank you for noting bonding in broadcast mode, regardless of Mikrotik :) Not sure if I'll be able to give it a try in some near future, but still it's good to know there seems to be a standards-based approach... – Sergey Ushakov Feb 27 '17 at 1:39
10

I am looking for a technology to achieve TCP connection fault tolerance with the help of two links between hosts and without time delays for route failure detection. Something like this:

                       link1   packet1copy1->
                     --------------------------
      packet1->     /                          \    packet1copy1/packet1copy2->
host1--------router1                            router2 ------------------------host2
                    \  link2   packet1copy2->  /
                     --------------------------

There are a few things working against your proposal...

  1. You are going to make host1 and host2 work very hard to untangle your intentional packet duplication scheme for no good reason
  2. You are burning horsepower on your ipsec encryption points for no good reason
  3. TCP has been refined for over three decades to automatically recover from infrastructure flaws and failures; "helping" TCP in such a way fixes the wrong problem. You need to make your infrastructure react to mitigate problems, you should not duct-tape TCP to survive problematic infrastructure.

I am going to answer with the the same comment I made, since your failure detection requirements are twenty seconds...

Build 2 IPSec tunnels with ISP diversity as required. Run a routing protocol through your IPSec tunnels, and tune the protocol timers to fail around sustained infrastructure packet loss. If you have Cisco end to end, EIGRP has long had very fast convergence around failures, although Link state protocols are getting the same these days with the IETF loop free alternative implementations.

Optionally use IP SLA on both sides to take down a tunnel that does not meet any jitter / delay / packet loss requirements.

11
  • Mike, with all due respect I cant accept your criticism for the following reasons: 1) my question seeks a fault tolerance by replication type of solution, while your solutions are of fault tolerance by redundancy type; both approaches are normally deemed valid, yet they tend to yield different quality of service levels, and I seek for better level of service; 2) fault tolerance by replication tends to be more expensive, but I would not take the word "expensive" too serious here :) that said, please accept my "thank you" and upvote for good overview, yet I refrain from accepting your answer – Sergey Ushakov Jul 23 '13 at 5:02
  • 2
    @s-n-ushakov, like I said... if you want fault tolerance by replication, you are using the wrong protocol. TCP was made for fault tolerance by redundancy. If you want fault tolerance by replication, may I introduce you to our friend known as UDP. UDP is much better-suited for what you want; however, that means you're about to rewrite your primary business application just because you are in love with a strange network design (with no known hardware to implement this bidirectional packet replication, I might add) – Mike Pennington Jul 23 '13 at 11:06
  • well, sometimes the application-level protocol is not our choice... and knowledge of your peer infrastructure may be limited in business world... and it might be cool to have, say, HTTP designed and implemented over UDP :) and speaking seriously, thank you for pointing to link state protocols, they may be a relief, though not the final solution; BTW TCP itself has already provision at least for a part of solution being sought: The TCP must recover from data that is ... duplicated ... -- RFC 793, section 1.5, subsection "Reliability" – Sergey Ushakov Jul 24 '13 at 1:37
  • 6
    Feel free to quote RFC 793, Section 1.5... in response, I will quote RFC 1925, Section (3): With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. – Mike Pennington Jul 24 '13 at 2:05
  • 2
    Engage Communications is selling a TDM over IP solution. You are asking for a TCP over IP solution... you could overlay IP over TDM over IP, but again... this is really crazy. You should hire a real network engineer – Mike Pennington Jul 26 '13 at 12:24
4

OK, from the top;

  • Down vote on your question from me; your question isn't clear enough based on your responses in comments to other peoples answer. You have assumed the solution is networking engineering related but you don't seem to know, and give the impression that you hope someone is going to give you the answer you need.

  • You have the following problem requirement;

host1 and host2 are connected via router1 and router2 with two links between them. Each router duplicates every packet coming from hosts before forwarding them into both links simultaneously. Then either the peer router or the destination host IP stack take care of redundant packet elimination.

  • Unless your end host's connection to their local router is double the speed of the traffic going over a single link between router1 and router2, which you haven't mentioned, your hosts will need two connections to their local router. There is NO native software or product ANYWHERE that can run on an end hosts and take two TCP streams down the same NIC or two separate ones and pull from an alternate stream missing packets from the first stream. How do I know this? Because that is not how networking works, IP & TCP simply weren't designed to work like that. There maybe products for duplicating packets, but these are a niche, not wide spread, becasue it's the wrong answer to the question.

Why is this such a bonkers request;

  • You seem to be trying to put a round peg into a square hole. My understanding of your problem requirement is that you want redundancy for your application's data travelling between to remote hosts. Data is sent twice end to end in case of a link failure. That is all you are protecting against here with dual TCP flows though, physical layer 1 failure. If there is a pause in sending a packet from one host to the other, it will be late arriving down both router-to-router links. If a transient problem occurs on one link but not the other, such as congestion, the router at the end of the link, would need to track both TCP streams simultaneously to see that when a packet arrives on link2 with the proceeding sequence number in its header, and nothing has arrived on link1, then the packet on link1 is late, and if it does turn up, it needs to drop it.

    What if you find your self in a situation where there is congestion on link1 but no traffic is dropped, due to a good QoS schema, but it is queues, packets down link1 are now always behind link2. What if link2 fails now and the router passes packets on link1 to the end hosts, it's going to receive dup packets, and stop and retransmit etc, and cause a delay. Nothing was achieved here.

Moving on to a solution;

  • A better idea in my opinion would be to have dual layer 2 links between the two end hosts, extending their broadcast domains to include each others NIC. You can do this via direct layer 2 interconnects, MPLS/VPLS extension, carrier layer 2 service, take you pick, that isn't strictly relevant here. Extending the layer 2 network between hosts means you don't need to mess with TCP or do any crazy black magic or band-aid type fixes. TCP will be completely agnostic of the underlying technology and you will still have you layer 1 / physical link redundancy.

  • If you use an MPLS based solution you can use features like traffic engineering (MPLS-TE) to monitor the latency across the links and always use the link with the lowest latency. You can use BFD with MPLS FRR, which can get you 50ms~ fail over time between links. I know you said you don't want a redundancy fail over solution, but 50ms is pretty fast in my opinion. If your application can't handle a 50ms loss of connectivity, then you need to go back to the application drawing board. No system is up 100% of the time, you must plan for failures, planned maintenance, and outages through malicious intent / security related; to all occur at some point. You must be realistic.

In one comment you said the following;

well, IP SLA is the technology being used at least at one end so far... :) still it takes quite a time for both ends to detect link failure, and the application gets out of sync sometimes... and the links may be twinkling sometimes... that's why we are looking for something delay-free

No such thing; Time must pass for possible events to become actualities. You need to re-think this to an "acceptable" level of delay.

Also in another comment you said;

BGP it takes quite a time to discover that the route deemed operational is now down; finally the routers realize this and switch active routes, but it takes time, and application-level protocol may suffer

BGP has a hello timer, this is detecting the the presence of it's immediate neighbour. Default is 30 seconds, I suspect this is what you are referring too. If both routers in your topology speak BGP with the ISP at each site or even directly to each other, over those peerings build IP-in-IP tunnels of GRE or L2TP(v3) tunnels between the two routers, over those tunnels run BFD or IP SLA. Now you can detect end-to-end connectivity loss in 1 or 2 seconds, and reroute to the other tunnel using tacking objects.

All in all, you seem to be mixing up different layers of technology. BGP isn't suppose to provide fast re-routing, TCP isn't supposed to be duplicated, and so on. You're looking at the wrong levels of abstraction to tackle this problem. I hope this has helped.

11
  • 2
    He doesn't need them, he can run MPLS over GRE for example, MPLS over IPSEC. He could invest in L2 links possibly? Who knows or cares what his budget is, not me; I'm not saying my ideas are the best, I'm simply trying to provide solutions to the problem that are sane and reliable, irrelevant of cost or availability, and explain further the issues he faces and reasons for making one choice over another. Its a purely technical answer. – jwbensley Jul 26 '13 at 14:50
  • 1
    @s-n-ushakov There is no such thing as zero-time – jwbensley Jul 29 '13 at 9:09
  • 1
    It doesn't say in that document, to repeat my self though Time must pass for possible events to become actualities - There is no such thing as zero time. The box has to check for losses, delays, drops etc, that takes time, it may be mili or micro seconds, but it takes some period of time. Just like BFD for example, if you set the hello time to 50ms, with a default hold time of 3x hello, you have to wait 150ms for fail-over to occur. Now please stop comparing a TDM backup solution to your scenario. By it's very nature it is possible to office a TDM service like the TCP redundancy you require – jwbensley Jul 29 '13 at 9:32
  • 1
    ...because you know when a TDM packet should exactly arrive. If you don't full understand how E1s/T1s work, I suggest you read about that first. Then you will understand one reason for having TDM links is for reliability such as guaranteed latency. They run at a fixed speed and frame rate per second. IP/TCP is all over the scale. TDM is much more predictable and this runs at a lower layer than TCP, it would be like duplicating Ethernet frames over two links. The fact that these boxes are running TDM over IP adds in some potential for change and skewing of the two TDM streams, that is why... – jwbensley Jul 29 '13 at 9:35
  • 1
    ...those boxes have skew timers and out-of-order frame detectors (reading sequence numbers). – jwbensley Jul 29 '13 at 9:35
1

This is an application layer problem and not a network level problem. This is because one of the core principles of IP is prevent duplicates especially when TCP retransmission is invoked.
In highly critical environments, the approach will be to have 2 NICs on the end hosts and get the application to generate 2 unique packets. With this approach you can use existing technologies and network principles using variable paths and metrics.

3
  • sorry, but can't agree that this is an application layer problem; application has a right to just expect a TCP link of sufficient quality; TCP itself has provisions for recovery after minor network failures, and there are numerous solutions providing network fault tolerance by alternate routing; unfortunately all of them I know of are of recover-fast-after-failure kind rather than of no-need-to-recover one; I perceive this task as just a reduntant network engineering one; after all, if we can have a RAID, why can't we have a RAIN? :) – Sergey Ushakov Jul 24 '13 at 4:18
  • Two NICs with two tcp sessions means the OP must decide which TCP session is more reliable. – radio-free-europe Jul 26 '13 at 1:18
  • Just to avoid misunderstanding: I never meant two TCP sessions. TCP session should be one. That's the routers' task to take care of redundancy and zero delay TCP traffic failover. – Sergey Ushakov Jul 29 '13 at 5:23
0

I'm not aware of tricks or protocols that can perform this type of forward replication on the network devices in question - for this type of application I would recommend redundancy and fast failure detection using BGP fast-failover, BFD and other tools. However, I came across this open source project called 'Tunnel Splitter' http://coderrr.wordpress.com/2010/01/10/tunnel-splitter-accelerating-a-single-tcp-connection-over-multiple-isps/ that seems to fit what you're looking for. In short, the TS boxes installed at each site proxy the TCP connections between host1 and host2, and then split the traffic between them over tunnels. As each tunnel has a unique source address, PBR (policy-based routing) can be used at the routers to direct traffic for tunnel1 over link1 and tunnel2 over link2. The TS boxes terminate the tunnels, and have a single tcp connection to host1 and host2. Of course, you would need to really,really test this but it seems to work on the whiteboard!

2
  • sounds promising and fitting the bill (though not industrial-grade), but unfortunately GitHub responds with 404 for this project already... do you know what happened to this project afterwards? – Sergey Ushakov Jul 24 '13 at 2:11
  • unfortunately I do not. You may have to contact the authors directly. – smoothbSE Jul 24 '13 at 5:07
0

I know this is an old thread but there now is a solution for the video over ip world in "ST 2022-7:2019 - SMPTE Standard - Seamless Protection Switching of RTP Datagrams," in ST 2022-7:2019 , vol., no., pp.1-11, 13 May 2019, doi: 10.5594/SMPTE.ST2022-7.2019. https://ieeexplore.ieee.org/document/8716822

It basically sends two copies of every packet over two paths then combines then buffers a certain number ro allow for dropped packets. It will automatically select the packets to deliver from each stream seamlessly.

A standards based solution for tcp/ip using this as a base would be very useful.

2
  • True - for UDP-based RTP that can definitely work. The OP was specifically asking about TCP though where it can't ever work the way he wants. – Zac67 Apr 1 at 15:51
  • Michael McCarthy: Thank you for noting a solution for a related problem, though a bit specific indeed. @Zac67 Fortunately solutions for the original problem do exist, with TCP not excluded :) One is mentioned in the accepted answer, and Linux IP Bonding in broadcast mode is another technology to mention, maybe a less proprietary one... – Sergey Ushakov Apr 3 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.