Valid use cases for packet deduplication? [closed]

Question

(On-the-fly) packet de-duplication is an advertised feature of some devices in which exact copies of packets can be allegedly filtered out at near line-speed. Ignoring the plausibility of this working in the real world (just think of the amounts of data, how many packets or for which can you realistically store and retrieve for comparison?) for now, what would be valid use cases?

I can't think of any use cases apart from

• those arising from a bad (monitoring) network design
• those arising from technical limitations of monitoring features, e.g. one spam port for everything

I wouldn't consider such uses cases valid, so are there any that don't fall into these categories? Or am I right in my assumption that none exist and that in a perfect world this feature would be needed?

Answer 1

Usually this is discussed in the context of network capture appliances. So - for example - if you have taps on four links and a given packet happens to traverse all four then you'll naturally see four copies of the same packet. De-duplication is the capability to deliver only one copy of this packet to a protocol analyzer.

ETA: Some actual use-cases might help to illustrate both the need and the function better. Imagine a classic enterprise network that consists of dozens of switches organized into a series of core, aggregation and end-device access switches. A few of these switches might connect to the firewalls and routers that go to the Internet while others connect to a WAN and others still are for servers and users. If there's a desire to be able to monitor traffic on a wide-scale basis it tends to make sense to deploy some combination of in-line taps and on-switch port mirroring (i.e. SPAN ports).

The result of this (common) monitoring mechanism is that the output of these various taps is brought back to a common point for analysis by sniffers, performance analyzers, security devices, etc. One problem that arises with this is that I'll see the same packet as it traverses each switch. In the case of communications between two servers on two access switches I might see that packet three times - once on its source, once on an intermediate aggregation switch and then again on its destination. At the same time if I want to see traffic from one of those servers to the Internet I could see the same packet on its source, the aggregation, a core switch and then whatever connects to the Internet (four copies of the same packet).

De-duplication (and, yes, it's a real feature that several vendors implement in actual shipping products) makes sure that an otherwise identical packet is only presented once. This is really crucial because the condition of seeing the same packet multiple times can also be indicative of forwarding loops or other network problems. Similarly any kind of performance management solution is going to be gathering inflated data, security devices may see a DoS/traffic flood, etc.

Incidentally - the same issue occurs when gathering Netflow/sFlow data from multiple switches, but it's arguably somewhat trickier to implement de-dupe in that situation as there's a need to interpolate information that's not exactly in sync (vs looking at packets that should hash equivalently).

Either way the root cause is the same - taking a bunch of isolate point-in-time data and aggregating it on a device that has no way of knowing that it's receiving the same information but processed serially. I suppose one could set up three or four parallel packet captures on an as-needed basis (..and varying based on which switches in the path had the needed information) but it ends up being a lot more convenient to be able to look at the traffic from a bunch of different places at once and have it pre-processed in such a way that you can immediately look at what's interesting rather than spending hours (or likely days) determining where to look and how to pull apart the information.

Answer 2

It's impossible to tell for sure, since you have chosen not to disclose the context of your question. But assuming you are talking about packet capture or flow monitoring devices, the issue is that, in a large network, you will be capturing or sampling at several points in your network -- there isn't one main aggregation point.

Consider a large network with several exit points (Internet, extranets, etc), as well as two or more data centers. Where are you going to capture data? You can try to capture everything in the core, but that has problems:

1. The speeds are usually higher, making equipment substantially more expensive.
2. Not all data flows traverse the core.

You can capture at the edges, where speeds are lower, but then you have many capture points (more equipment). Often, you end up with a compromise, with some captures at the core or data center aggregation as well as some edge locations. The result is a high probability that you will capture the same packet more than once-- once when it enters the network, once wen it passes through the core, and perhaps once and again when it reaches its destination. In a large network, it's nearly impossible not to have duplicate packets if you want to have comprehensive monitoring.

Packet de-duplicators are "smart" enough to detect that they have captured multiple copies of the same packet and only send one to whatever monitoring device (IPS, Netflow, FPC, etc) you may have.