2

Consider an application (mostly TCP/UDP, but not strictly so) where the packet sizes vary, and may be very long. The use case / problem statement is to identify duplicate frames that are collected at multiple points in the network. (Meaning that if we collect a frame from router X, and a frame from router Y, how can we determine with high probability that these are or are not the same frame)... Further assume that we can create a (good) hash on the first NNN bytes of the frame relatively cheaply. In my mind I'm thinking that the hash could be generated on somewhere in the neighborhood of the first 100-200 bytes. This is certainly enough to get the {L2, IP, TCP|UDP} headers, as well as most common application protocols where those headers are directly after the L4 headers (HTTP, VOIP, streaming video, etc). As long as we can get the headers AND at least a little bit of the payload "data" I can mostly convince myself that this hash will accurately identify uniqueness.

Can anyone think of specific applications where the hash on just the early part of the frame would NOT be a good indicator of uniqueness? Anyone have a compelling argument as to how deep that hash would need to be? (i.e. is 128B sufficient for most cases?)

Improve this question
1
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer.
    – Ron Maupin ♦
    Aug 11, 2017 at 4:20

1 Answer 1

Reset to default
2

Unless you truly capture the frames on the same ethernet segment, the frames will be different after your routers rewrite mac addresses, decrement the IP TTLs, and recalculates checksums. And with VLAN tagging, frame sizes may vary depending on exactly where in the network they are pulled.

Ignoring the effects of NAT/PAT rewrites of IP addresses and ports, or firewalls changing TCP sequence numbers, I suggest the following fields to derive the canonical frames for the largest LAN diameter:

  • 5-tuple (src ip & port, dst ip & port, proto)
  • tcp sequence number
  • tcp acknowledgment number
  • ip total length

Ideally, to ignore duplicate packets from retransmissions, an ip timestamp option would be used and could be added to form the canonical. Other custom header options could be introduced into the stream for true fingerprinting of frames.

Improve this answer
2
  • 1
    Hmm, maybe I didn't ask that clearly. This answer was helpful in general, but I'd already gotten that far. Maybe if I ask another way: are there protocols/applications that folks can think of where if the L3 and L4 port information is identical, and the first XXX bytes past the L3/L4 headers are also identical, but the packets themselves are NOT the same?
    – ljwobker
    Apr 25, 2015 at 6:09
  • NFS over UDP probably meets your requirement.
    – james
    Jan 14, 2019 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.