Why is Wireshark unhappy with my Netflow packet sequence numbers?

Question

I wrote a Netflow v10 (IPFIX) traffic generator. It spoofs Source IP addresses to pretend to be coming from multiple sites, and has fake users it sends reports about. The idea is very similar to Solarwinds Flow Generator, if anyone has ever come across that tool.

It seems to work pretty good, but when I do a packet capture with Wireshark, Wireshark tells me my sequence numbers aren't right.

As far as I can tell from reading the IPFIX RFC, the sequence numbers start at 0 (in the first data packet), and in subsequent packets are incremented by the number of flows within the previous packet (not counting template or option flows). They are a way for the collector to know if it missed any packets.

So for a given site (exporter IP), if I'm sending 10 data flows per packet, the first packet will have a sequence of 0, the next will have a sequence of 10, then 20, and so on. That is exactly what I'm doing, but Wireshark is unhappy.

Wireshark mentions what it expects the sequence numbers to be, but the numbers it expects don't make any sense. My first packet coming from 80.40.20.41 for example had one Data Template, and 17 data flows. I had a sequence number of 0, and wireshark says it was expecting it to be 71. It appears to be counting flows that came from other source IPs (other exporters), and expecting the sequence numbers to be synchronized between different sites for some reason.

Anyone know what my issue might be?

Here is my packet capture, if anyone has a minute to have a look: https://drive.google.com/file/d/1_Sz-ndnbA8w0FZwBriXykXb-O3hZFoqC/view?usp=sharing

---

Edit:

It appears that if I make each exporter's observation domain unique, Wireshark is happy with my sequence numbers. Why would different exporters need different observation domain IDs? How can that even be synchronized? If 2 different routers are acting as netflow exporters and sending a single netflow collector data, how do the 2 routers know what to use for an observation domain ID so that they don't interfere with each other? Is the observation domain something that an admin has to configure by hand on each exporter to make sure they are unique?

Answer 1

Based on my understanding of the spec, Wireshark is wrong, as it should be tracking the sequence number independently for each unique transport session or SCTP stream.

(Quoted content is from From RFC 7011)

Observation Domain ID

A 32-bit identifier of the Observation Domain that is locally unique to the Exporting Process. The Exporting Process uses the Observation Domain ID to uniquely identify to the Collecting Process the Observation Domain that metered the Flows. It is RECOMMENDED that this identifier also be unique per IPFIX Device. Collecting Processes SHOULD use the Transport Session and the Observation Domain ID field to separate different export streams originating from the same Exporter. The Observation Domain ID SHOULD be 0 when no specific Observation Domain ID is relevant for the entire IPFIX Message, for example, when exporting the Exporting Process Statistics, or in the case of a hierarchy of Collectors when aggregated Data Records are exported.

The observation domain ID is only required to be "locally unique". Ensuring that the ID is globally unique is only a recommendation and is not required, so a collecting process would not be able to rely on this.

Sequence Number

Incremental sequence counter modulo 2^32 of all IPFIX Data Records sent in the current stream from the current Observation Domain by the Exporting Process. Each SCTP Stream counts sequence numbers separately, while all messages in a TCP connection or UDP session are considered to be part of the same stream. This value can be used by the Collecting Process to identify whether any IPFIX Data Records have been missed. Template and Options Template Records do not increase the Sequence Number.

The sequence number is an incremental number which is scoped to the current stream (for UDP or TCP this is the transport session, and for SCTP this is a single stream within the transport session) and the observation domain ID. A collecting process would need to use the transport session parameters and observation domain ID to track each independent sequence number counter.

Transport Session

In the Stream Control Transmission Protocol (SCTP), the Transport Session is known as the SCTP association, which is uniquely identified by the SCTP endpoints [RFC4960]; in TCP, the Transport Session is known as the TCP connection, which is uniquely identified by the combination of IP addresses and TCP ports used. In UDP, the Transport Session is known as the UDP session, which is uniquely identified by the combination of IP addresses and UDP ports used.

A transport session is identified by the session's connection parameters. For example, for UDP, these are the source and destination addresses and source and destination ports.

Answer 2

Why would different exporters need different observation domain IDs? How can that even be synchronized? If 2 different routers are acting as netflow exporters and sending a single netflow collector data, how do the 2 routers know what to use for an observation domain ID so that they don't interfere with each other? Is the observation domain something that an admin has to configure by hand on each exporter to make sure they are unique?

You need to pay attention to what the RFC says. For example, what do you think an Observation Domain is? The RFC explains it:

Observation Domain

An Observation Domain is the largest set of Observation Points for which Flow information can be aggregated by a Metering Process. For example, a router line card may be an Observation Domain if it is composed of several interfaces, each of which is an Observation Point. In the IPFIX Message it generates, the Observation Domain includes its Observation Domain ID, which is unique per Exporting Process. That way, the Collecting Process can identify the specific Observation Domain from the Exporter that sends the IPFIX Messages. Every Observation Point is associated with an Observation Domain. It is RECOMMENDED that Observation Domain IDs also be unique per IPFIX Device.

If you do not make each device have a unique Observation Domain ID, then you run into the problem of how to coordinate the sequence numbers:

Sequence Number

Incremental sequence counter modulo 2^32 of all IPFIX Data Records sent in the current stream from the current Observation Domain by the Exporting Process. Each SCTP Stream counts sequence numbers separately, while all messages in a TCP connection or UDP session are considered to be part of the same stream. This value can be used by the Collecting Process to identify whether any IPFIX Data Records have been missed. Template and Options Template Records do not increase the Sequence Number.

A single device can actually have multiple Observation Domain IDs, but trying to have the same Observation Domain ID across multiple devices presents a challenge to coordinate the sequence IDs across the multiple devices. If an Observation Domain only exists on a single device, then that is not a problem. In other words. A device exporting NetFlow data will have one or more exporting processes, separate from other devices exporting NetFlow data, each with one or more of their own export processes, and, as the RFC points out: "Observation Domain ID, which is unique per Exporting Process."