Problem background: Our application communicates with multiple other nodes over SCTP. A few times in past issues were reported because of network fluctuation or too much delays that appeared for very short duration. Network snoop of the duration could have helped a lot to understand the behaviors of all nodes in that duration.
One approach I'm planning to take is monitoring SCTP Re-transmission using a network sniffer, tcpdump, ngrep, wireshark etc every 1-2 minutes dump 15-30 seconds of data in a file and check if re-transmissions are above a certain threshold then trigger above actions. Additionally, I would like to monitor any abnormal situations like higher re-transmissions, long delays, packet-drops etc and take a few measures before even things actually fail (may be I'm being too optimistic here)
one major problem with above approach this will run every few minutes, can't keep it running because it will also consume resources and may impact application performance or do we have sniffers with very low performance impact ? and fluctuation may occur when sniffer is not running.
Monitoring network and fluctuations sounds like a common problem other may have already worked upon, hoping to get some pointers to save some time :)