a way to log how much traffic is used by each website - network wide

Question

So basically i need to know how much traffic is used by each website on my entire network. I want to install something on my firewall device to monitor how much traffic does each website takes. I don't mean traffic per host - There's tonnes of tools made for that. I need to ONLY know about websites.

It is not viable to install a monitoring software on each device, I want it network wide.

Theoretically i thought of this:

• Since each client caches DNS, we cannot know in each request which website was visited, so using DNS to find out which website is being visited each request would be unwise.

• Some people might be using a custom DNS, which would also be a reason why the above might not work

• Each https website however does send a ClientHello, which can be intercepted by my firewall to know which site is being visited. And i also know that in order to get a reply from the server, the NAT device (we can say in this case my firewall) opens a random port so server can reply there.

However i could not find any tool which does that, or at least does that efficiently.

I started writing my own tool but quickly realized it won't work out really well, since it wont scale.

github.com/amrinder-cs/ClientHello-Capture

^ This, for now it just captures the ClientHello and doesn't log, but i'll be having atleast 6k users in a single network (a large enterprise) so my tool might not scale up.

Edit: Found the answer, Deep Packet Inspection, currently using netify.ai

Answer 1

That requirement sounds easy but it isn't.

You'll need a firewall (or comparable device) that logs application-layer data - the website name is only visible there. You can pretty much forget about DNS for that purpose.

While that is rather easy when dealing with HTTP only, it gets harder when it comes to HTTPS: the firewall needs to analyze the server name indication SNI field for the SSL handshake. The actual HTTPS traffic is encrypted - in order to analyze/log that, you'd need to break encryption MitM style and re-encrypt using your own CA certificate that needs to be rolled out to the clients.

If you're looking for product recommendations, those are explicitly off topic here as on most SE sites.