4

If I am looking at traffic from my network and I see (e.g., using SmartWhois) that a lot of user traffic is going to certain CDNs. e.g., in one network 87% of my traffic goes to Amazon. Is there a way to dis-aggregate this, even a little, to get a sense of what all of this traffic means?

I see that certain popular Web sites are located on certain CDNs (e.g., click the CDN names on the legend of this chart: CDN Market Share) but matching things up this way feels like I am simply guessing, particularly with Amazon -- the CDN market leader.

I was wondering if certain CDN clients might be assigned certain IP addresses within the CDN block -- I thought that someone else might have come up with a lookup table or a database for sale, but I can't find anything like that by googling, so maybe that is not a viable approach.

I currently only have easy access to the IP address that flows are going to. With that data, is dis-aggregating this CDN traffic further impossible? If I know it is not possible I will stop looking.

Improve this question
2
  • As @John Jensen mentions below, what problem are you trying to solve? Are you attempting to montior/police user behavior? Are you trying to create a geoIP type solution? Or is it just plain old-fashioned curiosity about where your traffic is headed? - - - If we know what problem you're looking at, we may be able to suggest alternate ways of solving it.
    – Brett Lykins
    Commented Jul 13, 2013 at 4:45
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer.
    – Ron Maupin ♦
    Commented Jan 6, 2021 at 15:37

4 Answers 4

Reset to default
5

What problem are you trying to solve?

Most CDN's work off of Anycast/GeoIP to serve the requested content as close to the source of the request as possible. I'm also not sure what you mean or what you're trying to accomplish by "finding out what users are doing" - CDN's are designed to be transparent to users to provide a better user experience when browsing the web (the biggest use case for a CDN - there are obviously others). I'm having a tough time thinking of what an average user would use a CDN for for reasons that would warrant monitoring of this nature.

If you did want to build a lookup table or a database, it shouldn't be that difficult, since you could query whois or an IRR to get IP/routing information (assuming the CDN operator does the Right Thing and puts their info in the IRR - most of the big ones do). I'd start with figuring out which CDN's own which blocks and then cross-referencing which IP belongs to which block, and then you could make a distinction on who owns the CDN IP that your users were hitting. It's possible, but it will get hairy very quickly and may very well be an exercise in futility (thinking if CDN's resell services to other smaller companies and solutions like CloudFront).

Improve this answer
1
  • +1, particularly the point about CDNs are designed to be transparent to the user
    – Mike Pennington
    Commented Jul 11, 2013 at 18:50
0

If you are attempting to monitor web traffic you will probably want some type of web logging device to cross reference your flow information against. Time and source IP address would then be analyzed against your web logging appliance or web filter to see what URL that particular user accessed.

A solution like websense web filter would work for this.

Improve this answer
0

I was wondering if certain CDN clients might be assigned certain IP addresses within the CDN block -- I thought that someone else might have come up with a lookup table or a database for sale, but I can't find anything like that by googling, so maybe that is not a viable approach.

Usually this kind of data (ip address distribution between projects) is internal, so trying to achieve it will lead you to the grey area at best, with all the consequences.

I currently only have easy access to the IP address that flows are going to. With that data, is dis-aggregating this CDN traffic further impossible? If I know it is not possible I will stop looking.

I think you need to go deeper in L7 headers (particularly, HTTP and DNS) and somehow correlate it to URLs of the websites serving data through CDN. So, no, there's not much can be done having only IP addresses.

Improve this answer
0

CDN activity is an interesting question and one I see coming up more and more. I was delivering a training course earlier in a London (UK) university on network monitoring techniques and one of the biggest consumers of bandwidth on their 1Gb Internet connection was CDN services.

I recently published a blog post on the subject which you can access from this link. For me the only way to work out what is happening is by focusing on HTTP headers and DNS traffic.

Some info from the blog post

"Network monitoring tools will run into problems if they start reporting on this activity based on the ownership of IP addresses alone. Most of the bandwidth consumption will be associated with the CDN sites and not the sites which the users are visiting to access the content. This becomes problematic for network administrators who may need to implement changes to manage this bandwidth consumption. If they block or throttle the CDN sites then they will restrict access to thousands of other services.

The solution to the problem is to look beyond layer 3 analysis and monitor HTTP headers and DNS queries. This data is normally sourced via packet capture with an application which can do deep packet inspection"

Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.