How to Block Frostwire, utorrent and Other P2P Protocols


By Art Reisman, CTO, http://www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Disclaimer: It is considered controversial and by some definitions illegal for a US-based ISP to use deep packet inspection on the public Internet.

At APconnections, we subscribe to the philosophy that there is more to be gained by explaining your technology secrets than by obfuscating them with marketing babble. Read on to learn how I hunt down aggressive P2P traffic.

In order to create a successful tool for blocking a P2P application, you must first figure out how to identify P2P traffic. I do this by looking at the output data dump from a P2P session.

To see what is inside the data packets I use a custom sniffer that we developed. Then to create a traffic load, I use a basic Windows computer loaded up with the latest utorrent client.

Editors Note: The last time I used a P2P engine on a Windows computer, I ended up reloading my Windows OS once a week. Downloading random P2P files is sure to bring in the latest viruses, and unimaginable filth will populate your computer.

The custom sniffer is built into our NetGladiator device, and it does several things:

1) It detects and dumps the data inside packets as they cross the wire to a file that I can look at later.

2) It maps non printable ASCII characters to printable ASCII characters. In this way, when I dump the contents of an IP packet to a file, I don’t get all kinds of special characters embedded in the file. Since P2P data is encoded random music files and video, you can’t view data without this filter. If you try, you’ll get all kinds of garbled scrolling on the screen when you look at the raw data with a text editor.

So what does the raw data output dump of a P2P client look like ?

Here is a snippet of some of the utorrent raw data I was looking at just this morning. The sniffer has converted the non printable characters to “x”.
You can clearly see some repeating data patterns forming below. That is the key to identifying anything with layer 7. Sometimes it is obvious, while sometimes you really have work to find a pattern.

Packet 1 exx_0ixx`12fb*!s[`|#l0fwxkf)d1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:ka 31:v4:utk21:y1:qe
Packet 2 exx_0jxx`1kmb*!su,fsl0’_xk<)d1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:xv4^1:v4:utk21:y1:qe
Packet 3 exx_0kxx`1exb*!sz{)8l0|!xkvid1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:09hd1:v4:utk21:y1:qe
Packet 4 exx_0lxx`19-b*!sq%^:l0tpxk-ld1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:=x{j1:v4:utk21:y1:qe

The next step is to develop a layer 7 regular expression to identify the patterns in the data. In the output you’ll notice the string “exx” appears in line, and that is what you look for. A repeating pattern is a good place to start.

The regular expression I decided to use looks something like:

exx.0.xx.*qe

This translates to: match any string starting with “exx” followed, by any character “.” followed by “0”, followed by “xx”, followed by any sequence of characters ending with “qe”.

Note: When I tested this regular expression it turns out to only catch a fraction of the Utorrent, but it is a start. What you don’t want to do is make your regular expression so simple that you get false positives. A layer 7 product that creates a high degree of false positives is pretty useless.

The next thing I do with my new regular expression is a test for accuracy of target detection and false positives.

Accuracy of detection is done by clearing your test network of everything except the p2p target you are trying to catch, and then running your layer 7 device with your new regular expression and see how well it does.

Below is an example from my NetGladiator in a new sniffer mode. In this mode I have the layer 7 detection on, and I can analyze the detection accuracy. In the output below, the sniffer puts a tag on every connection that matches my utorrent regular expression. In this case, my tag is indicated by the word “dad” at the end of the row. Notice how every connection is tagged. This means I am getting 100 percent hit rate for utorrent. Obviously I doctored the output for this post :)

ndex SRCP DSTP Wavg Avg IP1 IP2 Ptcl Port Pool TOS
0 0 0 17 53 255.255.255.255 95.85.150.34 — 2 99 dad
1 0 0 16 48 255.255.255.255 95.82.250.60 — 2 99 dad
2 0 0 16 48 255.255.255.255 95.147.1.179 — 2 99 dad
3 0 0 18 52 255.255.255.255 95.252.60.94 — 2 99 dad
4 0 0 12 24 255.255.255.255 201.250.236.194 — 2 99 dad
5 0 0 18 52 255.255.255.255 2.3.200.165 — 2 99 dad
6 0 0 10 0 255.255.255.255 99.251.180.164 — 2 99 dad
7 0 0 88 732 255.255.255.255 95.146.136.13 — 2 99 dad
8 0 0 12 0 255.255.255.255 189.202.6.133 — 2 99 dad
9 0 0 12 24 255.255.255.255 79.180.76.172 — 2 99 dad
10 0 0 16 48 255.255.255.255 95.96.179.38 — 2 99 dad
11 0 0 11 16 255.255.255.255 189.111.5.238 — 2 99 dad
12 0 0 17 52 255.255.255.255 201.160.220.251 — 2 99 dad
13 0 0 27 54 255.255.255.255 95.73.104.105 — 2 99 dad
14 0 0 10 0 255.255.255.255 95.83.176.3 — 2 99 dad
15 0 0 14 28 255.255.255.255 123.193.132.219 — 2 99 dad
16 0 0 14 32 255.255.255.255 188.191.192.157 — 2 99 dad
17 0 0 10 0 255.255.255.255 95.83.132.169 — 2 99 dad
18 0 0 24 33 255.255.255.255 99.244.128.223 — 2 99 dad
19 0 0 17 53 255.255.255.255 97.90.124.181 — 2 99 dad

A bit more on reading this sniffer output…

Notice columns 4 and 5, which indicate data transfer rates in bytes per second. These columns contain numbers that are less than 100 bytes per second – Very small data transfers. This is mostly because as soon as that connection is identified as utorrent, the NetGladiator drops all future packets on the connection and it never really gets going. One thing I did notice is that the modern utorrent protocol hops around very quickly from connection to connection. It attempts not to show it’s cards. Why do I mention this? Because in layer 7 shaping of P2P, speed of detection is everything. If you wait a few milliseconds too long to analyze and detect a torrent, it is already too late because the torrent has transferred enough data to keep it going. It’s just a conjecture, but I suspect this is one of the main reasons why this utorrent is so popular. By hopping from source to source, it is very hard for an ISP to block this one without the latest equipment. I recently wrote a companion article regarding the speed of the technology behind a good layer 7 device.

The last part of testing a regular expression involves looking for false positives. For this we use a commercial grade simulator. Our simulator uses a series of pre-programmed web crawlers that visit tens of thousands of web pages an hour at our test facility. We then take our layer 7 device with our new regular expression and make sure that none of the web crawlers accidentally get blocked while reading thousands of web pages. If this test passes we are good to go with our new regular expression.

Editors Note: Our primary bandwidth shaping product manages P2P without using deep packet inspection.
The following layer 7 techniques can be run on our NetGladiator Intrusion Prevention System. We also advise that public ISPs check their country regulations before deploying a deep packet inspection device on a public network.

White Paper: A Simple Guide to Network Capacity Planning


After many years of consulting and supporting the networking world with WAN optimization devices, we have sensed a lingering fear among Network Administrators who wonder if their capacity is within the normal range.

So the question remains:

How much bandwidth can you survive with before you impact morale or productivity?

The formal term we use to describe the number of users sharing a network link to the Internet is  contention ratio. This term  is defined as  the size of an Internet trunk divided by the number of users. We normally think of Internet trunks in units of megabits. For example, 10 users sharing a one megabit trunk would have a 10-to- 1 contention ratio. If sharing the bandwidth on the trunk equally and simultaneously, each user could sustain a constant feed of 100kbs, which is exactly 1/10 of the overall bandwidth.

From a business standpoint, it is whatever a customer will put up with and pay for without canceling their service. This definition may seem ethically suspect, but whether in the bygone days of telecommunications phone service or contemporary Internet bandwidth business, there are long-standing precedents for overselling. What do you think a circuit busy signal is caused by? Or a dropped cell phone call?

So, without pulling any punches, what exactly will a customer tolerate before pulling the plug?

Here are some basic observations about consumers and acceptable contention ratios:

  • Rural customers in the US and Canada: Contention ratios of 50 to 1 are common
  • International customers in remote areas of the world: Contention ratios of 80 to 1 are common
  • Internet providers in urban areas: Contention ratios of 15 to 1 are to be expected
  • Generic Business ratio 50 to 1 , and sometimes higher

Update Jan 2015, quite a bit has happened since these original numbers were published. Internet prices have plummeted, here is my updated observation.

Rural customers in the US and Canada: Contention ratios of 10 to 1 are common
International customers in remote areas of the world: Contention ratios of 20 to 1 are common
Internet providers in urban areas: Contention ratios of 2 to 1 are to be expected
Generic Business ratio 5 to 1 , and sometimes higher

As a rule Businesses can general get away with slightly higher contention ratios.  Most business use does not create the same load as recreational use, such as YouTube and File Sharing. Obviously, many businesses will suffer the effects of recreational use and perhaps haphazardly turn their heads on enforcement of such use. The above ratio of 50 to 1 is a general guideline of what a business should be able to work with, assuming they are willing to police their network usage and enforce policy.

The numbers above are a good, rough starting point, but things are not as simple as they look. There is a statistical twist as bandwidth amounts get higher.

Contention ratios can actually increase as the overall Internet trunk size gets larger. For example, if 50 people can share one megabit without mutiny, it should follow that 100 people can share two megabits without mutiny as the ratio has not changed. It is still 50 to 1.

However, from observations of hundreds of ISPs, we can easily conclude that perhaps 110 people can share two megabits with the same tolerance as 50 people sharing one megabit. What this means is that the larger the ISP, the more bandwidth at a fixed cost per megabit, and thus the larger the contention ratios you can get away with.

Is this really true? And if so, what are its implications for your business?

This is simply an empirical observation, backed up by talking to literally thousands of ISPs over the course of four years and noticing how their oversubscription ratios increase with the size of their trunk.

A conservative estimate is that, starting with the baseline ratio listed above, you can safely add 10 percent more subscribers above and beyond the original contention ratio for each megabit of trunk they share.

Thus, to provide an illustration, 50 people sharing one megabit can safely be increased to 110 people sharing two megabits, and at four megabits you can easily handle 280 customers. With this understanding, getting more from your bandwidth becomes that much easier.

I also ran across this thread in a discussion group for Resnet Adminstrators around the country.

From Resnet Listserv

Brandon  Enright at University of California San Diego breaks it down as follows:
Right now we’re at .2 Mbps per student.  We could go as low as .1 right
now without much of any impact.  Things would start to get really ugly
for us at .05 Mpbs / student.

So at 10k students I think our lower-bound is 500 Mbps.

I can’t disclose what we’re paying for bandwidth but even if we fully
saturated 2Gbps for the 95% percentile calculation it would come out to
be less than $5 per student per month.  Those seem like reasonable
enough costs to let the students run wild.
Brandon

Editors note: I am not sure why a public institution can’t  exactly disclose what they are paying for bandwidth ( Brian does give a good hint), as this would be useful to the world for comparison; however many Universities get lower than commercial rates through state infrastructure not available to private operators.

Related Article ISP contention ratios.

By Art Reisman

Art Reisman CTO www.netequalizer.com

Editor’s note: Art Reisman is the CTO of APconnections. APconnections designs and manufactures the popular NetEqualizer bandwidth shaper.

Equalizing Compared to Application Shaping (Traditional Layer-7 “Deep Packet Inspection” Products)


Editor’s Note: (Updated with new material March 2012)  Since we first wrote this article, many customers have implemented the NetEqualizer not only to shape their Internet traffic, but also to shape their company WAN.  Additionally, concerns about DPI and loss of privacy have bubbled up. (Updated with new material September 2010)  Since we first published this article, “deep packet inspection”, also known as Application Shaping, has taken some serious industry hits with respect to US-based ISPs.   

==============================================================================================
Author’s Note: We often get asked how NetEqualizer compares to Packeteer (Bluecoat), NetEnforcer (Allot), Network Composer (Cymphonix), Exinda, and a plethora of other well-known companies that do Application Shaping (aka “packet shaping”, “deep packet inspection”, or “Layer-7” shaping).   After several years of these questions, and discussing different aspects with former and current application shaping with IT administrators, we’ve developed a response that should clarify the differences between NetEqualizer’s behavior- based approach and the rest of the pack.
We thought of putting our response into a short, bullet-by-bullet table format, but then decided that since this decision often involves tens of thousands of dollars, 15 minutes of education on the subject with content to support the bullet chart was in order.  If you want to skip the details, see our Summary Table at the end of this article

However, if you’re looking to really understand the differences, and to have the question answered as objectively as possible, please take a few minutes to read on…
==============================================================================================

How NetEqualizer compares to Bluecoat, Allot, Cymphonix, & Exinda

In the following sections, we will cover specifically when and where Application Shaping is used, how it can be used to your advantage, and also when it may not be a good option for what you are trying to accomplish.  We will also discuss how Equalizing, NetEqualizer’s behavior-based shaping, fits into the landscape of application shaping, and how in many cases Equalizing is a much better alternative.

Download the full article (PDF)  Equalizing Compared To Application Shaping White Paper

Read the rest of this entry »

NetEqualizer Bandwidth Shaping Solution: Hotels & Resorts


In working with some of the world’s leading hotels and resorts, we’ve repeatedly heard the same issues and challenges facing network administrators. Here are just a few:

Download Hotels White Paper

  • We need to do more with less bandwidth.
  • We need a solution that’s low cost, low maintenance, and easy to set up.
  • We need to meet the expectations of our tech-savvy customers and prevent Internet congestion during times of peak usage.
  • We need a solution that can meet the demands of a constantly changing clientele. We need to offer tiered internet access for our hotel guests, and provide managed access for conference attendees.

In this article, we’ll talk about how the NetEqualizer has been used to solve these issues for many Hotels and Resorts around the world.

Download article (PDF) Hotels & Resorts White Paper

Read full article …

NetEqualizer Bandwidth Shaping Solution: Business Centers


In working with numerous Business Center network administrators, we have heard the same issues and challenges repeatedly. Here are just a few:

Download Business Centers White Paper

  • We need to do more with less bandwidth.
  • We need a solution that’s low cost, low maintenance, and easy to set up.
  • We need to support selling fixed bandwidth to our customers, by office and/or user.
  • We need to be able to report on subscriber usage.
  • We need to increase user satisfaction and reduce network troubleshooting calls

In this article, we’ll talk about how the NetEqualizer has been used to solve these issues for many Business Centers around the world.

Download article (PDF) Business Centers White Paper

Read full article …

NetEqualizer Bandwidth Shaping Solution: Telecom, Satellite Systems, Cable, and Wired and Wireless ISPs


In working with Internet providers around the world, we’ve repeatedly heard the same issues and challenges facing network administrators. Here are just a few:

Download ISP White Paper

  • We need to support selling fixed bandwidth to our customers.
  • We need to be able to report on subscriber usage.
  • We need the ability to increase subscriber ratio, or not have a subscriber cutback, before having to buy more bandwidth.
  • We need to meet the varying needs of all of our users.
  • We need to manage P2P traffic.
  • We need to give VoIP traffic priority.
  • We need to make exemptions for customers routing all of their traffic through VPN tunnels.
  • We need a solution that’s low cost, low maintenance, and easy to set up.
  • We need a solution that will grow with our network.
  • We need a solution that will meet CALEA requirements.

In this article, we will talk about how the NetEqualizer has been used to solve these issues for Internet providers worldwide.

Download article (PDF) ISP White Paper

Read full article …

NetEqualizer Bandwidth Shaping Solution: Libraries


In working with libraries across the country, we have heard the same issues and challenges repeatedly from network administrators.  Here are just a few:

Download Library White Paper

  • We need to meet the varying needs of all of our patrons while keeping the network truly open to the public.
  • We need to ensure access to our online resources for remote users (online catalogs, databases, etc.).
  • We need to do more with less bandwidth.
  • We need a solution that’s low cost, low maintenance, and easy to set up.
  • We need a solution that will grow with our network.

In this article, we’ll talk about how the NetEqualizer has been used to solve these issues for many libraries around the world.

Download article (PDF) Library White Paper

Read full article …

%d bloggers like this: