NetEqualizer News: February 2014


February 2014

Greetings!

Enjoy another issue of NetEqualizer News! This month, we preview our new Software Update 8.0 Beta (which includes Protocol Tracking reports), as well as our new Mogicho licensed movie content service for customers with our caching option, and lastly we remind you about our NetEqualizer trade-in policy. As always, feel free to pass this along to others who might be interested in NetEqualizer News.

A message from Art…
Art Reisman, CTO – APconnections

As we move into February, we have been glued to the TV watching the Olympic games – well, actually glued to “on demand” video replays on our iPads and laptops. We love that there are no commercials and that we can watch whenever we want! If you find yourself doing the same, you will not be surprised that more and more customers mention that video is a huge percentage of their Internet traffic.
art_canoe_picture
We recognize this trend, and are working on this in several areas. This month we talk about our new Protocol Tracking reports, which will help you to make sense of all the video on your network. Also, as I mentioned last month, I am working on partnerships with the movie industry. We call this offering “Mogicho”, and you can read more about it below. 2014 begins, we are excited to see what the year brings. In the United States, the economy is finally improving, at least when measured by job creation, stock market growth, and real estate sales. Hopefully, this trend continues, as we are ready for the Great Recession to be officially over! We hope that you are seeing an improving economy in your part of the world too.

We love it when we hear back from you – so if you have a story you would like to share with us of how we have helped you, let us know. Email me directly at art@apconnections.net. I would love to hear from you!

Software Update 8.0 Beta: Protocol Tracking Reports

We are excited to announce the 8.0 Beta Release for NetEqualizer where we introduce Protocol Tracking reports!

Protocol Tracking reports extend our Dynamic Real-Time Reporting (RTR) – Traffic Reporting capabilities. Through Protocol Tracking, we enable you track various Internet protocols (Hulu, YouTube, Netflix, iTunes, Pandora, etc.) and then view these protocols and bandwidth usage for an IP or your entire network in a pie or bar chart format.

s1

We built our RTR tool last year, and based on the feedback that we received, we have continued our investment in visibility and reporting in 2014. As many of you note that you need to better understand how much video is on your network, and what types of traffic are consuming your bandwidth, we felt that the time was right for Protocol Tracking reports.

RTR_Protocol_Tracking_One_IP_Pie2

When we introduced our NetGladiator IPS in mid-2012, we built it out with a powerful Layer 7 engine. We have now incorporated this engine into our NetEqualizer code base, so that we can offer you a way to report on protocols on your network. We see this as a complementary component that enhances our NetEqualizer shaping features, and we are excited to offer you this additional visibility into your network traffic.

How to Participate in the 8.0 Beta

The 8.0 Beta is subject to availability. If you are interested in the Beta, and meet the following requirements, please contact us to see if this is a good fit for your situation.

Beta requirements:

– must be current on NSS

– must be running NE3000 or NE4000

– must be on Software Update 7.5

– must be available to provide feedback on the 8.0 Beta features

Once 8.0 reaches GA, these features will be free to customers with valid NetEqualizer Software and Support who are running version 7.5. If you are not current with NSS, contact us today at:

sales@apconnections.net

-or-

303-997-1300


Trade In Your Old NetEqualizer!

Many of you may not be aware that we offer a generous trade-in credit when it is time to retire your old NetEqualizer. We will give you 50%* of the original unit price plus 50% of any license upgrades towards a new unit.

We believe that this offer is unmatched in the industry. Why do we do this? Because we believe that bandwidth shaping should be affordable, and as part of that, we help you to protect your original investment.

How to know if you need a new NetEQ?

1) You can no longer run our current software. NE2000’s earlier than August 2011 cannot run 7.0+.

2) Your hardware warranty has long expired and your unit is over five years old. We offer four years of hardware warranty (NHW), which must be purchased each consecutive year from your original purchase. If you are outside of warranty, your unit is probably past its useful life.

3) You want the faster processors and more memory that new hardware can give you. Just like a PC or tablet, the hardware that we offer today is faster and has more memory than older boxes.

4) And the most important reason, because you want one!

And remember, while we will continue to offer license upgrades on our NE2000 series as needed, if you have a NE2000 purchased before August 2011, it will only be supported through 12/31/2014. If you have an older NE2000, please contact us sometime in 2014 to discuss your options.

If you have questions on trade-in units, feel free to contact us at:

sales@apconnections.net

-or-

303-997-1300

* This offer does not apply to POE or Lite units.


Coming Soon: Mogicho
A new movie offering that leverages our NetEqualizer Caching Option (NCO) infrastructure

In addition to our other projects, we are also progressing with our “Mogicho” offer to bring an amazing assortment of licensed movie content into our NCO caching server. We will be finalizing the license with the content providers next month, and we are excited to say it will be a nice value-add to any ISP needing to enhance their offer to their constituents. We think Mogicho will be especially of interest to ISPs with less than a 5Gbps Netflix streams2 (where Netflix currently is not interested in providing Netflix Open Connect (caching engine)).

Where will this be available?

Mogicho is a new distribution channel for unique online movie content, targeted at rural and remote users not served by Major Internet Providers (M-ISPs).

We are initially releasing Mogicho within the United States and Canada. There are millions of consumers in the United States and Canada who are served by second-tier Internet providers. These second-tier providers (2T-ISPs) include small town cable operators, small town phone companies, and independent rural WISPs.

How does Mogicho work?

Mogicho movie content is secured & stored locally on the NetEqualizer at the 2T-ISP NOC. Video is streamed through the 2T-ISP Access Points directly to customers. Video is available at higher speeds than today, as each customer is not competing with all customers for video to come through the M-ISP throttling point.

s3Looking for more announcements this spring on Mogicho, and if you have questions, contact us at:

sales@apconnections.net

-or-

303-997-1300


Best Of The Blog

Guest Article From a WISP Owner in the Trenches

By Rory Conaway – Triad Wireless

Editors Note: A great read if you are thinking of starting a WISP and need a little inspiration. Re-posted with permission from Rory Conaway, Triad Wireless. Rory is president and CEO of Triad Wireless, an engineering and design firm in Phoenix. Triad Wireless specializes in unique RF data and network designs for municipalities, public safety and educational campuses. E-mail comments to rory@triadwireless.net.

Tales from the Towers – Chapter 50: CRY ‘HAVOC!’, AND LET SLIP THE DOGS OF WAR

Interesting fellow that Shakespeare because not only did he write plays, he also acted in them. And although Tales from the Towers doesn’t hold a candle (pre-electric times, you can groan now) to Mr. William’s contributions to culture, I have a double life too. If you haven’t guessed it yet, writing articles really isn’t my full-time job (my wife is giving me the look that says I should find another hobby), I actually run a WISP, do installs, and handle tech support calls. After 10 years though, and many mistakes and successes, I’ve decided to rethink my network from the ground up as if I was starting tomorrow and share that.

The idea is to help lay out a simplified road map that will bring forth thousands of new WISPs into the market that can start breaking down the digital divide without taxpayer money and creating a new business. Since a thousand bee stings can take out the biggest animal, the more companies that jump into the industry, the better the chances of competing against the incumbents. It’s time to open the floodgates of small business entrepreneurs and begin the war for last mile bandwidth delivery everywhere. And although few outside Star Trek fans will recognize one of Shakespeare’s most famous sayings, they will recognize this modern variation, “Who let the dogs out”! Hopefully it’s the WISP industry…

Photo Of The Month

IMG_0948

Storm Chasing Convention
Given the fact that the winter weather has been so severe this year, Art decided to check in with Greg Forbes (The Weather Channel severe weather expert) at a local storm chaser convention. Here he is with him last weekend. Unfortunately, after talking with him, he can’t make any promises about good weather in the spring.

A Brief History of Peer to Peer File Sharing and the Attempts to Block It


By Art Reisman

The following history is based on my notes and observations as both a user of peer to peer, and as a network engineer tasked with cleaning  it up.

Round One, Napster, Centralized Server, Circa 2002

Napster was a centralized service, unlike the peer to peer behemoths of today there was never any question of where the copyrighted material was being stored and pirated from. Even though Napster did not condone pirated music and movies on their site, the courts decided by allowing copyrighted material to exist on their servers, they were in violation of copyright law. Napster’s days of free love were soon over.

From an historic perspective the importance of the decision to force the shut down of Napster was that it gave rise to a whole new breed of p2p applications. We detailed this phenomenon in our 2008 article.

Round Two, Mega-Upload  Shutdown, Centralized Server, 2012

We again saw a doubling down on p2p client sites (they expanded) when the Mega-Upload site, a centralized sharing site, was shutdown back in Jan 2012.

“On the legal side, the recent widely publicized MegaUpload takedown refocused attention on less centralized forms of file sharing (i.e. P2P). Similarly, improvements in P2P technology coupled with a growth in file sharing file size from content like Blue-Ray video also lead many users to revisit P2P.”

Read the full article from deepfield.net

The shut down of Mega-Upload had a personal effect on me as I had used it to distribute a 30 minute account from a 92-year-old WWII vet where he recalled, in oral detail, his experience of surviving a German prison camp.

Blocking by Signature, Alias Layer 7 Shaping, Alias Deep packet inspection. Late 1990’s till present

Initially, the shining star savior in the forefront against spotting illegal content on your network, this technology can be expensive and fail miserably in the face of newer encrypted p2p applications. It also can get quite expensive to keep up with the ever changing application signatures, and yet it is still often the first line of defense attempted by ISPs.

We covered this topic in detail, in our recent article,  Layer 7 Shaping Dying With SSL.

Blocking by Website

Blocking the source sites where users download their p2p clients is still possible. We see this method applied at mostly private secondary schools, where content blocking is an accepted practice. This method does not work for computers and devices that already have p2p clients. Once loaded, p2p files can come from anywhere and there is no centralized site to block.

Blocking Uninitiated Requests. Circa Mid-2000

The idea behind this method is to prevent your Network from serving up any content what so ever! Sounds a bit harsh, but the average Internet consumer rarely, if ever, hosts anything intended for public consumption. Yes at one time, during the early stages of the Internet, my geek friends would set up home pages similar to what everybody exposes on Facebook today. Now, with the advent hosting sites, there is just no reason for a user to host content locally, and thus, no need to allow access from the outside. Most firewalls have a setting to disallow uninitiated requests into your network (obviously with an exemption for your publicly facing servers).

We actually have an advanced version of this feature in our NetGladiator security device. We watch each IP address on your internal network and take note of outgoing requests, nobody comes in unless they were invited. For example, if we see a user on the Network make a request to a Yahoo Server , we expect a response to come back from a Yahoo server; however if we see a Yahoo server contact a user on your network without a pending request, we block that incoming request. In the world of p2p this should prevent an outside client from requesting a receiving a copyrighted file hosted on your network, after all no p2p client is going to randomly send out invites to outside servers or would they?

I spent a few hours researching this subject, and here is what I found (this may need further citations). It turns out that p2p distribution may be a bit more sophisticated and has ways to get around the block uninitiated query firewall technique.

P2P networks such as Pirate Bay use a directory service of super nodes to keep track of what content peers have and where to find them. When you load up your p2p client for the first time, it just needs to find one super node to get connected, from there it can start searching for available files.

Note: You would think that if these super nodes were aiding and abetting in illegal content that the RIAA could just shut them down like they did Napster. There are two issues with this assumption:

1) The super nodes do not necessarily host content, hence they are not violating any copyright laws. They simply coordinate the network in the same way DNS service keep track of URL names and were to find servers.
2) The super nodes are not hosted by Pirate Bay, they are basically commandeered from their network of users, who unwittingly or unknowingly agree to perform this directory service when clicking the license agreement that nobody ever reads.

From my research I have talked to network administrators that claim despite blocking uninitiated outside requests on their firewalls, they still get RIAA notices. How can this be?

There are only two ways this can happen.

1) The RIAA is taking liberty to simply accuse a network of illegal content based on the directory listings of a super node. In other words if they find a directory on a super node pointing to copyrighted files on your network, that might be information enough to accuse you.

2) More likely, and much more complex, is that the Super nodes are brokering the transaction as a condition of being connected. Basically this means that when a p2p client within your network, contacts a super node for information, the super node directs the client to send data to a third-party client on another network. Thus the send of information from the inside of your network looks to the firewall as if it was initiated from within. You may have to think about this, but it makes sense.

Behavior based thwarting of p2p. Circa 2004 – NetEqualizer

Behavior-based shaping relies on spotting the unique footprint of a client sending and receiving p2p applications. From our experience, these clients just do not know how to lay low and stay under the radar. It’s like the criminal smuggling drugs doing 100 MPH on the highway, they just can’t help themselves. Part of the p2p methodology is to find as many sources of files as possible, and then, download from all sources simultaneously. Combine this behavior with the fact that most p2p consumers are trying to build up a library of content, and thus initiating many file requests, and you get a behavior footprint that can easily be spotted. By spotting this behavior and making life miserable for these users, you can achieve self compliance on your network.

Read a smarter way to block p2p traffic.

Blocking the RIAA probing servers

If you know where the RIAA is probing from you can deny all traffic to their probes and thus prevent the probe of files on your network, and ensuing nasty letters to desist.

How to Block Frostwire, utorrent and Other P2P Protocols


By Art Reisman, CTO, http://www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Disclaimer: It is considered controversial and by some definitions illegal for a US-based ISP to use deep packet inspection on the public Internet.

At APconnections, we subscribe to the philosophy that there is more to be gained by explaining your technology secrets than by obfuscating them with marketing babble. Read on to learn how I hunt down aggressive P2P traffic.

In order to create a successful tool for blocking a P2P application, you must first figure out how to identify P2P traffic. I do this by looking at the output data dump from a P2P session.

To see what is inside the data packets I use a custom sniffer that we developed. Then to create a traffic load, I use a basic Windows computer loaded up with the latest utorrent client.

Editors Note: The last time I used a P2P engine on a Windows computer, I ended up reloading my Windows OS once a week. Downloading random P2P files is sure to bring in the latest viruses, and unimaginable filth will populate your computer.

The custom sniffer is built into our NetGladiator device, and it does several things:

1) It detects and dumps the data inside packets as they cross the wire to a file that I can look at later.

2) It maps non printable ASCII characters to printable ASCII characters. In this way, when I dump the contents of an IP packet to a file, I don’t get all kinds of special characters embedded in the file. Since P2P data is encoded random music files and video, you can’t view data without this filter. If you try, you’ll get all kinds of garbled scrolling on the screen when you look at the raw data with a text editor.

So what does the raw data output dump of a P2P client look like ?

Here is a snippet of some of the utorrent raw data I was looking at just this morning. The sniffer has converted the non printable characters to “x”.
You can clearly see some repeating data patterns forming below. That is the key to identifying anything with layer 7. Sometimes it is obvious, while sometimes you really have work to find a pattern.

Packet 1 exx_0ixx`12fb*!s[`|#l0fwxkf)d1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:ka 31:v4:utk21:y1:qe
Packet 2 exx_0jxx`1kmb*!su,fsl0’_xk<)d1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:xv4^1:v4:utk21:y1:qe
Packet 3 exx_0kxx`1exb*!sz{)8l0|!xkvid1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:09hd1:v4:utk21:y1:qe
Packet 4 exx_0lxx`19-b*!sq%^:l0tpxk-ld1:ad2:id20:c;&h45h”2x#5wg;|l{j{e1:q4:ping1:t4:=x{j1:v4:utk21:y1:qe

The next step is to develop a layer 7 regular expression to identify the patterns in the data. In the output you’ll notice the string “exx” appears in line, and that is what you look for. A repeating pattern is a good place to start.

The regular expression I decided to use looks something like:

exx.0.xx.*qe

This translates to: match any string starting with “exx” followed, by any character “.” followed by “0”, followed by “xx”, followed by any sequence of characters ending with “qe”.

Note: When I tested this regular expression it turns out to only catch a fraction of the Utorrent, but it is a start. What you don’t want to do is make your regular expression so simple that you get false positives. A layer 7 product that creates a high degree of false positives is pretty useless.

The next thing I do with my new regular expression is a test for accuracy of target detection and false positives.

Accuracy of detection is done by clearing your test network of everything except the p2p target you are trying to catch, and then running your layer 7 device with your new regular expression and see how well it does.

Below is an example from my NetGladiator in a new sniffer mode. In this mode I have the layer 7 detection on, and I can analyze the detection accuracy. In the output below, the sniffer puts a tag on every connection that matches my utorrent regular expression. In this case, my tag is indicated by the word “dad” at the end of the row. Notice how every connection is tagged. This means I am getting 100 percent hit rate for utorrent. Obviously I doctored the output for this post :)

ndex SRCP DSTP Wavg Avg IP1 IP2 Ptcl Port Pool TOS
0 0 0 17 53 255.255.255.255 95.85.150.34 — 2 99 dad
1 0 0 16 48 255.255.255.255 95.82.250.60 — 2 99 dad
2 0 0 16 48 255.255.255.255 95.147.1.179 — 2 99 dad
3 0 0 18 52 255.255.255.255 95.252.60.94 — 2 99 dad
4 0 0 12 24 255.255.255.255 201.250.236.194 — 2 99 dad
5 0 0 18 52 255.255.255.255 2.3.200.165 — 2 99 dad
6 0 0 10 0 255.255.255.255 99.251.180.164 — 2 99 dad
7 0 0 88 732 255.255.255.255 95.146.136.13 — 2 99 dad
8 0 0 12 0 255.255.255.255 189.202.6.133 — 2 99 dad
9 0 0 12 24 255.255.255.255 79.180.76.172 — 2 99 dad
10 0 0 16 48 255.255.255.255 95.96.179.38 — 2 99 dad
11 0 0 11 16 255.255.255.255 189.111.5.238 — 2 99 dad
12 0 0 17 52 255.255.255.255 201.160.220.251 — 2 99 dad
13 0 0 27 54 255.255.255.255 95.73.104.105 — 2 99 dad
14 0 0 10 0 255.255.255.255 95.83.176.3 — 2 99 dad
15 0 0 14 28 255.255.255.255 123.193.132.219 — 2 99 dad
16 0 0 14 32 255.255.255.255 188.191.192.157 — 2 99 dad
17 0 0 10 0 255.255.255.255 95.83.132.169 — 2 99 dad
18 0 0 24 33 255.255.255.255 99.244.128.223 — 2 99 dad
19 0 0 17 53 255.255.255.255 97.90.124.181 — 2 99 dad

A bit more on reading this sniffer output…

Notice columns 4 and 5, which indicate data transfer rates in bytes per second. These columns contain numbers that are less than 100 bytes per second – Very small data transfers. This is mostly because as soon as that connection is identified as utorrent, the NetGladiator drops all future packets on the connection and it never really gets going. One thing I did notice is that the modern utorrent protocol hops around very quickly from connection to connection. It attempts not to show it’s cards. Why do I mention this? Because in layer 7 shaping of P2P, speed of detection is everything. If you wait a few milliseconds too long to analyze and detect a torrent, it is already too late because the torrent has transferred enough data to keep it going. It’s just a conjecture, but I suspect this is one of the main reasons why this utorrent is so popular. By hopping from source to source, it is very hard for an ISP to block this one without the latest equipment. I recently wrote a companion article regarding the speed of the technology behind a good layer 7 device.

The last part of testing a regular expression involves looking for false positives. For this we use a commercial grade simulator. Our simulator uses a series of pre-programmed web crawlers that visit tens of thousands of web pages an hour at our test facility. We then take our layer 7 device with our new regular expression and make sure that none of the web crawlers accidentally get blocked while reading thousands of web pages. If this test passes we are good to go with our new regular expression.

Editors Note: Our primary bandwidth shaping product manages P2P without using deep packet inspection.
The following layer 7 techniques can be run on our NetGladiator Intrusion Prevention System. We also advise that public ISPs check their country regulations before deploying a deep packet inspection device on a public network.

NetGladiator: A Layer 7 Shaper in Sheep’s Clothing


When explaining our NetGladiator technology the other day, a customer was very intrigued with our Layer 7 engine. He likened it to a caged tiger under the hood, gobbling up and spitting out data packets with the speed and cunning of the world’s most powerful feline.

He was surprised to see this level of capability in equipment offered at our prices.  He was impressed with the speed attained for the price point of our solution (more on this later in the article)…

In order to create a rock-solid IPS (Intrusion Prevention System), capable of handling network speeds of up to 1 gigabit with standard Intel hardware, we had to devise a technology breakthrough in Layer 7 processing. Existing technologies were just too slow to keep up with network speed expectations.

In order to support higher speeds, most vendors use semi-custom chip sets and a technology called “ASIC“. This works well but is very expensive to manufacture.

How do typical Layer 7 engines work?

Our IPS story starts with our old Layer 7 engine. It was sitting idle on our NetEqualizer product. We had shelved it when we got away from from Layer 7 shaping in favor of Equalizing technology, which is a superior solution for traffic shaping.  However, when we decided to move ahead with our new IPS this year, we realized we needed a fast-class analysis engine, one that could look at all data packets in real time. Our existing Layer 7 shaper only analyzed headers because that was adequate for its previous mission (detecting P2P streams).  For our new IPS system, we needed a solution that could do a deep dive into the data packets.  The IPS mission requires that you look at all the data – every packet crossing into a customer network.

The first step was to revamp the older engine and configure it to look at every packet. The results were disappointing.  With the load of analyzing every packet, we could not get throughput any higher than about 20 megabits, far short of our goal of 1 gigabit.

What do we do differently with our updated Layer 7 engine?

Necessity is the mother of invention, and so we invented a better Layer 7 engine.

The key was to take advantage of multiple processors for analysis of data without delaying data packets. The way the old technology worked was that it would intercept a data packet on a data link, hold it, analyze it for P2P patterns, and then send it on.  With this method, as packets come faster and faster you end up not having enough CPU time to do the analysis and still send the packet on without adding latency.  Many customers find this out the hard way when they update their data speeds from older slower T1 technology.  Typical analysis engines on affordable routers and firewalls often just can’t keep up with line speeds.

What we did was take advantage of a utility in the Linux Kernel called “clone skb”.  This allows you to make a temporary copy of the data packet without the overhead of copying.  More importantly, it allows us to send the packet on without delay and do the analysis within a millisecond (not quite line speed, but fast enough to stop an intruder).

We then combined the cloning with a new technology in the Linux kernel called Kernel Threading.  This is different than the technology that large multi-threaded HTTP servers use because it happens at the kernel level, and we do not have to copy the packet up to some higher-level server for analysis. Copying a packet for analysis is a huge bottleneck and very time-consuming.

What were our Results?

With kernel threading, cloning, and a high-end Intel SMP processor, we can make use of 16 CPU’s doing packet analysis at the same time and we now have attained speeds close to our 1 gigabit target.

When we developed our bandwidth shaping technology in 2003/2004, we leveraged technology innovation to create a superior bandwidth control appliance (read our NetEqualizer Story).  With the NetGladiator IPS, we have once again leveraged technology innovation to enable us to provide an intrusion prevention system at a very compelling price (register to get our price list), hence our customer’s remark about great speed for the price.

What other benefits does our low cost, high-speed layer 7 engine allow for? Is it just for IPS?

The sky is the limit here.  Any type of pattern you want to look at in real-time can now be done at one tenth (1/10th) the cost of the ASIC class of shapers.  Although we are not a fan of unauthorized intrusion into private data of the public Internet (we support Net Neutrality), there are hundreds of other uses which can be configured with our engine.

Some that we might consider in the future include:

– Spam filtering
– Unwanted protocols in your business
– Content blocking
– Keyword spotting

If you are interested in testing and experimenting in any of these areas with our raw technology, feel free to contact us ips@netgladiator.net.

%d bloggers like this: