Your heard it here first, our prediction on how video will evolve to conserve bandwidth

May 8, 2013 — netequalizer

Editors Note:

I suspect somebody out there has already thought of this,  but in my quick internet search I could not find any references to this specific idea, so I am takaing journalistic first  claim unofficial first rights to this idea.

The best example I think of to exemplify efficiency in video, are the old style cartoons,  such as the parody of South Park. If you ever watch south park animation,  the production quality  is done deliberately cheesy, very few moving parts with fixed backgrounds. In the South Park case, the intention was obviously not to save production costs.  The cheap animation is part of the comedy. That was not always the case,  the evolution of this sort of stop animation cartoon was from the early days  before computer animation took over the work of human artists working frame by frame. The fewer moving parts in a scene, the less work for the animator.  They could re-use existing drawings of a figure and just change the orientation of the mouth in perhaps three positions to animate talking.

Modern video compression tries to take advantage of some of the inherit static data from image to image , such that, each new frame is transmitted with less information.  At best, this is a hit or miss proposition.  There are likely many frivolous moving parts in a back ground that perhaps on the small screen of hand held device are not necessary.

My prediction is we will soon see a collaboration between production of video and Internet transport providers that allows for the average small device video production to have a much smaller footprint in transit.

Some of the basics of this technique would involve.

1) deliberately blurring or sending a background separate from the action. Think of a wide shot of break away lay-up in a basketball game. All you really need to see is the player and the basket in the frame the brain is going to ignore background details such as the crowd, they might as well be static character animations, especially on the scale of the screen of your Iphone not the same experience as your 56 inch HD flat screen.

2) Many of the videos in circulation the internet are news casts of a talking head giving the latest headlines. If you wanted to be extreme, you could  make the production such that the head is  tiny and animate it like a south park character,  this will take a much smaller footprint but technically still be video, and it would be much more like to play through without pausing.

3) The content sender can actually send a different production of the same video for low-bandwidth clients.

Note the reason why the production side of the house must get involved with the compression and delivery side of video is that the compression engines can only make assumptions on what is important and what is not, when removing information (pixels) from a video.

With a smart production engine geared toward the Internet, there is big savings here. Video is busting out all over the Internet and conserving from a production side only makes sense if you want to get your content deployed and viewed everywhere .

The security industry also does something similar taking advantage with fixed cameras on fixed backgrounds.

Related How much YouTube can the Internet Handle

Related Out of the box ideas on how to speed up your Internet

Blog dedicated to video compression, Euclid Discoveries.

 

 

Posted in Bandwidth Management, Educational Articles, Research. Tags: , , , , . 1 Comment »

Five Tips to Control Encrypted Traffic on Your Network

April 23, 2013 — netequalizer

Editors Note:

Our intent with our tips is to exemplify some of the impracticalities involved with “brute force” shaping of encrypted traffic, and to offer some alternatives.

1) Insert Pre-Encryption software at each end node on your network.

This technique requires a special a custom APP that would need to be installed on Iphones, Ipads, and the laptops of end users. The app is designed  to relay all data to a centralized shaping device in an un-encrypted format.

  •   assumes that the a centralized  IT department has the authority to require special software on all devices using the network. It would not be feasible for environments where end users freely use their own equipment.

ssltraffic

2) Use a sniffer traffic shaper that can decrypt the traffic on the fly.

  • The older 40 bit encryption codes could be hacked by a computer in about a one week, the newer 128 bit encryption codes would require the computer to run longer than the age of the Universe.

3) Just drop encrypted traffic, don’t allow it, forcing users to turn off SSL on their browsers.   Note: A traffic shaper, can spot encrypted traffic, it  just can’t tell you specifically what it is by content.

  • Seems rather draconian to block secure private transmissions, however the need to encrypt traffic over the Internet is vastly overblown. It is actually extremely unlikely for a personal information or credit card to get stolen in transit , but that is another subject
  • Really not practical where you have autonomous or public users, it will cause confusion at best, a revolt at worst.

4) Perhaps re-think what you are trying to accomplish.   There are more heuristic approaches to managing traffic which are immune to encryption.  Please feel free to contact us for more details on a heuristic approach to shaping encrypted traffic.

5) Charge a premium for encrypted traffic.  This would be more practical than blocking encrypted traffic, and would perhaps offset some of the costs for associate with the  overuse of p2p encrypted traffic.

Posted in Bandwidth Management. Tags: , , , , , , , , . Leave a Comment »

Imagine Unlimited Bandwidth

March 27, 2013 — netequalizer

By Art Reisman – CTO – www.netequalizer.com

Art Reisman CTO www.netequalizer.com

I was feeling a bit idealistic today about the future of bandwidth, so I jotted these words down. I hope it brightens your day

Imagine there’s no congestion
 It’s easy if you try
No hidden fees surprise us
Above us high speed guy
Imagine all providers, giving bandwidth away

Imagine there’s no Quota’s
It isn’t hard to use
 No killer apps that die for
A lack of bandwidth too
Imagine all the gamers living layer 7 free

You may say, I’m a streamer
But I’m just gonna download one
I hope some day you’ll join us
And your speed concerns will be done

Posted in Bandwidth Management, Business Advice, Commentary and Editorials. Tags: , , . Leave a Comment »

The Wireless Density Problem

March 21, 2013 — netequalizer

Recently, we have been involved in several projects where an IT consulting company has attempted to bring public wireless service into a high density arena. So far, the jury is out on how effective these service offerings have fared.

The motivation for such a project is driven by several factors.

1) Most standard cellular 4G data coverage is generally not adequate to handle 20,000 people with iPhones in a packed arena. I am sure the larger carriers are also feverishly working on a solution, but I have no inside information as to their approach nor chance of success.

Note: I’d be interested to learn about any arenas with great coverage?

2) Venue operators have customers that expect to be able to use their wireless devices during the course of a game to check stats, send pictures, etc.

3) Public frequency, wireless controllers, and access points are getting smarter rather quickly. Even though I have not seen clear success in these extremely high densities, free wireless solutions are gaining momentum.

We are actually doing a trial at a major sports venue in the coming weeks. From the perspective of the NetEqualizer, we are invited along to keep the  primary 1GB Internet pipe feeding the entire arena from going down. To date we have not been asked to referee the mayhem of access point regional gridlock and congestion in an arena setting, mostly because of of our price point and cost to deploy at each radio.

Why do these high density roll outs fail to meet expectation?

It seems, that 20+ thousand people in a small arena transmitting and receiving data over public frequencies really sucks for access points. The best way to picture this chaos would be to imagine listening to a million crickets on a warm summer night and trying to pick out the cadence of a single insect. Yes you might be able to single out a cricket  if it landed on your nose, but in a large arena not everybody can be next to an access point. The echoes from all the transmissions coming in to the radios in these high densities are unprecedented. Even with an initial success we see problems build as usage up take rises.  If you build it they will come! Typically what happens is that only a small percentage of attendees login to the wireless offering on the initial trial. The early success is tempered as usage doubles and doubles again eventually overwhelming the radios and their controllers.

My surprising conclusion

My prediction is that in the near future, we will start to see little plug in stations in high density venues. These stations will be compatible with next generation wireless devices, thus serving up data to your seat. You may scoff, but I am already hearing rumbles from many of our cutting edge high density housing internet providers on this issue. Due to wireless technology limitations they plan to keep their wired portals in their buildings, even in areas where they have spent heavily on wireless coverage.

Related Articles:

Siradel.com radio coverage

Addressing issues of wireless data coverage.

How to speed up access on your Iphone

Posted in Bandwidth Management. Tags: , , , , , , . 1 Comment »

How Much Bandwidth Do You Really Need?

February 17, 2013 — netequalizer

By Art Reisman – CTO – www.netequalizer.com

Art Reisman CTO www.netequalizer.com

When it comes to how much money to spend on the Internet, there seems to be this underlying feeling of guilt with everybody I talk to. From ISPs, to libraries or multinational corporations, they all have a feeling of bandwidth inadequacy. It is very similar to the guilt I used to feel back in College when I would skip my studies for some social activity (drinking). Only now it applies to bandwidth contention ratios. Everybody wants to know how they compare with the industry average in their sector. Are they spending on bandwidth appropriately, and if not, are they hurting their institution, will they become second-rate?

To ease the pain, I was hoping to put a together a nice chart on industry standard recommendations, validating that your bandwidth consumption was normal, and I just can’t bring myself to do it quite yet. There is this elephant in the room that we must contend with. So before I make up a nice chart on recommendations, a more relevant question is… how bad do you want your video service to be?

Your choices are:

  1. bad
  2. crappy
  3. downright awful

Although my answer may seem a bit sarcastic, there is a truth behind these choices. I sense that much of the guilt of our customers trying to provision bandwidth is based on the belief that somebody out there has enough bandwidth to reach some form of video Shangri-La; like playground children bragging about their father’s professions, claims of video ecstasy are somewhat exaggerated.

With the advent of video, it is unlikely any amount of bandwidth will ever outrun the demand; yes, there are some tricks with caching and cable on demand services, but that is a whole different article. The common trap with bandwidth upgrades is that there is a false sense of accomplishment experienced before actual video use picks up. If you go from a network where nobody is running video (because it just doesn’t work at all), and then you increase your bandwidth by a factor of 10, you will get a temporary reprieve where video seems reliable, but this will tempt your users to adopt it as part of their daily routine. In reality you are most likely not even close to meeting the potential end-game demand, and 3 months later you are likely facing another bandwidth upgrade with unhappy users.

To understand the video black hole, it helps to compare the potential demand curve pre and post video.

A  quality VOIP call, which used to be the measuring stick for decent Internet service runs about 54kbs. A quality  HD video stream can easily consume about 40 times that amount. 

Yes, there are vendors that claim video can be delivered at 250kbs or less, but they are assuming tiny little stop action screens.

Couple this tremendous increase in video stream size with a higher percentage of users that will ultimately want video, and you would need an upgrade of perhaps 60 times your pre-video bandwidth levels to meet the final demand. Some of our customers, with big budgets or government subsidized backbones, are getting close but, most go on a honeymoon with an upgrade of 10 times their bandwidth, only to end up asking the question, how much bandwidth do I really need?

So what is an acceptable contention ratio?

  • Typically in an urban area right now we are seeing anywhere from 200 to 400 users sharing 100 megabits.
  • In a rural area double that rati0 – 400 to 800 sharing 100 megabits.
  • In the smaller cities of Europe ratios drop to 100 people or less sharing 100 megabits.
  • And in remote areas served by satellite we see 40 to 50 sharing 2 megabits or less.
Posted in Bandwidth Management, Educational Articles, Research. Tags: , , , , , , , , , , , . 4 Comments »

A Brief History of Peer to Peer File Sharing and the Attempts to Block It

February 8, 2013 — netequalizer

By Art Reisman

The following history is based on my notes and observations as both a user of peer to peer, and as a network engineer tasked with cleaning  it up.

Round One, Napster, Centralized Server, Circa 2002

Napster was a centralized service, unlike the peer to peer behemoths of today there was never any question of where the copyrighted material was being stored and pirated from. Even though Napster did not condone pirated music and movies on their site, the courts decided by allowing copyrighted material to exist on their servers, they were in violation of copyright law. Napster’s days of free love were soon over.

From an historic perspective the importance of the decision to force the shut down of Napster was that it gave rise to a whole new breed of p2p applications. We detailed this phenomenon in our 2008 article.

Round Two, Mega-Upload  Shutdown, Centralized Server, 2012

We again saw a doubling down on p2p client sites (they expanded) when the Mega-Upload site, a centralized sharing site, was shutdown back in Jan 2012.

“On the legal side, the recent widely publicized MegaUpload takedown refocused attention on less centralized forms of file sharing (i.e. P2P). Similarly, improvements in P2P technology coupled with a growth in file sharing file size from content like Blue-Ray video also lead many users to revisit P2P.”

Read the full article from deepfield.net

The shut down of Mega-Upload had a personal effect on me as I had used it to distribute a 30 minute account from a 92-year-old WWII vet where he recalled, in oral detail, his experience of surviving a German prison camp.

Blocking by Signature, Alias Layer 7 Shaping, Alias Deep packet inspection. Late 1990′s till present

Initially, the shining star savior in the forefront against spotting illegal content on your network, this technology can be expensive and fail miserably in the face of newer encrypted p2p applications. It also can get quite expensive to keep up with the ever changing application signatures, and yet it is still often the first line of defense attempted by ISPs.

We covered this topic in detail, in our recent article,  Layer 7 Shaping Dying With SSL.

Blocking by Website

Blocking the source sites where users download their p2p clients is still possible. We see this method applied at mostly private secondary schools, where content blocking is an accepted practice. This method does not work for computers and devices that already have p2p clients. Once loaded, p2p files can come from anywhere and there is no centralized site to block.

Blocking Uninitiated Requests. Circa Mid-2000

The idea behind this method is to prevent your Network from serving up any content what so ever! Sounds a bit harsh, but the average Internet consumer rarely, if ever, hosts anything intended for public consumption. Yes at one time, during the early stages of the Internet, my geek friends would set up home pages similar to what everybody exposes on Facebook today. Now, with the advent hosting sites, there is just no reason for a user to host content locally, and thus, no need to allow access from the outside. Most firewalls have a setting to disallow uninitiated requests into your network (obviously with an exemption for your publicly facing servers).

We actually have an advanced version of this feature in our NetGladiator security device. We watch each IP address on your internal network and take note of outgoing requests, nobody comes in unless they were invited. For example, if we see a user on the Network make a request to a Yahoo Server , we expect a response to come back from a Yahoo server; however if we see a Yahoo server contact a user on your network without a pending request, we block that incoming request. In the world of p2p this should prevent an outside client from requesting a receiving a copyrighted file hosted on your network, after all no p2p client is going to randomly send out invites to outside servers or would they?

I spent a few hours researching this subject, and here is what I found (this may need further citations). It turns out that p2p distribution may be a bit more sophisticated and has ways to get around the block uninitiated query firewall technique.

P2P networks such as Pirate Bay use a directory service of super nodes to keep track of what content peers have and where to find them. When you load up your p2p client for the first time, it just needs to find one super node to get connected, from there it can start searching for available files.

Note: You would think that if these super nodes were aiding and abetting in illegal content that the RIAA could just shut them down like they did Napster. There are two issues with this assumption:
1) The super nodes do not necessarily host content, hence they are not violating any copyright laws. They simply coordinate the network in the same way DNS service keep track of URL names and were to find servers.
2) The super nodes are not hosted by Pirate Bay, they are basically commandeered from their network of users, who unwittingly or unknowingly agree to perform this directory service when clicking the license agreement that nobody ever reads.

From my research I have talked to network administrators that claim despite blocking uninitiated outside requests on their firewalls, they still get RIAA notices. How can this be?

There are only two ways this can happen.

1) The RIAA is taking liberty to simply accuse a network of illegal content based on the directory listings of a super node. In other words if they find a directory on a super node pointing to copyrighted files on your network, that might be information enough to accuse you.

2) More likely, and much more complex, is that the Super nodes are brokering the transaction as a condition of being connected. Basically this means that when a p2p client within your network, contacts a super node for information, the super node directs the client to send data to a third-party client on another network. Thus the send of information from the inside of your network looks to the firewall as if it was initiated from within. You may have to think about this, but it makes sense.

Behavior based thwarting of p2p. Circa 2004 – NetEqualizer

Behavior-based shaping relies on spotting the unique footprint of a client sending and receiving p2p applications. From our experience, these clients just do not know how to lay low and stay under the radar. It’s like the criminal smuggling drugs doing 100 MPH on the highway, they just can’t help themselves. Part of the p2p methodology is to find as many sources of files as possible, and then, download from all sources simultaneously. Combine this behavior with the fact that most p2p consumers are trying to build up a library of content, and thus initiating many file requests, and you get a behavior footprint that can easily be spotted. By spotting this behavior and making life miserable for these users, you can achieve self compliance on your network.

Read a smarter way to block p2p traffic.

Blocking the RIAA probing servers

If you know where the RIAA is probing from you can deny all traffic to their probes and thus prevent the probe of files on your network, and ensuing nasty letters to desist.

Posted in Bandwidth Management, Commentary and Editorials. Tags: , , , , , , , , , , , . 1 Comment »

Alternatives to Bandwidth Addiction

January 27, 2013 — netequalizer

By Art Reisman

CTO – http://www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Bandwidth providers are organized to sell bandwidth. In the face of bandwidth congestion, their fall back position is always to sell more bandwidth, never to slow consumption. Would a crack dealer send their clients to a treatment program?

For example, I have had hundreds of encounters with people at bandwidth resellers; all of our exchanges have been courteous and upbeat, and yet a vendor relationship rarely develops. Whether they are executives, account managers, or front-line technicians, the only time they call us is as a last resort to save an account, and for several good reasons.

1) It is much easier, conceptually, to sell a bandwidth upgrade rather than a piece of equipment.

2) Bandwidth contracts bring recurring revenue.

3) Providers can lock in a bandwidth contract, investors like contracts that guarantee revenue.

4) There is very little overhead to maintain a leased bandwidth line once up and running.

5) And as I eluded to before, would a crack dealer send a client to rehab?

6) Commercial bandwidth infrastructure costs have come down in the last several years.

7) Bandwidth upgrades are very often the most viable and easiest path to relieve a congested Internet connection.

Bandwidth optimization companies exist because at some point customers realize they cannot outrun their consumption. Believe it or not, the limiting factor to Internet access speed is not always the pure cost of raw bandwidth, enterprise infrastructure can be the limiting factor. Switches, routers, cabling, access points and back-hauls all have a price tag to upgrade, and sometimes it is easier to scale back on frivolous consumption.

The ROI of optimization is something your provider may not want you know.

The next time you consider a bandwidth upgrade at the bequest of your provider, you might want to look into some simple ways to optimize your consumption. You may not be able to fully arrest your increased demand with an optimizer, but realistically you can slow growth rate from a typical unchecked 20 percent a year to a more manageable 5 percent a year. With an optimization solution in place, your doubling time for bandwidth demand can easily reduce down from about 3.5 years to 15 years, which translates to huge cost savings.

Note: Companies such as level 3 offer optimization solutions, but with all do respect, I doubt those business units are exciting stock holders with revenue. My guess is they are a break even proposition; however I’d be glad to eat crow if I am wrong, I am purely speculating.  Sometimes companies are able to sell adjunct services at a nice profit.

Related NY times op-ed on bandwidth addiction

Posted in Bandwidth Management. Tags: , , , , , , , . 2 Comments »

The Voice Report Telecom Junkies Interview: “Bandwidth Battles: A New Approach”

January 23, 2013 — netequalizer

Linfield College logoListen in on a conversation with Andrew Wolf, telecom manager and NetEqualizer customer from Linfield College, and Art Reisman, CTO APconnections, as they spoke to George David, president of CCMI and publisher of TheVoiceReport.

Andrew switched from a Packeteer to a NetEqualizer in mid-2011.  In this interview Andrew talks about how the NetEqualizer has not only reduced Linfield College’s network congestion, but also has saved him both ongoing labor costs (no babysitting the solution or adding policies) and upfront costs on the hardware itself.

Listen the to broadcast: Bandwidth Battles: A New Approach
From TheVoiceReport Telecom Junkies, aired on 4/5/2012 | Length 12:16

College & University Guide

College & University Guide

Telecom manager Andrew Wolf at Linfield College had a problem – one just about all communications pros face or will face: huge file downloads were chewing up precious bandwidth and dragging down network performance. Plenty of traditional fixes were available, but the cost and staff to manage the apps were serious obstacles. Then Andrew landed on a unique “bandwidth behavior” approach from Art Reisman at NetEqualizer. End result – great performance at much lower costs, a real win-win. Get all the details in this latest episode of Telecom Junkies.

Want to learn more? See how others have benefited from NetEqualizer.  Read our NetEqualizer College & University testimonials.  Download our College & University Guide.

Posted in Bandwidth Management, Testimonials. Tags: , . Leave a Comment »

Check List for Integrating Active Directory to Your Bandwidth Controller

January 19, 2013 — netequalizer

By Art Reisman, CTO, www.netequalizer.com

Art Reisman CTO www.netequalizer.com

The problem statement: You have in place an authentication service such as Radius, LDAP, or Active Directory, and now you want to implement some form of class of service per customer. For example, data usage limits (quotas) or bandwidth speed restriction per user. To do so, you’ll need to integrate your authentication device with an  enforcement device, typically a bandwidth controller.

There are products out there such as Nomadix that do both (authentication and rate limiting),  but most authentication devices are not turn-key when it comes to a mechanism to set rate limits.

Your options are:

1) You can haggle your way through various forums that give advice on setting rate limits with AD,

2) Or you can embark on a software integration project using a consultant to accomplish your bandwidth restrictions.

In an effort to help customers appreciate and understand what goes into such an integration, I have shared notes that I have used as starting point when synchronizing our NetEqualizer with Radius.

1) Start by developing (borrowing if you can) a generic abstract interface (middle ware) that is not specific to Active Dircectory, LDAP or Radius. Keep it clean and basic so as not to tie your solution to any specific authentication server.  The investment in a middle ware interface is well worth the upfront cost.  By using a middle layer you will avoid a messy divorce of your authentication system from your bandwidth controller should the need arise.

2) Chances are your bandwidth controller speaks IP, and your AD device speaks user name. So you’ll need to understand how your AD can extract IP addresses from user names and send them down to your bandwidth controller.

3) Your bandwidth controller will need a list of IP’s or MAC addresses , and their committed bandwidth rates. It will need to get this information from your authentication database.

5) On a cold start, you’ll need to make bandwidth controller aware of all active users, and perhaps during the initial synchronization, you may want to pace yourself so as to not bog down your authentication controller with a million requests on start-up.

6) Once the bandwidth controller has an initial list of users on board, you’ll need to have a back ground re-synch (audit) mechanism to make sure all the rate limits and associated IP addresses are current.

7) What to do if the bandwidth controller senses traffic from an IP that it is unaware of? You’ll need a default guest rate limit of some kind for unknown IP addresses. Perhaps you’ll want the bandwidth controller to deny service to unknown IPs?

8) Don’t forget to put a timeout on requests from the bandwidth controller to the authentication device.

Posted in Bandwidth Management. Tags: , , , , , . Leave a Comment »

Bandwidth Control from the Public Side of a NAT Router, is it Possible?

January 8, 2013 — netequalizer

We have done some significant work in our upcoming release with respect to managing network traffic from the outside of private network segments.

The bottom line is we can now accomplish sophisticated bandwidth optimizations for segments of large networks hidden behind the NAT routers.

The problem:

One basic problem with a generic bandwidth controller, is that they typically treat all users behind a NAT router as one user.

When using NAT, a router takes one public IP and divides it up such that up to several thousand users on the private side of a network can share it. The most common reason for this, is that there are a limited number of public IPv4 addresses to hand out, so it is common for organizations and ISP’s to share the public IP’s that they own among many users.

When a router shares an IP with more than one user, it manipulates a special semi private part of the IP packet , called a “port”, to keep track of who’s data belongs to whom behind the router. The easiest way to visualize this is to think of a company with one public phone number and many private internal extensions on a PBX. In the case of this type of phone arrangement, all the employees share the public phone numbers for out side calls.

In the case of a Nat’d router, all the users behind the router share one public IP address. For the bandwidth controller sitting on the public side of the router, this can create issues, it can’t shape the individual traffic of each user because all their traffic appears as if it is coming from one IP address.

The obvious solution to this problem is to locate your bandwidth controller on the private side of the NAT router; but for a network with many NAT routers such as a large distributed wireless mesh network, the cost of extra bandwidth controllers becomes prohibitive.

Drum Roll: Enter NetEqualizer Super hero.

The Solution:

With our upcoming release we have made changes to essentially reverse engineer the NAT Port addressing scheme inside our bandwidth controller, even when located on the Internet side of the router, we can now, apply our equalizing shaping techniques to individual user streams with much more accuracy than before.

We do this by looking at the unique port mapping for each stream coming out of your router. So, if for example, two users in your mesh network, are accessing Facebook, we will treat those users bandwidth and allocations independently in our congestion control. The Benefit from these techniques is the ability to provide QoS for a Face-to-Face chat session while at the same time limiting the video to Facebook component.

Posted in Bandwidth Management. Tags: , , , . 2 Comments »

You Must Think Outside the Box to Bring QoS to the Cloud and Wireless Mesh Networks

December 10, 2012 — netequalizer

By Art Reisman
CTO – http://www.netequalizer.com

About 10 years ago, we had this idea for QoS across an Internet link. It was simple and elegant, and worked like a charm. Ten years later, as services spread out over the Internet cloud, our original techniques are more important than ever. You cannot provide QoS using TOS (diffserv) techniques over any public or semi public Internet link, but using our techniques we have proven the impossible is possible.

Why TOS bits don’t work over the Internet.

The main reason is that setting TOS bits are only effective when you control all sides of a conversation on a link, and this is not possible on most Internet links (think cloud computing and wireless mesh networks). For standard TOS services to work, you must control all the equipment in between the two end points. All it takes is one router in the path of a VoIP conversation to ignore a TOS bit, and its purpose becomes obsolete. Thus TOS bits for priority are really only practical inside a corporate LAN/WAN topology.

Look at the root cause of poor quality services and you will find alternative solutions.

Most people don’t realize the problem with congested VoIP, on any link, is due to the fact that their VoIP packets are getting crowded out by larger downloads and things like recreational video (this is also true for any interactive cloud access congestion). Often, the offending downloads are initiated by their own employees or users. A good behavior-based shaper will be able to favor VoIP streams over less essential data streams without any reliance on the sending party adhering to a TOS scheme.

How do we accomplish priority for VoIP?

We do this by monitoring all the streams on a link with one piece of equipment inserted anywhere in the congested link. In our current terminology, a stream consists of an IP (local), talking to another IP (remote Internet). When we see a large stream dominating the link, we step back and ask, is the link congested? Is that download crowding out other time-sensitive transactions such as VOIP? If the answer is yes to both questions, then we proactively take away some bandwidth from the offending stream. I know this sounds ridiculously simple, and does not seem plausible, but it works. It works very well and it works with just one device in the link irrespective of any other complex network engineering. It works with minimal set up. It works over MPLS links. I could go on and on, the only reason you have not heard of it is perhaps is that it goes against the grain of what most vendors are selling – and that is large orders for expensive high end routers using TOS bits.

Related article QoS over the Internet – is it possible?

Fast forward to our next release, how to provide QOS deep inside a cloud or mesh network where sending or receiving IP addresses are obfuscated.

Coming this winter we plan to improve upon our QoS techniques so we can drill down inside of Mesh and Cloud networks a bit better.

As the use of NAT, distributed across mesh networks, becomes more wide spread, and the bundling of services across cloud computing becomes more prevalent, one side effect has been that our stream based behavior shaping (QoS) is not as effective as it is when all IP addresses are visible (not masked behind a NAT/PAT device).

This is due to the fact that currently, we base our decision on a pair of IP’s talking to each other, but we do not consider the IP port numbers, and sometimes especially in a cloud or mesh network, services are trunked across a tunnel using the same IP. As these services get tunneled across a trunk, the data streams are bundled together using one common pair of IP’s and then the streams are broken out based on IP ports so they can be routed to their final destination. For example, in some cloud computing environments there is no way to differentiate the video stream within the tunnel coming from the cloud, from a smaller data access session. They can sometimes both be talking across the same set of IP’s to the cloud. In a normal open network we could slow the video (or in some cases give priority to it) by knowing the IP of the video server, and the IP of the receiving user,  but when the video server is buried within the tunnel sharing the IP’s of other services, our current equalizing (QOS techniques) become less effective.

Services within a tunnel, cloud, or mesh may be bundled using the same IPs, but they are often sorted out on different ports at the ends of the tunnel. With our new release coming this winter, we will start to look at streams as IP and port number, thus allowing for much greater resolution for QOS inside the Cloud and inside your mesh network. Stay tuned!

Posted in Bandwidth Management. Tags: , , , , , , , , , . Leave a Comment »

Will Bandwidth Shaping Ever Be Obsolete?

December 1, 2012 — netequalizer

By Art Reisman

CTO – www.netequalizer.com

I find public forums where universities openly share information about their bandwidth shaping policies an excellent source of information. Unlike commercial providers, these user groups have found technical collaboration is in their best interest, and they often openly discuss current trends in bandwidth control.

A recent university IT user group discussion thread kicked off with the following comment:

“We are in the process of trying to decide whether or not to upgrade or all together remove our packet shaper from our residence hall network.  My network engineers are confident we can accomplish rate limiting/shaping through use of our core equipment, but I am not convinced removing the appliance will turn out well.”

Notice that he is not talking about removing rate limits completely, just backing off from an expensive extra piece of packet shaping equipment and using the simpler rate limits available on his router.  The point of my reference to this discussion is not so much to discourse over the different approaches of rate limiting, but to emphasize, at this point in time, running wide-open without some sort of restriction is not even being considered.

Despite an 80 to 90 percent reduction in bulk bandwidth prices in the past few years, bandwidth is not quite yet cheap enough for an ISP to run wide-open. Will it ever be possible for an ISP to run wide-open without deliberately restricting their users?

The answer is not likely.

First of all, there seems to be no limit to the ways consumer devices and content providers will conspire to gobble bandwidth. The common assumption is that no matter what an ISP does to deliver higher speeds, consumer appetite will outstrip it.

Yes, an ISP can temporarily leap ahead of demand.

We do have a precedent from several years ago. In 2006, the University of Brighton in the UK was able to unplug our bandwidth shaper without issue. When I followed up with their IT director, he mentioned that their students’ total consumption was capped by the far end services of the Internet, and thus they did not hit their heads on the ceiling of the local pipes. Running without restriction, 10,000 students were not able to eat up their 1 gigabit pipe! I must caveat this experiment by saying that in the UK their university system had invested heavily in subsidized bandwidth and were far ahead of the average ISP curve for the times. Content services on the Internet for video were just not that widely used by students at the time. Such an experiment today would bring a pipe under a similar contention ratio to its knees in a few seconds. I suspect today one would need more or on the order of 15 to 25 gigabits to run wide open without contention-related problems.

It also seems that we are coming to the end of the line for bandwidth in the wireless world much more quickly than wired bandwidth.

It is unlikely consumers are going to carry cables around with their iPad’s and iPhones to plug into wall jacks any time soon. With the diminishing returns in investment for higher speeds on the wireless networks of the world, bandwidth control is the only way to keep order of some kind.

Lastly I do not expect bulk bandwidth prices to continue to fall at their present rate.

The last few years of falling prices are the result of a perfect storm of factors not likely to be repeated.

For these reasons, it is not likely that bandwidth control will be obsolete for at least another decade. I am sure we will be revisiting this issue in the next few years for an update.

Posted in Bandwidth Management, Commentary and Editorials, Wireless Internet. Tags: , , , , , , . 1 Comment »

Equalizing is the Silver Bullet for Quality of Service

November 30, 2012 — netequalizer

Silver Bullet (n.) – A simple and seemingly magical solution to a complex problem.

The amount of solutions available that have been developed to improve Quality of Service (QoS) for data traveling across a network (video, VoIP, etc.) are endless. Often, these tools appear to be simple, but seem to fall short in implementation:

Compression: Compressing files in transit helps reduce congestion by decreasing the amount of bandwidth a transfer requires. This appears to be a viable solution, but in practice, most of the large streams that tend to clog networks (high resolution media files, etc.) are already compressed. Thus, most networks won’t see much improvement in QoS when this method is used.

Layer 7 Inspection: Providing QoS to specific applications also sounds like a reasonable approach to the problem. However, most applications are increasingly utilizing encryption for transferring data, and thus determining the purpose of a network packet is a much harder problem. It also requires constant tweaking and updates to ensure the proper applications are given priority.

Type of Service: Each network packet has a flag as part of its payload that denotes its “type of service.” This flag was intended to help give QoS to packets based on their importance and purpose. This method, however, requires lots of custom router configurations and is not very reliable as far as who is able to set the flag, when, and why.

These solutions are analogous to the diet pill and weight loss products that inundate our lives on a daily basis. They are offering complex solutions to a simple problem:

Overweight? Buy this machine, watch these DVDs, take this pill.

When the real solution is:

Overweight? Eat better.

Simple solutions are what good engineering is all about, and it drives the entire philosophy behind Equalizing – the bandwidth control method implemented in our NetEqualizer. The truth is, you can accomplish 99% of your QoS needs on a fixed link SIMPLY by cranking down on the large streams of traffic. While the above approaches try to do this in various ways, nothing is easier and more hands-off than looking at the behavior of a connection relative to the available bandwidth, and subsequently throttling it as needed. No deep packet inspection, compression, or packet analysis required. No need to concern yourself with new Internet usage trends or the latest media file types. Just fair bandwidth, regardless of trunk size, for all of your users, at all times of day. When bandwidth is controlled, connection quality is allowed to be as good as possible for everyone!

Posted in Bandwidth Management. Tags: , , . Leave a Comment »

Internet User’s Bill of Rights

November 27, 2012 — netequalizer

This is the second article in our series. Our first was a Bill of Rights dictating the etiquette of software updates. We continue with a proposed Bill of Rights for consumers with respect to their Internet service.

1) Providers must divulge the contention ratio of their service.

At the core of all Internet service is a balancing act between the number of people that are sharing a resource and how much of that resource is available.

For example, a typical provider starts out with a big pipe of Internet access that is shared via exchange points with other large providers. They then subdivide this access out to their customers in ever smaller chunks — perhaps starting with a gigabit exchange point and then narrowing down to a 10 megabit local pipe that is shared with customers across a subdivision or area of town.

The speed you, the customer, can attain is limited to how many people might be sharing that 10 megabit local pipe at any one time. If you are promised one megabit service, it is likely that your provider would have you share your trunk with more than 10 subscribers and take advantage of the natural usage behavior, which assumes that not all users are active at one time.

The exact contention ratio will vary widely from area to area, but from experience, your provider will want to maximize the number of subscribers who can share the pipe, while minimizing service complaints due to a slow network. In some cases, I have seen as many as 1,000 subscribers sharing 10 megabits. This is a bit extreme, but even with a ratio as high as this, subscribers will average much faster speeds when compared to dial up.

2) Service speeds should be based on the amount of bandwidth available at the providers exchange point and NOT the last mile.

Even if your neighborhood (last mile) link remains clear, your provider’s connection can become saturated at its exchange point. The Internet is made up of different provider networks and backbones. If you send an e-mail to a friend who receives service from a company other than your provider, then your ISP must send that data on to another network at an exchange point. The speed of an exchange point is not infinite, but is dictated by the type of switching equipment. If the exchange point traffic exceeds the capacity of the switch or receiving carrier, then traffic will slow.

3) No preferential treatment to speed test sites.

It is possible for an ISP to give preferential treatment to individual speed test sites. Providers have all sorts of tools at their disposal to allow and disallow certain kinds of traffic. There should never be any preferential treatment to a speed test site.

4) No deliberate re-routing of traffic.

Another common tactic to save resources at the exchange points of a provider is to re-route file-sharing requests to stay within their network. For example, if you were using a common file-sharing application such as BitTorrent, and you were looking some non-copyrighted material, it would be in your best interest to contact resources all over the world to ensure the fastest download.

However, if your provider can keep you on their network, they can avoid clogging their exchange points. Since companies keep tabs on how much traffic they exchange in a balance sheet, making up for surpluses with cash, it is in their interest to keep traffic confined to their network, if possible.

5) Clearly disclose any time of day bandwidth restrictions.

The ability to increase bandwidth for a short period of time and then slow you down if you persist at downloading is another trick ISPs can use. Sometimes they call this burst speed, which can mean speeds being increased up to five megabits, and they make this sort of behavior look like a consumer benefit. Perhaps Internet usage will seem a bit faster, but it is really a marketing tool that allows ISPs to advertise higher connection speeds – even though these speeds can be sporadic and short-lived.

For example, you may only be able to attain five megabits at 12:00 a.m. on Tuesdays, or some other random unknown times. Your provider is likely just letting users have access to higher speeds at times of low usage. On the other hand, during busier times of day, it is rare that these higher speeds will be available.

There is now a consortium called M-Lab which has put together a sophisticated speed test site designed to give specific details on what your ISP is doing to your connection. See the article below for more information.

Related article Ten things your internet provider does not want you to know.

Related article On line shoppers bill of rights

Posted in Bandwidth Management, Bandwidth Monitoring, Commentary and Editorials. Tags: , , , . 4 Comments »

Layer 7 Application Shaping Dying with Increased SSL

November 5, 2012 — netequalizer

By Art Reisman
CTO – www.netequalizer.com

When you put a quorum of front line IT administrators  in a room, and an impromptu discussion break out, I become all ears. For example, last Monday, the discussion at our technical seminar at Washington University turned to the age-old subject of controlling P2P.

I was surprised to hear from several of our customers about just how difficult it has become to implement Layer 7 shaping. The new challenge stems from fact that SSL traffic cannot be decrypted and identified from a central bandwidth controller. Although we have known about this limitation for a long time, my sources tell me there has been a pick up in SSL adoption rates over the last several years. I don’t have exact numbers, but suffice it to say that SSL usage is way up.

A traditional Layer 7 shaper will report SSL traffic as “unknown.” A small amount of unknown traffic has always been considered tolerable, but now, with the pick up SSL traffic, rumor has it that some vendors are requiring a module on each end node to decrypt SSL pages. No matter what side of the Layer 7 debate you are on, this provision can be a legitimate show stopper for anybody providing public or semi-open Internet access, and here is why:

Imagine your ISP is requiring you to load a special module on your laptop or iPad to decrypt all your SSL information and send them the results? Obviously, this will not go over very well on a public Internet. This relegates Layer 7 technologies to networks where administrators have absolute control over all the end points in their network. I suppose this will not be a problem for private businesses, where recreational traffic is not allowed, and also in countries with extreme controls such as China and Iran, but for a public Internet providers in the free world,  whether it be student housing, a Library, or a municipal ISP, I don’t see any future in Layer 7 shaping.

Posted in Bandwidth Management, P2P blocking. Tags: , , , . 2 Comments »
« Older posts
Follow

Get every new post delivered to your Inbox.

Join 31 other followers

%d bloggers like this: