NetEqualizer News Blog

By Art Reisman

CTO – APconnections

Have you ever been on  a shared wireless network, in a Hotel , or Business, and noticed how your  connection can go from reasonable to completely unusable in a matter of seconds, and then cycle back to usable ?

The reason for this , is that once a network hits its bandwidth allocation, the providers router usually just starts dropping the excess packets. Intuitively, when your router is dropping packets, one would assume that the perceived slow down, per user, would be just a gradual shift slower.

What happens in reality is far worse…

1) Distant users get spiraling slower responses.

Martin Roth, a colleague of ours who founded one of the top performance analysis companies in the world, provided this explanation:

“Any device which is dropping packets “favors” streams with the shortest round trip time, because (according to the TCP protocol) the time after which a lost packet is recovered is depending on the round trip time. So when a company in Copenhagen/Denmark has a line to Australia and a line to Germany on the same internet router, and this router is discarding packets because of bandwidth limits/policing, the stream to Australia is getting much bigger “holes” per lost packet (up to 3 seconds) than the stream to Germany or another office in Copenhagen. This effect then increases when the TCP window size to Australia is reduced (because of the retransmissions), so there are fewer bytes per round trip and more holes between to round trips.”

In the screen shot above (courtesy of avenida.dk), the Bandwidth limit is 10 Mbit (= 1 Mbyte/s net traffic), so everything on top of that will get discarded. The problem is not the discards, this is standard TCP behaviour, but the connections that are forcefully closed because of the discards. After the peak in closed connections, there is a “dip” in bandwidth utilization, because we cut too many connections.

2) Once you hit a congestion point, where your router is forced to drop packets, overall congestion actually gets worse before it gets better.

When applications don’t get a response due to a dropped packet, instead of backing off and waiting, they tend to start sending re-tries, and this is why you may have noticed prolonged periods (3o seconds or more) of no service on a congested network. We call this the rolling brown out. Think of this situation as sort of a doubling down on bandwidth at the moment of congestion. Instead of easing into a full network and lightly bumping your head, all the devices demanding bandwidth ramp up their requests at precisely the moment when your network is congested, resulting in an explosion of packet dropping until everybody finally gives up.

How do you remedy outages caused by Congestion?

We have written extensively about solutions to prevent bottlenecks. Here is a quick summary of possible solutions

1) The most obvious being to increase the size of your link.

2) Enforce rate limits per user. The problem with this solution is that you can waste a good bit of bandwidth if the network is lightly loaded

3) Use something more sophisticated like a Netequalizer, a device that is designed to specifically counter the effects of congestion.

From Martin Roth of Avenida.dk

“With NetEqualizer we may get the same number of discards, but we get fewer connections closed, because we “kick” the few connections with the high bandwidth, so we do not get the “dip” in bandwidth utilization.

The graphs (above) were recorded using 1 second intervals, so here you can see the bandwidth is reached. In a standard SolarWinds graph with 10 minute averages the bandwidth utilization would be under 20% and the customer would not know they are hitting the limit.”

———————————————————————-

The excerpt below was a message from a reseller who had been struggling with congestion issues at a hotel, he tried basic rate limits on his router first. Rate Limits will buy you some time , but on an oversold network you can still hit the congestion point, and for this you need a smarter device.

“…NetEq delivered a 500% gain in available bandwidth by eliminating rate caps, possible through a mix of connection limits and Equalization.  Both are necessary.  The hotel went from 750 Kbit max per accesspoint (entire hotel lobby fights over 750Kbit; divided between who knows how many users) to 7Mbit or more available bandwidth for single users with heavy needs.

Are those rate limits on your router good enough?

I discovered a problem with my download speed while trying to recover my un-responsive iPad.  Apple’s solution required me attach my iPad to my Mac, and then to download a new iOS image from the Internet, through the Mac and onto the IPad.

Speed should have been no problem with my business class, 20 megabit Internet connection from a well-known provider, right?

So I thought.

When I started the iOS download, the little progress timer immediately registered 23 hours to go. Wow, that is long time to wait, and I needed my iPad for a trip the next morning.  I tried a couple of speed tests in parallel, and everything looked normal.  The question remained – where was the bottleneck on this iOS download?  Was it on Apple’s end or a problem with my provider?

Over the years I have learned that iOS  and Windows updates are the bane of many Internet Providers. They are constantly looking at ways to prevent them from gumming up their exchange points.  They will try to identify update traffic, either by using the source IP, or if that does not work, they can actually examine the download data to make a determination. In either case, once they have tagged it as an update, they will choose to slow it down to keep their exchange points clear during peak traffic hours.

To thwart their shaping and get my speed back up near 20 megabits as promised, I simply had to hide my intentions. This can be accomplished using any number of consumer grade VPN applications.

I turned on my  IPvanish, which automatically encrypts the data and original source of my iOS update. Once up and running with my VPN, my IOS update loaded in 23 minutes. A 60 fold speed increase from my previous attempt.

If you would like to read more, here are a couple of other posts about ISP’s throttling data:

There is something rotten in the state of online streaming.

How to get access to blocked Internet Sites.

Good luck!

This morning I read an article by Klint Finley about his experience with disabling Java Script.  I am about 8 hours into my experiment now, and here is what I have found so far.

The results were amazing for the on-line periodicals (traditional newspapers) that I like to browse through. Even with my 20 megabit Internet connection, some of these sites are just endless piles of garbage with advertisements and videos popping up, forcing screen refreshes, and making the content unreadable.  Some of them take so long to load, I just give up and get back to work. With Java script turned off, all that changed.   I have not tested the limits on this yet, but I was able to get through a couple of these sites clicking to various articles and my delays were about 1/10 of normal, which is a significant improvement.

On the downside I found some of  the web-based applications that I depend on to be nonfunctional.  Klint mentions issues with Google Docs, but it goes farther than that. My Google Calendar did not work and neither did my WordPress or Cisco Webex. What I am doing now on my MAC laptop is keeping two browsers active.  Firefox with Java Script disabled, and Safari with it enabled.  I feel that this is a good compromise and worth the effort of switching.

Editors Note: Turning off Java Script is only going to impact things that you launch from a traditional browser. The pre- loaded applications on your devices do not use Java Script.

Sunday night I decided I would finally try watching the Sopranos.  Amazon offers Sopranos content for $1.99 an episode, which saves me the hassle of getting a full year HBO subscription to get episodes.  First pass on my smart internet connected TV,  I could not get the Amazon stream to run at all, and so I reverted to watching it on my laptop.  It came up on the laptop, but the video was choppy and constantly breaking up, stalling etc.   In other words it was being throttled by Comcast.  Solution?

I just fired up my IPvanish which hides the source of the video from Comcast, and presto, I was able to watch the whole episode without an issue.   If you experience content streaming problems with your National ISP try using a VPN tunnel, it has worked for me quite well.

There are other posts about this practice.

There is something rotten in the state of online streaming.

How to get access to blocked Internet Sites.

Editors Note: I completely understand why they throttle content, and have covered the economics behind this before. I just don’t like the secrecy  and deception around it, hence I will continue to publish articles when I find it.

Art Reisman
CTO, http://www.netequalizer.com

When the consumer Internet came of age back in 1990, there was never any grand plan to insure a consistent speed from one point to another. Somewhere along the line, as the Internet went from an academic tool to an essential consumer device, providers in their effort to “out market” one another began to focus on speed as their primary differentiator. By definition, the Internet is a “best effort” corroboration between providers to move your data. No one provider can guarantee a consistent Internet speed for everything you do.  They only have control over their own physical lines, and even then, there are variables beyond their control (which I will address shortly).

Let’s take a look at the speed of wired networks common to most consumers, Cable and DSL.

The physical line into your house is generally what your cable or DSL provider is talking about when they advertise your Internet speed. Essentially, how fast is the link between the providers NOC and your house. Generally you will have a dedicated line for this, and so your speed on this last mile link does not vary.

The good news is that most consumers are more concerned with watching movies, video, listening to music, etc. than they are about pulling research data of some obscure server in Serbia. Given this reality, the Industry has gotten very smart, and popular content is not hosted at some distant server, but is usually distributed locally to each provider. The best example of this is Netflix. Your Netflix content is most likely coming from a server hosted a few miles from your house in your providers NOC, and not from some grand Netflix central location.

Why is Netflix data hosted locally ?

The dirty industry secret is that your provider pays a fee when you go off their network for data. There are also potential capacity problems when you go off their network.  Is this a bad thing? No not really, it is just a matter of efficiency. We see similar practices in other product distribution models. You don’t drive to New York to pick up a toaster, there is usually one waiting for you at your nearest discount store. For the some of the same reasons, that you don’t go to New York to pick up a toaster, your provider tries to host your digital data locally when possible.

What does this mean for your Internet Speed?

It means that when you retrieve content that your provider hosts locally you are likely going to get your advertised speed. This also holds true for some speed test sites, if they are hosted within your providers network they are going to register a constantly higher speed.

What happens to your Internet speed when you go off your providers network? 

There are several factors that will effect your speed.

The main governing factor affecting speed is the capacity the of your providers exchange point.  This is a switching point where your provider exchanges data with other networks.  Depending on how much investment your provider put into this infrastructure this switching point can back up when there is more data being moved than it has capacity to handle. When this happens you get gridlock at the exchange point, and  your Internet speed can plummet.  Gridlock is always a real possibility because your provider just cannot anticipate all the content you are retrieving and sometimes it is not hosted locally.

What does my provider to to alleviate gridlock not their exchange point?

Some providers will actually lower your Internet speed when you are crossing an exchange point.  Or if their circuits are overloaded in general. I experienced this effect which I described in detail a few months ago when I was updating my IPAD.

After the exchange point the speed at which you get your data external to your providers network depends on the whims of every provider and back bone along the route. That obscure research paper from that server in Serbia , may have to make multiple hops to get out of Serbia and then onto some international back bone, and finally to your providers exchange point. There is no way anyone can anticipate at what rate this data will arrive.

How can I run a speed test that better reflects my speed out to the real Internet, by passing locally hosted speed test servers?

A few years ago we ran into this tool set that deliberately tries to retrieve all kinds of remote data to measure your true internet speed. You can also search out files hosted on obscure servers and try to download them.  Perhaps I’ll run a follow up article documenting some of my experiences.