Why is Your Internet Connection So Slow?


By Art Reisman

CTO – APconnections

Have you ever been on  a shared wireless network, in a Hotel , or Business, and noticed how your  connection can go from reasonable to completely unusable in a matter of seconds, and then cycle back to usable ?

The reason for this , is that once a network hits its bandwidth allocation, the providers router usually just starts dropping the excess packets. Intuitively, when your router is dropping packets, one would assume that the perceived slow down, per user, would be just a gradual shift slower.

What happens in reality is far worse…

1) Distant users get spiraling slower responses.

Martin Roth, a colleague of ours who founded one of the top performance analysis companies in the world, provided this explanation:

“Any device which is dropping packets “favors” streams with the shortest round trip time, because (according to the TCP protocol) the time after which a lost packet is recovered is depending on the round trip time. So when a company in Copenhagen/Denmark has a line to Australia and a line to Germany on the same internet router, and this router is discarding packets because of bandwidth limits/policing, the stream to Australia is getting much bigger “holes” per lost packet (up to 3 seconds) than the stream to Germany or another office in Copenhagen. This effect then increases when the TCP window size to Australia is reduced (because of the retransmissions), so there are fewer bytes per round trip and more holes between to round trips.”

In the screen shot above (courtesy of avenida.dk), the Bandwidth limit is 10 Mbit (= 1 Mbyte/s net traffic), so everything on top of that will get discarded. The problem is not the discards, this is standard TCP behaviour, but the connections that are forcefully closed because of the discards. After the peak in closed connections, there is a “dip” in bandwidth utilization, because we cut too many connections.

2) Once you hit a congestion point, where your router is forced to drop packets, overall congestion actually gets worse before it gets better.

When applications don’t get a response due to a dropped packet, instead of backing off and waiting, they tend to start sending re-tries, and this is why you may have noticed prolonged periods (3o seconds or more) of no service on a congested network. We call this the rolling brown out. Think of this situation as sort of a doubling down on bandwidth at the moment of congestion. Instead of easing into a full network and lightly bumping your head, all the devices demanding bandwidth ramp up their requests at precisely the moment when your network is congested, resulting in an explosion of packet dropping until everybody finally gives up.

How do you remedy outages caused by Congestion?

We have written extensively about solutions to prevent bottlenecks. Here is a quick summary of possible solutions

1) The most obvious being to increase the size of your link.

2) Enforce rate limits per user. The problem with this solution is that you can waste a good bit of bandwidth if the network is lightly loaded

3) Use something more sophisticated like a Netequalizer, a device that is designed to specifically counter the effects of congestion.

From Martin Roth of Avenida.dk

“With NetEqualizer we may get the same number of discards, but we get fewer connections closed, because we “kick” the few connections with the high bandwidth, so we do not get the “dip” in bandwidth utilization.

The graphs (above) were recorded using 1 second intervals, so here you can see the bandwidth is reached. In a standard SolarWinds graph with 10 minute averages the bandwidth utilization would be under 20% and the customer would not know they are hitting the limit.”

———————————————————————-

The excerpt below was a message from a reseller who had been struggling with congestion issues at a hotel, he tried basic rate limits on his router first. Rate Limits will buy you some time , but on an oversold network you can still hit the congestion point, and for this you need a smarter device.

“…NetEq delivered a 500% gain in available bandwidth by eliminating rate caps, possible through a mix of connection limits and Equalization.  Both are necessary.  The hotel went from 750 Kbit max per accesspoint (entire hotel lobby fights over 750Kbit; divided between who knows how many users) to 7Mbit or more available bandwidth for single users with heavy needs.

The ability to fully load the pipe, then reach out and instantly take back up to a third of it for an immediate need like a speedtest was also really eye-opening.  The pipe is already maxed out, but there is always a third of it that can be immediately cleared in time to perform something new and high-priority like a speed test.”
 
Rate Caps: nobody ever gets a fast Internet connection.
Equalized: the pipe stays as full as possible, yet anybody with a business-class need gets served a major portion of the pipe on demand. “
– Ben Whitaker – jetsetnetworks.com

Are those rate limits on your router good enough?

How to Speed Up Windows/Apple Updates


I discovered a problem with my download speed while trying to recover my un-responsive iPad.  Apple’s solution required me attach my iPad to my Mac, and then to download a new iOS image from the Internet, through the Mac and onto the IPad.

Speed should have been no problem with my business class, 20 megabit Internet connection from a well-known provider, right?

So I thought.

When I started the iOS download, the little progress timer immediately registered 23 hours to go. Wow, that is long time to wait, and I needed my iPad for a trip the next morning.  I tried a couple of speed tests in parallel, and everything looked normal.  The question remained – where was the bottleneck on this iOS download?  Was it on Apple’s end or a problem with my provider?

Over the years I have learned that iOS  and Windows updates are the bane of many Internet Providers. They are constantly looking at ways to prevent them from gumming up their exchange points.  They will try to identify update traffic, either by using the source IP, or if that does not work, they can actually examine the download data to make a determination. In either case, once they have tagged it as an update, they will choose to slow it down to keep their exchange points clear during peak traffic hours.

To thwart their shaping and get my speed back up near 20 megabits as promised, I simply had to hide my intentions. This can be accomplished using any number of consumer grade VPN applications.

I turned on my  IPvanish, which automatically encrypts the data and original source of my iOS update. Once up and running with my VPN, my IOS update loaded in 23 minutes. A 60 fold speed increase from my previous attempt.

If you would like to read more, here are a couple of other posts about ISP’s throttling data:

There is something rotten in the state of online streaming.

How to get access to blocked Internet Sites.

Good luck!

Seven Must Know Network Troubleshooting Tips


Screen Shot 2016-04-05 at 10.07.59 AM.png

By Art Reisman

CTO, APconnections
www.netequalizer.com

To get started you’ll need to get ahold of two key software tools: 1) Ping Tool and 2) a Network Scan Tool, both which I describe in more detail below.  And for advanced analysis (experts only), I will then show you how you can use a bandwidth shaper/sniffer if needed.

Ping Tool

Ping is a great tool to determine what your network responsiveness is (in milliseconds), identified by trying to get a response from a typical website. If you do not already know how to use Ping on your device there are hundreds of references to Ping and how to use it.  Simply google “how to use ping ” on  your favorite device or computer to learn how to use it.

For example, I found these instructions for my MAC; and there are similar instructions for Windows, iPhone, Linux, Android, etc.

  1. Open Network Utility (located inside Applications > Utilities).
  2. Click Ping.
  3. Fill out the “Enter the network that you want to ping” field. You can enter the IP address or a web URL. For example, enter http://www.bbc.co.uk/iplayer to test the ping with that website.
  4. Click Ping.

Network Scan Tool

There are a variety of network SCAN tools/apps available for just about any consumer device or computer.  The decent ones will cost a few dollars, but I have never regretted purchasing one.  I use mine often for very common home and business network issues as I will detail in the tips below. Be sure and use the term “network scan tool” when searching, so you do not get confusing results about unrelated document scanning tools.

Once you get your scan tool installed, test it out by selecting Network Scan. Here is the output from my MAC scan tool.  I will be referencing this output later in the article.

Network Scan Output
Screen Shot 2016-04-05 at 5.33.19 AM

 

Tip #1: Using Ping to see if you are really connected to your Network

I like to open a window on my laptop and keep Ping going all day, it looks like this:

yahoo.com Ping  Output

Screen Shot 2016-04-05 at 8.25.10 AM

Amazingly, seemingly on cue, I lost connectivity to my Internet while I was running the tool for the screen capture above, and no, it was not planned or contrived.  I kicked off my ping by contacting http://www.yahoo.com (type in “ping http://www.yahoo.com”), a public website. And you can see that my round-trip time was around 40 milliseconds before it went dead. Any ping results under 100 milliseconds are normal.

 

Tip #2: How to Deal with Slow Ping Times

In the case above, my Internet Connection just went dead; it came back a minute or so later, and was most likely not related to anything local on my network.

If you start to see missed pings or slow Ping Times above 100 milliseconds, it is most likely due to congestion on your network.  To improve your response times, try turning off other devices/applications and see if that helps.  Even your TV video can suck down a good chunk of bandwidth.

Note: Always test two public websites with a ping before jumping to any conclusions. It is not likely but occasionally a big site like Yahoo will have sporadic response times.

Note: If you have a satellite link, slow and missed pings are normal just a fact-of-life.

 

Tip #3: If you can’t ping a public site, try pinging your local Wireless Router

To ping your local router all you need to find is the IP address of your router. And on almost all networks you can guess it quite easily by looking up the IP address of your computer, and then replacing the last number with a 1.

For example, on my computer I click on my little apple icon, then System Preferences, and then Networking, and I get this screen.  You can see in the Status are it tells me that my IP address is 192.168.1.131.

Finding my IP address output

Screen Shot 2016-04-05 at 10.52.14 AM

The trick to finding your router’s IP address is to replace the last number of any IP address on your network with a 1.  So in my case, I start with my IP address of 192.168.1.131, and I swap the 131 with 1.  I then ping using 192.168.1.1 as my argument, by typing in “ping 192.168.1.1”. A  ping to my router looks like this:

Router Ping  Output

Screen Shot 2016-04-05 at 10.56.30 AM

In the case above I was able to ping my local router and get a response. So what does this tell me?  If I can ping my local wireless router but I can’t ping Yahoo or any other public site, most likely the problem is with my Internet Provider.  To rule out problems with your wireless router or cables, I recommend that you re-boot your wireless router and check the cables coming into it as a next step.

In one case of failure, I actually saw a tree limb on the cable coming from the utility pole to the house. When I called my Internet Provider, I was able to relay this information, which saved a good bit of time in resolving issue.

 

Tip  #4: Look for IP loops

Last week I was getting an error message when I powered up my laptop, saying that some other device had my IP address, and I determined that I was unable to attach to the wireless router. WHAT a strange message!  Fortunately, with my scan tool I can see all the other devices on my network. And although I do not know exactly how I got into this situation, I was quickly able to find the device with the duplicate IP address and powercycle it. This resolved the problem in this case.

 

Tip #5: Look for Rogue Devices

If you never give out the security code to your wireless router, you should not have any unwanted visitors on your network.  To be certain, I again turn to the scan tool.  From my scan output, in the image above (titled “Network Scan Output” near the top of this post), you can see that there are about 15 devices attached to my network. I can account for all of them so for now I have no intruders.

 

Tip #6: Maybe it is just Mischief

There was a time when I left my wireless router wide open as I live in a fairly rural neighborhood and was just being complacent. I was surprised to see that one of my neighbors was on my access point, but which one?

I did some profiling.  Neighbor to my west is a judge with his own network, probably not him.  Across the street, a retired librarian, so probably not her.  That left the Neighbor to my Southwest, kitty corner, a house with all kinds of extended family coming and going, and no network router of their own, at least that I could detect. I had my suspect. And I could also assume they never suspected I was aware of them.

The proper thing to do would have been to block them and lock my wireless router. But since I wanted to have a little fun, I plugged in my bandwidth controller and set their bandwidth down to a fraction of a Megabit.  This had the effect of making their connection painfully dreadfully slow, almost unusable but with a ray of hope.  After a week, he went away and then I completely blocked him (just in case he decided to come back!).

 

Tip #7: Advanced Analysis with a Bandwidth Shaper/Sniffer

If the Ping tool and the Scan tool don’t shed any light on an issue, the next step is to use a more advanced Packet Sniffer. Usually this requires a separate piece of equipment that you insert into your network between your router and network users. I use my NetEqualizer because I have several of them laying around the house.

Often times the problem with your network is some rogue application consuming all of the resources. This can be in the form of consuming total bandwidth, or it could also be seen as overwhelming your wireless router with packets (there are many viruses designed to do just this).

The image below is from a live snapshot depicting bandwidth utilization on a business network. Screen Shot 2016-01-27 at 12.26.49 PM

That top number, circled in red, is a YouTube video, and it is consuming about 3 megabits of bandwidth. Directly underneath that are a couple of cloud service applications from Amazon, and they are consuming 1/10 of what the YouTube video demolishes. On some lower cost Internet links one YouTube can make the service unusable to other applications.

With my sniffer I can also see total packets consumed by a device, which can be a problem on many networks if somebody opens an email with a virus. Without a sniffer it is very hard to track down the culprit.

I hope these tips help you to troubleshoot your network.  Please let us know if you have any questions or tips that you would like to contribute.

Speed up Your Browser, Free Yourself From Java Script


This morning I read an article by Klint Finley about his experience with disabling Java Script.  I am about 8 hours into my experiment now, and here is what I have found so far.

The results were amazing for the on-line periodicals (traditional newspapers) that I like to browse through. Even with my 20 megabit Internet connection, some of these sites are just endless piles of garbage with advertisements and videos popping up, forcing screen refreshes, and making the content unreadable.  Some of them take so long to load, I just give up and get back to work. With Java script turned off, all that changed.   I have not tested the limits on this yet, but I was able to get through a couple of these sites clicking to various articles and my delays were about 1/10 of normal, which is a significant improvement.

On the downside I found some of  the web-based applications that I depend on to be nonfunctional.  Klint mentions issues with Google Docs, but it goes farther than that. My Google Calendar did not work and neither did my WordPress or Cisco Webex. What I am doing now on my MAC laptop is keeping two browsers active.  Firefox with Java Script disabled, and Safari with it enabled.  I feel that this is a good compromise and worth the effort of switching.

Editors Note: Turning off Java Script is only going to impact things that you launch from a traditional browser. The pre- loaded applications on your devices do not use Java Script.

 

Comcast at It Again, Shaping Amazon Content


Sunday night I decided I would finally try watching the Sopranos.  Amazon offers Sopranos content for $1.99 an episode, which saves me the hassle of getting a full year HBO subscription to get episodes.  First pass on my smart internet connected TV,  I could not get the Amazon stream to run at all, and so I reverted to watching it on my laptop.  It came up on the laptop, but the video was choppy and constantly breaking up, stalling etc.   In other words it was being throttled by Comcast.  Solution?

I just fired up my IPvanish which hides the source of the video from Comcast, and presto, I was able to watch the whole episode without an issue.   If you experience content streaming problems with your National ISP try using a VPN tunnel, it has worked for me quite well.

There are other posts about this practice.

There is something rotten in the state of online streaming.

How to get access to blocked Internet Sites.

Editors Note: I completely understand why they throttle content, and have covered the economics behind this before. I just don’t like the secrecy  and deception around it, hence I will continue to publish articles when I find it.

Art Reisman
CTO, http://www.netequalizer.com

What is Your True Internet Speed? Are those Speed Tests Telling the Truth?


When the consumer Internet came of age back in 1990, there was never any grand plan to insure a consistent speed from one point to another. Somewhere along the line, as the Internet went from an academic tool to an essential consumer device, providers in their effort to “out market” one another began to focus on speed as their primary differentiator. By definition, the Internet is a “best effort” corroboration between providers to move your data. No one provider can guarantee a consistent Internet speed for everything you do.  They only have control over their own physical lines, and even then, there are variables beyond their control (which I will address shortly).

Let’s take a look at the speed of wired networks common to most consumers, Cable and DSL.

The physical line into your house is generally what your cable or DSL provider is talking about when they advertise your Internet speed. Essentially, how fast is the link between the providers NOC and your house. Generally you will have a dedicated line for this, and so your speed on this last mile link does not vary.

The good news is that most consumers are more concerned with watching movies, video, listening to music, etc. than they are about pulling research data of some obscure server in Serbia. Given this reality, the Industry has gotten very smart, and popular content is not hosted at some distant server, but is usually distributed locally to each provider. The best example of this is Netflix. Your Netflix content is most likely coming from a server hosted a few miles from your house in your providers NOC, and not from some grand Netflix central location.

Why is Netflix data hosted locally ?

The dirty industry secret is that your provider pays a fee when you go off their network for data. There are also potential capacity problems when you go off their network.  Is this a bad thing? No not really, it is just a matter of efficiency. We see similar practices in other product distribution models. You don’t drive to New York to pick up a toaster, there is usually one waiting for you at your nearest discount store. For the some of the same reasons, that you don’t go to New York to pick up a toaster, your provider tries to host your digital data locally when possible.

What does this mean for your Internet Speed?

It means that when you retrieve content that your provider hosts locally you are likely going to get your advertised speed. This also holds true for some speed test sites, if they are hosted within your providers network they are going to register a constantly higher speed.

What happens to your Internet speed when you go off your providers network? 

There are several factors that will effect your speed.

The main governing factor affecting speed is the capacity the of your providers exchange point.  This is a switching point where your provider exchanges data with other networks.  Depending on how much investment your provider put into this infrastructure this switching point can back up when there is more data being moved than it has capacity to handle. When this happens you get gridlock at the exchange point, and  your Internet speed can plummet.  Gridlock is always a real possibility because your provider just cannot anticipate all the content you are retrieving and sometimes it is not hosted locally.

What does my provider to to alleviate gridlock not their exchange point?

Some providers will actually lower your Internet speed when you are crossing an exchange point.  Or if their circuits are overloaded in general. I experienced this effect which I described in detail a few months ago when I was updating my IPAD.

After the exchange point the speed at which you get your data external to your providers network depends on the whims of every provider and back bone along the route. That obscure research paper from that server in Serbia , may have to make multiple hops to get out of Serbia and then onto some international back bone, and finally to your providers exchange point. There is no way anyone can anticipate at what rate this data will arrive.

How can I run a speed test that better reflects my speed out to the real Internet, by passing locally hosted speed test servers?

A few years ago we ran into this tool set that deliberately tries to retrieve all kinds of remote data to measure your true internet speed. You can also search out files hosted on obscure servers and try to download them.  Perhaps I’ll run a follow up article documenting some of my experiences.

Dear Comcast, Please Stop Slowing my iOS Update


Last week I was forced to re-load my iPad from scratch. So I fired it up and went through the routine that wipes it clean and re-loads the entire OS from the Apple cloud.  As I watched the progress moniker it slowly climbed from 1 hour, then 2 hours, then all the way up to 23 hours –  and then it just stayed there. Now I know the iOS, or whatever they call it on the iPad, is big, but 23 hours big?  I double-checked the download throughput on my NetEqualizer status screen, and sure enough, it was only running at about 60 to 100kbs, no where near my advertised Business Class 20 megabits. So I did a little experiment. I turned on my VPN tunnel, unplugged my iPad for a minute, and then took some steps to hide my DNS (so Comcast had no way to see my DNS requests).  I then restarted my update and sure enough it sped up to about 10 megabits.

To make sure I was not imagining anything I repeated the test.

Without VPN  (slow)

With VPN (fast)

So what is going here, does the VPN make things go faster?   No not really, but it does prevent Comcast from recognizing my iOS update from Apple and singling it out for slower bandwidth.

Why does Comcast (allegedly) shape my download from Apple?

The long story behind this basically boils down to this: it is likely that Comcast really does not have a big enough switch going out to the Internet to support the deluge of bandwidth needed when a group of subscribers all try to update their devices at once.  Especially during peak hours!  Therefor, in order to keep basic services from becoming slow, they single out a few big hitters such as iOS updates.

%d bloggers like this: