Stick a Fork in Third Party Caching (Squid Proxy)

June 24, 2019 — netequalizer

I was just going through our blog archives and noticed that many of the caching articles we promoted circa 2011 are still getting hits.  Many of the hits are coming from less developed countries where bandwidth is relatively expensive when compared to the western world.  I hope that businesses and ISPs hoping for a miracle using caching will find this article, as it applies to all third-party caching engines, not just the one we used to offer as an add-on to the NetEqualizer.

So why do I make such a bold statement about third-party caching becoming obsolete?

#1) There have been some recent changes in the way Google provides YouTube content, which makes caching it almost impossible.  All of their YouTube videos are generated dynamically and broken up into segments, to allow differential custom advertising.  (I yearn for the days without the ads!)

#2) Almost all pages and files on the Internet are marked “Do not Cache” in the HTML headers. Some of them will cache effectively, but you must assume the designer plans on making dynamic, on the fly, changes to their content.  Caching an obsolete page and delivering it to an end user could actually result in serious issues, and perhaps even a lawsuit, if you cause some form of economic harm by ignoring the “do not cache” directive.

#3) Streaming content as well as most HTML content is now encrypted, and since we are not the NSA, we do not have a back door to decrypt and deliver from our caching engines.

As you may have noticed I have been careful to point out that caching is obsolete on third-party caching engines, not all caching engines, so what gives?

Some of the larger content providers, such as Netflix, will work with larger ISPs to provide large caching servers for their proprietary and encrypted content. This is a win-win for both Netflix and the Last Mile ISP.  There are some restrictions on who Netflix will support with this technology.  The point is that it is Netflix providing the caching engine, for their content only, with their proprietary software, and a third-party engine cannot offer this service.  There may be other content providers providing a similar technology.  However, for now, you can stick a fork in any generic third-party caching server.

Posted in Caching and Video, Educational Articles. Tags: , . 1 Comment »

Seven Must Know Network Troubleshooting Tips

April 6, 2016 — netequalizer

Screen Shot 2016-04-05 at 10.07.59 AM.png

By Art Reisman

CTO, APconnections
www.netequalizer.com

To get started you’ll need to get ahold of two key software tools: 1) Ping Tool and 2) a Network Scan Tool, both which I describe in more detail below.  And for advanced analysis (experts only), I will then show you how you can use a bandwidth shaper/sniffer if needed.

Ping Tool

Ping is a great tool to determine what your network responsiveness is (in milliseconds), identified by trying to get a response from a typical website. If you do not already know how to use Ping on your device there are hundreds of references to Ping and how to use it.  Simply google “how to use ping ” on  your favorite device or computer to learn how to use it.

For example, I found these instructions for my MAC; and there are similar instructions for Windows, iPhone, Linux, Android, etc.

  1. Open Network Utility (located inside Applications > Utilities).
  2. Click Ping.
  3. Fill out the “Enter the network that you want to ping” field. You can enter the IP address or a web URL. For example, enter http://www.bbc.co.uk/iplayer to test the ping with that website.
  4. Click Ping.

Network Scan Tool

There are a variety of network SCAN tools/apps available for just about any consumer device or computer.  The decent ones will cost a few dollars, but I have never regretted purchasing one.  I use mine often for very common home and business network issues as I will detail in the tips below. Be sure and use the term “network scan tool” when searching, so you do not get confusing results about unrelated document scanning tools.

Once you get your scan tool installed, test it out by selecting Network Scan. Here is the output from my MAC scan tool.  I will be referencing this output later in the article.

Network Scan Output
Screen Shot 2016-04-05 at 5.33.19 AM

 

Tip #1: Using Ping to see if you are really connected to your Network

I like to open a window on my laptop and keep Ping going all day, it looks like this:

yahoo.com Ping  Output

Screen Shot 2016-04-05 at 8.25.10 AM

Amazingly, seemingly on cue, I lost connectivity to my Internet while I was running the tool for the screen capture above, and no, it was not planned or contrived.  I kicked off my ping by contacting http://www.yahoo.com (type in “ping http://www.yahoo.com”), a public website. And you can see that my round-trip time was around 40 milliseconds before it went dead. Any ping results under 100 milliseconds are normal.

 

Tip #2: How to Deal with Slow Ping Times

In the case above, my Internet Connection just went dead; it came back a minute or so later, and was most likely not related to anything local on my network.

If you start to see missed pings or slow Ping Times above 100 milliseconds, it is most likely due to congestion on your network.  To improve your response times, try turning off other devices/applications and see if that helps.  Even your TV video can suck down a good chunk of bandwidth.

Note: Always test two public websites with a ping before jumping to any conclusions. It is not likely but occasionally a big site like Yahoo will have sporadic response times.

Note: If you have a satellite link, slow and missed pings are normal just a fact-of-life.

 

Tip #3: If you can’t ping a public site, try pinging your local Wireless Router

To ping your local router all you need to find is the IP address of your router. And on almost all networks you can guess it quite easily by looking up the IP address of your computer, and then replacing the last number with a 1.

For example, on my computer I click on my little apple icon, then System Preferences, and then Networking, and I get this screen.  You can see in the Status are it tells me that my IP address is 192.168.1.131.

Finding my IP address output

Screen Shot 2016-04-05 at 10.52.14 AM

The trick to finding your router’s IP address is to replace the last number of any IP address on your network with a 1.  So in my case, I start with my IP address of 192.168.1.131, and I swap the 131 with 1.  I then ping using 192.168.1.1 as my argument, by typing in “ping 192.168.1.1”. A  ping to my router looks like this:

Router Ping  Output

Screen Shot 2016-04-05 at 10.56.30 AM

In the case above I was able to ping my local router and get a response. So what does this tell me?  If I can ping my local wireless router but I can’t ping Yahoo or any other public site, most likely the problem is with my Internet Provider.  To rule out problems with your wireless router or cables, I recommend that you re-boot your wireless router and check the cables coming into it as a next step.

In one case of failure, I actually saw a tree limb on the cable coming from the utility pole to the house. When I called my Internet Provider, I was able to relay this information, which saved a good bit of time in resolving issue.

 

Tip  #4: Look for IP loops

Last week I was getting an error message when I powered up my laptop, saying that some other device had my IP address, and I determined that I was unable to attach to the wireless router. WHAT a strange message!  Fortunately, with my scan tool I can see all the other devices on my network. And although I do not know exactly how I got into this situation, I was quickly able to find the device with the duplicate IP address and powercycle it. This resolved the problem in this case.

 

Tip #5: Look for Rogue Devices

If you never give out the security code to your wireless router, you should not have any unwanted visitors on your network.  To be certain, I again turn to the scan tool.  From my scan output, in the image above (titled “Network Scan Output” near the top of this post), you can see that there are about 15 devices attached to my network. I can account for all of them so for now I have no intruders.

 

Tip #6: Maybe it is just Mischief

There was a time when I left my wireless router wide open as I live in a fairly rural neighborhood and was just being complacent. I was surprised to see that one of my neighbors was on my access point, but which one?

I did some profiling.  Neighbor to my west is a judge with his own network, probably not him.  Across the street, a retired librarian, so probably not her.  That left the Neighbor to my Southwest, kitty corner, a house with all kinds of extended family coming and going, and no network router of their own, at least that I could detect. I had my suspect. And I could also assume they never suspected I was aware of them.

The proper thing to do would have been to block them and lock my wireless router. But since I wanted to have a little fun, I plugged in my bandwidth controller and set their bandwidth down to a fraction of a Megabit.  This had the effect of making their connection painfully dreadfully slow, almost unusable but with a ray of hope.  After a week, he went away and then I completely blocked him (just in case he decided to come back!).

 

Tip #7: Advanced Analysis with a Bandwidth Shaper/Sniffer

If the Ping tool and the Scan tool don’t shed any light on an issue, the next step is to use a more advanced Packet Sniffer. Usually this requires a separate piece of equipment that you insert into your network between your router and network users. I use my NetEqualizer because I have several of them laying around the house.

Often times the problem with your network is some rogue application consuming all of the resources. This can be in the form of consuming total bandwidth, or it could also be seen as overwhelming your wireless router with packets (there are many viruses designed to do just this).

The image below is from a live snapshot depicting bandwidth utilization on a business network. Screen Shot 2016-01-27 at 12.26.49 PM

That top number, circled in red, is a YouTube video, and it is consuming about 3 megabits of bandwidth. Directly underneath that are a couple of cloud service applications from Amazon, and they are consuming 1/10 of what the YouTube video demolishes. On some lower cost Internet links one YouTube can make the service unusable to other applications.

With my sniffer I can also see total packets consumed by a device, which can be a problem on many networks if somebody opens an email with a virus. Without a sniffer it is very hard to track down the culprit.

I hope these tips help you to troubleshoot your network.  Please let us know if you have any questions or tips that you would like to contribute.

Posted in Bandwidth Monitoring, Educational Articles, Network Speed. Leave a Comment »

The Real Story Behind Your Internet Speed

March 24, 2016 — netequalizer

ixp

When your ISP  claims the Internet Speed to your home  is 100 megabits, they are most likely referring to the speed of the link between your home and their Network Operations center ( depicted by the Green lines in the diagram above) . Typically you do not share this link anybody else , hence you truly do have a dedicated 100 megabit link as per their claims.  The caveat is this speed can only be reliably attained  when you are accessing data from their Network Operations Center or from another user also hosted by your ISP .

 

Now, notice the little Purple circle called (IX) in the diagram. This is called an Internet exchange point. It is a choke point where ISPs exchange data to and from each each other. For example, if you are accessing a Web site hosted in France , most likely you will go through one or more IX points to get to your data.  Unless you are in France   of course , but our assumption is that you are not in France .

Your ISP hates it when you go through their exchange point to access a remote server for a couple of reasons.

 

  1. They have to pay money if they originate more data, from their network than the other ISPs using the exchange point.
  2. Exchange points are bottle necks with limited capacity, and if too many users go through it at the same time, their speeds will degrade. For example, if you try to download a big file from the server in France while other users from your ISP are going out to other countries, the speed of your download cannot be guaranteed.  And most likely will not be any where near your 100 megabit promise during peak usage hours.

So why should you care about Internet Exchange Points and how does that impact your Internet speed ?

The answer is it depends on how you use the Internet.

ISPs have gotten very smart , and keep copies of most content local in their NOC or within their Network.The average consumer is not likely to suffer degradation if they stick to commonly used content.  Netflix is a good example. There is no one big Netflix server in Cyber space, there are  in fact, many copies of Netflix servers , and one or more are located at your local ISP.  So if you use the Internet to watch popular content from Netflix, then you will likely get your advertised  Internet speed.

If you are an old timer like me, you might want to explore avant-garde content not available from Netflix, you will likely run into some issues with streaming video. For me with my ISP it is hit or miss. During their off peak hours I usually have no trouble, but many times I am unable to get reliable video from distant servers during peak hours.

The next time that you get a promotional advertisement from an ISP touting their service, ask the sales rep about the throughput you will get going through their exchange server on your way to France and see what they say :)

References IXP toolkit.org

Posted in Educational Articles, Research. Leave a Comment »

Behind The Scenes, How Many Users Can an Access Point Handle ?

June 16, 2015 — netequalizer

Assume you are teaching a class with thirty students, and every one of them needs help with their homework, what would you do? You’d probably schedule a time slot for each student to come in and talk to you one on one (assuming they all had different problems and there was no overlap in your tutoring).

Fast forward to your wireless access point.  You have perhaps heard all the rhetoric about 3.5 gigaherts, or 5.3 megahertz ?

Unfortunately, the word frequency is tossed around in tech buzzword circles the same way car companies and their marketing arms talk about engine sizes. I have no idea what 2.5 Liter Engine is,  it might sound cool and it might be better than a 2 liter engine, but in reality I don’t know how to compare the two numbers. So to answer our original question, we first need a little background on frequencies to get beyond the marketing speak.

A good example of a frequency, that is also easy to visualize, are ripples on pond. When you drop a rock in the water, ripples propagate out in all directions. Now imagine if  you stood in the water, thigh deep across the pond,  and the ripples hit your leg once each second.  The frequency of the ripples in the water would be 1 hertz, or one peak per second. With access points, there are similar ripples that we call radio waves. Although you can’t see them, like the ripples on the water, they are essentially the same thing. Little peaks and values of electromagnetic waves going up and down and hitting the antenna of the wireless device in your computer or Iphone. So when a marketing person tells you their AP is 2.4 Gigahertz, that means those little ripples coming out of  it are hitting your head, and everything else around them, 2.4 billion times each second. That is quite a few ripples per second.

Now in order to transmit a bit of data, the AP actually stops and starts transmitting ripples. One moment it is sending out 2.4 billion ripples pdf second the next moment it is not.  Now this is where it gets a bit weird, at least for me. The 2.4 billion ripples a second really have no meaning as far as data transmission by themselves; what the AP does is set up a schedule of time slots, let’s say 10 million time slots a second, where it is either transmitting ripples, or it turns the ripple generator off. Everybody that is in communication with the AP is aware of the schedule and all the 10 million time slots.  Think of these time slots as dates on your Calendar, and if you have a sunny day, call that a one, while if you have a cloudy day call that a 0.  Cloudy days are a binary 1 and clear day a binary 0. After we string together 8 days we have a sequence of 1’s and 0’s and a full byte. Now 8 days is a long time to transmit a byte, that is why the AP does not use 24 hours for a time slot, but it could , if we were some laid back hippie society where time did not matter.

So let’s go back over what we have learned and plug in some realistic parameters.
Let’s start with a frequency of 2.4 gigahertz. The fastest an AP can realistically turn this ripple generator off and on is about 1/4 the frequency or about 600 time slots/bits per second. This assumes a perfect world and all the bits get out without any interference from other things generating ripples (like your microwave) or something. So in reality the effective rate might be more on the order of 100 million bits a second.
Now let’s say there are 20 users in the room, sharing the available bits equally. They would all be able to run 5 megabits each. But again, there is over head switching between these users (sometimes they talk at the same time and have to constantly back off and re-synch)  Realistically with 20 users all competing for talk time,  1 to 2 megabits per user is more likely.

Other factors that can affect the number of users.
As you can imagine the radio AP manufacturers do all sorts of things to get better numbers. The latest AP’s have multiple antennas and run in two frequencies (two ripple generators) for more bits.

There are also often interference problems with multiple AP’s in the area , all making ripples . The transmission of  ripples for one AP do not stop at a fixed boundary, and this complexity will cause the data rates to slow down while the AP’s sort themselves out.

For related readings on Users and Access Points:

How Many Users Can a Wireless Access Point Handle?

How to Build Your Own Linux Access Points

How to use Access Points to set up and In-Home Music System

Posted in Educational Articles, Network Capacity. Leave a Comment »

Changing times, Five Points to Consider When Trying to Shape Internet Traffic

January 5, 2015 — netequalizer

By Art Reisman, CTO, APconnections www.netequalizer.com

1 ) Traditional Layer 7 traffic shaper methods are NOT able to identify encrypted traffic. In fact, short of an NSA back door, built into some encryption schemes, traditional Layer 7 traffic shapers are slowly becoming obsolete as the percentage of encrypted traffic expands.
2 ) As of 2014, it was estimated that up to 6 percent of the traffic on the Internet is encrypted, and this is expected to double in the next year or so.
3) It is possible to identify the source and destination of traffic even on encrypted streams. The sending and receiving IP’s of encrypted traffic are never encrypted, hence large content providers, such as Facebook, YouTube, and Netflix may be identified by their IP address, but there some major caveats.

– it is common for the actual content from major content providers to be served from regional servers under different domain names (they are often registered to third parties). Simply trying to identify traffic content from its originating domain is too simplistic.

– I have been able to trace proxied traffic back to its originating domain with accuracy by first doing some experiments. I start by initiating a download from a known source, such as YouTube or Netflix, and then I can figure out the actual IP address of the proxy that the download is coming from. From this, I then know that this particular IP is most likely the source of any subsequent YouTube. The shortfall with relying on this technique is that IP addresses change regionally, and there are many of them. You cannot assume what was true today will be true tomorrow with respect to any proxy domain serving up content. Think of the domains used for content like a leased food cart that changes menus each week.

4) Some traffic can be identified by behavior, even when it is encrypted. For example, the footprint of a single computer with a large connection count can usually be narrowed down to one of two things. It is usually either BitTorrent, or some kind of virus on a local computer. BitTorrents tend to open many small connections and hold them open for long periods of time. But again there are caveats. Legit BitTorrent providers such as Universities distributing public material will use just a few connections to accomplish the data transfer. Whereas consumer grade BitTorrents, often used for illegal file sharing, may use 100’s of connections to move a file.

5)  I have been alerted to solutions that require organizations to retrofit all endpoints with pre-encryption utilities, thus allowing the traffic shaper to receive data before it is encrypted.  I am not privy to the mechanics on how this is implemented, but I would assume outside of very tightly controlled networks, such a method would be a big imposition on users.

Posted in Bandwidth Management, Educational Articles. Leave a Comment »

How to keep your IP address static with DHCP

November 7, 2014 — netequalizer

One of the features we support with the NetEqualizer product is a Quota tool, which keeps a running count of total bytes used per IP on a network. A typical IT administrator wants to keep track of data on a per user basis over time, hence some form of Quota tool is essential.  However, a potential drawback of our methodology is that we track usage by IP.   Most networks use a technology called DHCP that dynamically hands out a new IP address each time you power up and power down your computer or wireless device. Most network administrators can track a specific user to an IP in the moment, but they have no idea who had the IP address last week or last month.  Note: there are authentication tools such as Radius or Nomadix that can be used to track users by name but, this adds a complex layer of additional overhead to a simple network.

Yesterday, when working with a customer, the subject came up about our Quota tool, and its drawback of not being able to track a user by IP over time, and the customer turned that into a teaching moment for me.

You see, a DHCP server will always try and give the same IP address back to the same device if the previous IP address is available.   So the key is keeping that IP address available; and there is a simple trick to make sure that this happens.

When you set up a DHCP server it will ask you the range of IP addresses you want to use. All one needs to do is ensure that the defined range is much bigger than the number of devices that will be on your network, and then you can be almost certain that a device will always get the same IP.  This is because the DHCP server only re-uses previously assigned IP addresses when all IP addresses have been assigned, and this would only happen if you defined your IP address range to a relative small number relative to the number of potential devices on your network.   There is no real extra cost for defining your DHCP address range as a Class B instead of the typical default Class C, which then expands your range from 255 to 64,000.  So make sure your ranges are large enough and feel free to track your users by IP without worry.

If you would like to learn more about our Quota tool, also known as “User Quota”, you can read more about it in our User Guide.

Posted in Bandwidth Monitoring, Educational Articles. Tags: , . Leave a Comment »

Notes from a cyber criminal

October 10, 2014 — netequalizer

After a couple of recent high profile data thefts,   I put the question to myself,  how does a cyber thief convert a large amount of credit cards into a financial windfall?

I did some research, and then momentarily put on the shoes of a cyber thief, here are my notes and thoughts:

I am the greatest hacker in the world and I just got a-hold of twenty million  Home Depot debit cards and account numbers. What is my next move. Well I guess I could just start shopping at Home Depot every day and maxing out all my stolen account cards with a bunch of Lawn Mowers , Garden Hoses, and other items. How many times could I do this before I got caught ?  Probably not that many, I am sure the buying patterns would be flagged even before the consumer realized their card was stolen , especially if I was nowhere near the home area code of my victim(s).  And then I’d have to fence all those items to turn it into cash. But let’s assume I acted quickly and went on a home depot shopping spree with my twenty million cards.  Since I am a big time crook I am looking for a haul I can retire on, and so I’d want to buy and fence at least a few hundred thousand dollars worth of stuff out the gate. Now that is going to be quite a few craig(s) list advertisements, and one logistical nightmare to move those goods, and also I am leaving a trail back to me because at some point I have to exchange the goods with the buyer and they are going to want to pay by check . Let me re-think this…

Okay so I am getting smarter, forget the conventional method , what if I find some Russian portal where I can just sell the Home Depot cards and have the funds paid in Bitcoin to some third-party account that is untraceable.  How many people actually have Bitcoin accounts, and how many are interested in buying stolen credit cards on the black market, and then how to insure that the numbers have not been deactivated ? Suppose I sell to some Mafia type and the cards are not valid anymore ? Will they track me down and kill me ? Forget the Bitcoin,  I’ll have to use Paypal , again leaving a trail of some kind.  So now how do I market my credit card fencing site, I have 20 million cards to move and no customers.  A television advertisement , an underworld blog post ?  I need customers to buy these cards and I need them fast , once I start selling them Home Depot will only take a few days to shut down their cards . Maybe I can just have an agent hawk them in Thailand for $3 each , that way I stay anonymous, yeh that’s what I’ll do whew , I’ll be happy if I can net a few thousand dollars.

Conclusion: Although the theft of a data makes a great headline and is certainly not to be taken lightly , the ability for the crook(s) to convert bounty into a financial windfall, although possible is most likely a far more difficult task than the data theft . Stealing the data is one thing, but profiting from it on anything but the smallest scale is very difficult if not impossible.

The real problem for the hacked commercial institution is not the covering the loss of revenue from the theft, but the loss of company value from loss of public trust which can mount into the billions.

Although my main business is Bandwidth Control I do spend a good deal of thought cycles on Security as on occasion the two go hand in hand. For example some of the utilities we use on our NetEqualizer are used to thwart DOS attacks.  We also have our NetGladiator product which is simply the best and smartest tool out there for preventing an attack through your Website.

Posted in Educational Articles, Security. Leave a Comment »

An Easy Way to Get Rid of Wireless Dead Spots and Get Whole Home Music

May 11, 2014 — netequalizer

By Steve Wagor, Co-Founder APconnections

Wireless dead spots are a common problem in homes and offices that expand beyond the range of single wireless access point. For example in my home office, my little Linksys Access point works great on my main floor , but down in my basement the signal just does not reach very well. The problem with a simple access point is if you need to expand your area you must mesh a new one, and off the shelf they do not know how to talk to each other.

For those of you have tried to expand your home network into a mesh with multiple access points there are howto’s out there for rigging them up

Many use wireless access points that are homemade, or the commercial style made for long range. With these solutions you will most likely need a rubber ducky antenna and either some old computers or at least small board computers with attached wireless cards. You will also need to know a bit of networking and setup most of these types of things via what some people would consider complex commands to link them all up into the mesh.

Well its a lot easier than that if you don’t need miles and miles of coverage using off the shelf Apple products. These are small devices with no external antennas.

First you need to install an Apple Extreme access point:
http://www.apple.com/airport-extreme
– at the time of this being written it is $199 and has been at that price for at least a couple of years now.

Now for every dead spot you just need the Apple Express:
http://www.apple.com/airport-express/
– at the time of this being written it is $99 and has been at that price for at least a couple of years now too.

So for every dead spot you have you can solve the problem for $99 after the Apple Extreme is installed. And Apple has very good install instructions on the product line so you don’t need to be a network professional to configure it. Most of it is simple point and click and all done via a GUI and without having to go to a command line ever.

For whole home music fairly effortlessly you can use the Analog/Optical Audio Jack on the back of the Airport Express and plug into your stereo or externally powered speakers. Now connect your iPhone or Mac product up to the same wireless network provided by your Airport Extreme and you can use Airplay to toggle on all or any of the stereos that your network has access to. So if you let your guests access your wireless network and they have an iPhone with Airplay then they could let you listen to anything they are playing by using Airplay to play it on your stereo for example while you are working out together in your home gym.

Posted in Educational Articles, Topics. Tags: , . 2 Comments »

Clone(skb), The inside story on Packet Sniffing Efficiently on a Linux Platform

February 8, 2014 — netequalizer

Even if you are not a  complete geek you might find this interesting.

The two common tools in standard Linux used in many commercial packet sniffing firewalls are, IPtables, and  the Layer7 Packet Classifier.  These low level rule sets are often used in commercial firewalls to identify protocols ( Youtube, Netflix etc) and  then to take action by blocking them or reducing their footprint;  however in their current form, they can bog down your firewall when exposed to higher throughput levels.  The  basic problems as you  run at high line speeds are

  •  The path through the Linux Kernel is bottle necked around an Interface port. What this means is that for every packet that must be analyzed for a specific protocol, the interface port where packets arrive, is put on hold while the analysis is completed. Think of a line of cars being searched going through a border patrol check point. Picture the back up as each car is completely searched at the gate while other cars wait in line. This is essentially what happens in the standard Linux-based packet classifier, all packets are searched while other packets wait in line. Eventually this can cause latency.
  • The publicly available protocol patterns are not owned and supported by any entity and  they are somewhat unreliable. I know, because I wrote and tested many of them over 10 years ago and they are still published and re-used. In fairness, protocol accuracy will always be the Achilles heel of layer 7 detection. There is however some good news in this area which I will cover shortly.

Technology Changes in the Kernel to alleviate the bottleneck

A couple of years ago we had an idea to create a low-cost turn-key intrusion detection device. To build something that could stand up to todays commercial line speeds we would require a better layer 7 detection engine that the standard IPtables solution.  We ended building a very nice Intrusion detection device called the NetGladiator.  One of the stumbling blocks of building this device that we overcame was to maintain a commercial grade line speed of up to 1 gigabyte while still being able to inspect packets.  How did we do it?

Okay so I am a geek, but while poking around in the Linux Kernel I  noticed an interesting call titled Clone(skb). What clone skb does, is allow you to make a very fast copy of an IP packet and its data as it comes through the kernel.  I also noticed in  the newer Linux kernel there was a mechanism for multi-threading.  If you go back to my analogy of cars lined up the border  you can think of  multi-threading and cloning each car such that:

1) Car comes to the border station,

2) clone (copy) it, wave it through without delay

3) send the clone off to a processing lab for analysis ,  a really close by lab near the border

4) If the analysis  from the lab comes back with contraband in the clone, then send a helicopter after the original car and arrest the occupants

5) Throw the clone away

We have taken the cloning and multi-threading elements of the Linux Kernel and produced a low cost, accurate packet classifier that can run at 10 times the line speeds as the standard tools. It will be released in Mid February

 

Posted in Bandwidth Management, Educational Articles. Leave a Comment »

Using OpenDNS on Your Wireless Network to Prevent DMCA infringements

September 19, 2013 — netequalizer

Editor’s Note:  The following was written by guest columnist, Sam Beskur, CTO of Global Gossip.  APconnections and Global Gossip have partnered to offer a  joint hotel service solution, HMSIO.  Read our HMSIO service offering datasheet to learn more.

Traffic Filtering with OpenDNS

 

Abstract

AUP (Acceptable Use Policy) violations which include DMCA infringements on illegal downloads (P2P, Usenet or otherwise) have been hugely troublesome in many locations where we provide public access WiFi.  Nearly all major carriers here in the US now have some form of notification system to alert customers when violation occur and the once that don’t send notifications are silently tracking this behavior.

As a managed service provider it is incredibly frustrating to receive these violation notifications as they never contain information one needs to stop the abuse but only the WAN IP of the offending location.  An end user who committed the infraction is often behind a NATed private address (192.168.x.x or 172.x.x.x) and for reasons still unknown to me they never provide information on the site hosting the illegal material, botnet, adware etc.

When a customer, on whose behalf one may be providing managed services for, receives one of these notifications this can jeopardize your account.

Expensive layer 7 DPI appliances will do the job in filtering P2P traffic but often times customers are reluctant to invest in these devices for a number of reasons: yet another appliance device to power, configure, maintain, support, another point of failure, another config to backup, no more Rackspace, etc, etc ad nausea.

Summary

Below we outline an approach that uses a cloud approach based on OpenDNS and NetEq which has very nearly eliminated all AUP violations across the networks we manage.

Anyone can use the public OpenDNS servers at the following addresses:

208.67.222.222

208.67.220.220

If however, one wishes to use the advanced filter capabilities you will need to subscribe to and create a paid account and register the static WAN IP of the address you are trying to filter.  Prices vary.

  1. Adjusted our content filter/traffic shaper (NetEqualizer) to limit/block # P2P connections.

  1. Configure your router / gateway device / dhcp server to use 208.67.222.220,  208.67.222.222  as primary and secondary DNS server.

     

  1. Once you have an OpenDNS account add your location for filtering and configure DNS blocking of P2P and malware sites         

  1. In order to prevent the more technically savvy end users from specifying ones own DNS server (8.8.8.8, 4.2.2.2, 4.2.2.1, etc.) it is a VERY good idea to configure your gateway to block all traffic on port 53 to all endpoints accept the OpenDNS servers.  DNS uses UDP port 53 so configuring this within IPTables (maybe even another feature for NetEqualizer) or within Cisco IOS is fairly trivial.  If you’re router doesn’t allow this hack it or get another one.

     

Depending on your setup there are a number of other techniques that can be added to this approach to further augment your ability to track NATed end user traffic but as I mentioned these steps alone have very nearly eliminated our AUP violation notifications.

Posted in Educational Articles, HEOA/RIAA. 1 Comment »

Your heard it here first, our prediction on how video will evolve to conserve bandwidth

May 8, 2013 — netequalizer

Editors Note:

I suspect somebody out there has already thought of this,  but in my quick internet search I could not find any references to this specific idea, so I am takaing journalistic first  claim unofficial first rights to this idea.

The best example I think of to exemplify efficiency in video, are the old style cartoons,  such as the parody of South Park. If you ever watch south park animation,  the production quality  is done deliberately cheesy, very few moving parts with fixed backgrounds. In the South Park case, the intention was obviously not to save production costs.  The cheap animation is part of the comedy. That was not always the case,  the evolution of this sort of stop animation cartoon was from the early days  before computer animation took over the work of human artists working frame by frame. The fewer moving parts in a scene, the less work for the animator.  They could re-use existing drawings of a figure and just change the orientation of the mouth in perhaps three positions to animate talking.

Modern video compression tries to take advantage of some of the inherit static data from image to image , such that, each new frame is transmitted with less information.  At best, this is a hit or miss proposition.  There are likely many frivolous moving parts in a back ground that perhaps on the small screen of hand held device are not necessary.

My prediction is we will soon see a collaboration between production of video and Internet transport providers that allows for the average small device video production to have a much smaller footprint in transit.

Some of the basics of this technique would involve.

1) deliberately blurring or sending a background separate from the action. Think of a wide shot of break away lay-up in a basketball game. All you really need to see is the player and the basket in the frame the brain is going to ignore background details such as the crowd, they might as well be static character animations, especially on the scale of the screen of your Iphone not the same experience as your 56 inch HD flat screen.

2) Many of the videos in circulation the internet are news casts of a talking head giving the latest headlines. If you wanted to be extreme, you could  make the production such that the head is  tiny and animate it like a south park character,  this will take a much smaller footprint but technically still be video, and it would be much more like to play through without pausing.

3) The content sender can actually send a different production of the same video for low-bandwidth clients.

Note the reason why the production side of the house must get involved with the compression and delivery side of video is that the compression engines can only make assumptions on what is important and what is not, when removing information (pixels) from a video.

With a smart production engine geared toward the Internet, there is big savings here. Video is busting out all over the Internet and conserving from a production side only makes sense if you want to get your content deployed and viewed everywhere .

The security industry also does something similar taking advantage with fixed cameras on fixed backgrounds.

Related How much YouTube can the Internet Handle

Related Out of the box ideas on how to speed up your Internet

Blog dedicated to video compression, Euclid Discoveries.

 

 

Posted in Bandwidth Management, Educational Articles, Research. Tags: , , , , . 1 Comment »

How Much Bandwidth Do You Really Need?

February 17, 2013 — netequalizer

By Art Reisman – CTO – www.netequalizer.com

Art Reisman CTO www.netequalizer.com

When it comes to how much money to spend on the Internet, there seems to be this underlying feeling of guilt with everybody I talk to. From ISPs, to libraries or multinational corporations, they all have a feeling of bandwidth inadequacy. It is very similar to the guilt I used to feel back in College when I would skip my studies for some social activity (drinking). Only now it applies to bandwidth contention ratios. Everybody wants to know how they compare with the industry average in their sector. Are they spending on bandwidth appropriately, and if not, are they hurting their institution, will they become second-rate?

To ease the pain, I was hoping to put a together a nice chart on industry standard recommendations, validating that your bandwidth consumption was normal, and I just can’t bring myself to do it quite yet. There is this elephant in the room that we must contend with. So before I make up a nice chart on recommendations, a more relevant question is… how bad do you want your video service to be?

Your choices are:

  1. bad
  2. crappy
  3. downright awful

Although my answer may seem a bit sarcastic, there is a truth behind these choices. I sense that much of the guilt of our customers trying to provision bandwidth is based on the belief that somebody out there has enough bandwidth to reach some form of video Shangri-La; like playground children bragging about their father’s professions, claims of video ecstasy are somewhat exaggerated.

With the advent of video, it is unlikely any amount of bandwidth will ever outrun the demand; yes, there are some tricks with caching and cable on demand services, but that is a whole different article. The common trap with bandwidth upgrades is that there is a false sense of accomplishment experienced before actual video use picks up. If you go from a network where nobody is running video (because it just doesn’t work at all), and then you increase your bandwidth by a factor of 10, you will get a temporary reprieve where video seems reliable, but this will tempt your users to adopt it as part of their daily routine. In reality you are most likely not even close to meeting the potential end-game demand, and 3 months later you are likely facing another bandwidth upgrade with unhappy users.

To understand the video black hole, it helps to compare the potential demand curve pre and post video.

A  quality VOIP call, which used to be the measuring stick for decent Internet service runs about 54kbs. A quality  HD video stream can easily consume about 40 times that amount. 

Yes, there are vendors that claim video can be delivered at 250kbs or less, but they are assuming tiny little stop action screens.

Couple this tremendous increase in video stream size with a higher percentage of users that will ultimately want video, and you would need an upgrade of perhaps 60 times your pre-video bandwidth levels to meet the final demand. Some of our customers, with big budgets or government subsidized backbones, are getting close but, most go on a honeymoon with an upgrade of 10 times their bandwidth, only to end up asking the question, how much bandwidth do I really need?

So what is an acceptable contention ratio?

  • Typically in an urban area right now we are seeing anywhere from 200 to 400 users sharing 100 megabits.
  • In a rural area double that rati0 – 400 to 800 sharing 100 megabits.
  • In the smaller cities of Europe ratios drop to 100 people or less sharing 100 megabits.
  • And in remote areas served by satellite we see 40 to 50 sharing 2 megabits or less.
Posted in Bandwidth Management, Educational Articles, Research. Tags: , , , , , , , , , , , . 4 Comments »

Consumer Bill of Rights for Software Updates

November 25, 2012 — netequalizer

This morning I attached my iPhone to my Mac so I could import some of my latest Thanksgiving pictures. I have done this particular sync perhaps a 100 times in the past, but today I was in a hurry and wanted get everything on my Mac so I could  shoot an e-mail out with the new pictures. Yes I know it is possible to send email from an iPhone directly, but the tiny little box of screen is like working with my eyes closed and my hands behind my back.

Upon initiating the sync, my Mac informed me that something needed an update to complete the operation, not sure why, but it was adamant there was no other way. I clicked the update button and 20 minutes later the update was still running so I gave up. Have you ever wanted to scream “I DON’T WANT THE UPDATE! I AM COMPLETELY HAPPY WITH THE WAY THINGS ARE!” Shortly after this incident, I remembered how congress had passed a bill rights for airline passengers. I suspect as our electronic equipment becomes essential to every day life, somebody is going to come along with a bill of rights for technology users, so I thought it would be a good time to get a head start.

Bill of Rights for updates to smart devices:

1) Tell the user how long an update is going to take before they click a button. If you don’t know how long it will take, then make it a two step process where step one calculates how long it will take, and step two is the update.

2) Give the user an easy option to see what is in the update before they click.

3) Never force a user to take an update unless there is some radical change in technology that requires it.

4) Give the user the option to cancel the update in progress at any time without any consequences.

5) Don’t let your engineering team make some lame excuses as to why you can’t follow the Bill of Rights above. I would be glad to come in as consultant and help make your update process follow the Bill of Rights, and yes I can write the code if needed.

* Yes I am guilty of not always having the best update process for our product line. However we are getting much better. :)

7)  Don’t make a user close applications during the update. If you can’t figure out how to update your software with my applications open than see 5) above.

8) These rules apply to smart TV’s and cable boxes – I missed the first 5 minutes to a big game last year while my visio TV updated itself.

Coming soon the Bill of Rights for truth in Bandwidth Speed and why the Internet is not intended to run video.

The original Computer Users Bill of Rights.

Related Internet Users Bill of Rights

Posted in Educational Articles, Research. Tags: , , , , , , . 2 Comments »

Editors Choice: The Best of Speeding up Your Internet

September 22, 2012 — netequalizer

Edited by Art Reisman

CTO – www.netequalizer.com

Over the years we have written a variety of articles related to Internet Access Speed and all of the factors that can affect your service. Below, I have consolidated some of my favorites along with a quick convenient synopsis.

How to determine the true speed of video over your Internet connection: If you have ever wondered why you can sometimes watch a full-length movie without an issue while at other times you can’t get the shortest of YouTube videos to play without interruption, this article will shed some light on what is going on behind the scenes.

FCC is the latest dupe when it comes to Internet speeds: After the Wall Street Journal published an article on Internet provider speed claims, I decided to peel back the onion a bit. This article exposes anomalies between my speed tests and what I experienced when accessing real data.

How to speed up your Internet connection with a bandwidth controller: This is more of a technical article for Internet Service Providers. It details techniques used to eliminate congestion on their links and thus increase the perception of higher speeds to their end users.

You may be the victim of Internet congestion: An article aimed at consumer and business users to explain some of the variance in your network speeds when congestion rears its ugly head.

Just how fast is your 4g network?: When I wrote this article, I was a bit frustrated with all the amazing claims of speed coming with wireless 4G devices. There are some fundamental gating factors that will forever insure that your wired connection will likely always be a magnitude faster than any wireless data device.

How does your ISP enforce your Internet speed?: Goes into some of the techniques used on upstream routers to control the speed of Internet and data connections.

Burstable Internet connections, are they of any value?: Sheds light on the ambiguity of the term “burstable.”

Speeding up your Internet connection with an optimizing appliance: Breaks down the tradeoffs of various techniques.

Why caching alone will speed up your Internet: One of my favorite articles. Caching, although a good idea, often creates great unattainable expectations. Find out why.

QoS is a matter of sacrifice: Explains how quality of service is a “zero sum” game, and why somebody must lose when favoring one type of traffic.

Using QoS to speed up traffic: More on the pros and cons of using a QoS device.

Nine tips and tricks to speed up your Internet connection: A great collection of 15 tips, this article seems to be timeless and continually grows in popularity.

Network bottlenecks when your router drops packets: A simple, yet technical, explanation of how hitting your line speed limit on your router causes a domino effect.

Why is the Internet access in my hotel so slow: Okay I admit i , this was an attempt to draw some attention to our NetEqualizer which solves this problem about 99 percent of the time for the hotel industry. You can bring the horse to water but you cannot make them drink.

Speed test tools from M-labs: The most reliable speed test tool there is, uses techniques that cannot easily be fooled by special treatment from your provider.

Are hotels jamming 3g access?: They may not be jamming 3g but they are certainly in no hurry to make it better.

Five more tips in testing your Internet speed: More tips to test Internet speed.

Posted in Educational Articles, Network Speed, Research. Tags: , , . 1 Comment »

How to Speed Up Data Access on Your iPhone

July 31, 2012 — netequalizer

By Art Reisman

Art Reisman CTO www.netequalizer.com

Editor’s note: Art Reisman is the CTO of APconnections. APconnections designs and manufactures the popular NetEqualizer bandwidth shaper.

Ever wonder if there was anything you can do to make your iPhone access a little bit faster?

When on Your Provider’s 4g Network and Data Access is Slow.

The most likely reason for slow data access is congestion on the provider line. 3g and 4g networks all have a limited sized pipe from the nearest tower back to the Internet. It really does not matter what your theoretical data speed is, when there are more people using the tower than the back-haul pipe can handle, you can temporarily lose service, even when your phone is showing three or four bars.

The other point of contention can be the amount of users connected to a tower exceeds the the towers carrying capacity in terms of frequency.  If this occurs you likely will not only lose data connectivity but also the ability to make and receive phone calls.

Unfortunately, you only have a couple of options in this situation.

– If you are in a stadium with a large crowd, your best bet is to text during the action. Pick a time when you know the majority of people are not trying to send data. If you wait for a timeout or end of the game, you’ll find this corresponds to the times when the network slows to a crawl, so try to finish your access before the last out of the game or the end of the quarter.

– Get away from the area of congestion. I have experienced complete lockout of up to 30 minutes, when trying to text, as a sold out stadium emptied out. In this situation my only chance was to walk about 1/2 mile or so from the venue to get a text out. Once away from the main stadium, my iPhone connected to a tower with a different back haul away from the congested stadium towers.

When connected to a local wireless network and access is slow.

Get close to the nearest access point.

Oftentimes, on a wireless network, the person with the strongest signal wins. Unlike the cellular data network , 802.11  protocols used by public wireless access points have no way to time-slice data access. Basically, this means the device that talks the loudest will get all the bandwidth. In order to talk the loudest, you need to be closest to the access point.

On a relatively uncrowded network you might have noticed that you get fairly good speed even on a moderate or weak signal.  However, when there are a large number of users competing for the attention of a local access point, the loudest have the ability to dominate all the bandwidth, leaving nothing for the weaker iPhones. The phenomenon of the loudest talker getting all the bandwidth is called the hidden node problem. For a good explanation of the hidden node issue you can reference our white paper on the problem.

Shameless plug: If you happen to be a provider or know somebody that works for a provider please tell them to call us and we’d be glad to explain the simplicity of equalizing and how it can restore sanity to a congested network.

Posted in Bandwidth Management, Educational Articles, Network Speed. 5 Comments »
« Older posts
%d bloggers like this: