Why Caching Alone Will Not Solve Your Congestion Issue

November 2, 2011 — netequalizer

Editors Note:
The intent of this article to is to help set appropriate expectations of using a caching server on an uncontrolled Internet link. There are some great speed gains to be had with a caching server; however, caching alone will not remedy a heavily congested Internet connection.

 

Are you going down the path of using a caching server (such as Squid) to decrease peak usage load on a congested Internet link? 

You might be surprised to learn that Internet link congestion cannot be mitigated with a caching server alone. Contention can only be eliminated by:

1) Increasing bandwidth

2) Some form of bandwidth control

3) Or a combination of 1) and 2)

A common assumption about caching is that somehow you will be able to cache a large portion of common web content – such that a significant amount of your user traffic will not traverse your backbone to your provider. Unfortunately, caching a large portion of web content to attain a significant hit ratio is not practical, and here is why:

Lets say your Internet trunk delivers 100 megabits and is heavily saturated prior to implementing caching or a bandwidth control solution. What happens when you add a caching server to the mix?

From our experience, a good hit rate to cache will likely not exceed 10 percent. Yes, we have heard claims of 50 percent, but have not seen this in practice. We assume this is an urban myth or just a special case.

Why is the hit rate at best only 10 percent?

Because the Internet is huge relative to a cache, and you can only cache a tiny fraction of total Internet content. Even Google, with billions invested in data storage, does not come close. You can attempt to keep trending popular content in the cache, but the majority of access requests to the Internet will tend to be somewhat random and impossible to anticipate. Yes, a good number of hits might hit the Yahoo home page and read the popular articles, but many users more are going to do unique things. For example, common hits like email and Facebook are all very different for each user, and cannot be maintained in the cache. User hobbies are also all different, and thus they traverse different web pages and watch different videos. The point is you can’t anticipate this data and keep it in a local cache any more reliably than guessing the weather long term. You can get a small statistical advantage, and that accounts for the 10 percent that you get right.

Note: Without a statistical advantage your hit rate would be effectively be 0.

Even with caching at a 10 percent hit rate, your link traffic will not decline.

With caching in place, any gain in efficiency will be countered by a corresponding increase in total usage. Why is this?

If you assume a 10 percent hit rate to cache, you will end up getting a 10 percent increase in Internet usage and thus, if your pipe to the Internet was near congestion when you put the caching solution in, it will still be congested. Yes, the hits to cache will be fast and amazing, but the 90 percent of the hits that do not come from the cache will equal 100 percent of your Internet link. The resulting effect will be that 90 percent of your Internet accesses will be sluggish due to the congested link.

Another way to understand is by practical example.

Let’s start with a very congested 100 megabit Internet link. Web hits are slow, YouTube takes forever, email responses are slow, and Skype calls break up. To solve these issues, you put in a caching server.

Now 10 percent of your hits come from cache, but since you did nothing to mitigate overall bandwidth usage, your users will simply eat up the extra 10 percent from cache and then some. It is like giving a drug addict a free hit of their preferred drug. If you serve up a fast YouTube, it will just encourage more YouTube usage.

Even with a good caching solution in place, if somebody tries to access Grandma’s Facebook page, it will have to come over the congested link, and it may time out and not load right away. Or, if somebody makes a Skype call it will still be slow. In other words, the 90 percent of the hits not in cache are still slow even though some video and some pages play fast, so the question is:

If 10 percent of your traffic is really fast, and 90 percent is doggedly slow, did your caching solution help?

The answer is yes, of course it helped, 10 percent of users are getting nice, uninterrupted YouTube. It just may not seem that way when the complaints keep rolling in. :)

 

Editors Update August 20 2013

This article written back in 2011  still says it all, and we continue to confirm  by talking to our ISP customers, that, at best a  generic caching engine will get a 10 percent hit rate for people watching movies. However this hit rate has little effect on solving congestion issues on the Internet link itself.

Posted in Bandwidth Management. Tags: , . 6 Comments »

Eleven Tips to Improve VoIP & Video on the Internet Using NetEqualizer and DiffServ/TOS Bits

October 30, 2011 — netequalizer

When talking to potential customers that do not have a NetEqualizer in place (yet), we often hear concerns from companies with recently installed VoIP systems that they are having trouble hearing incoming calls on their phones.  Typically, the root cause for this poor connection is that users are downloading files simultaneously with their VoIP calls.

Router technologies use a technology called DiffServ to enforce priority. Diffserv is reliable at preventing your outgoing Internet data users from interfering with your VoIP calls; however, most router technologies cannot prevent incoming Internet data traffic from overwhelming your incoming VoIP stream. This makes for the interesting dilemma on a call where they can hear you but you can’t hear them.

Fortunately, our bandwidth shaping technology, unlike a basic router, already uses techniques that allow an enterprise to prevent incoming data from overwhelming their VoIP/Skype calls.  We call this technology “Equalizing,” and we have recently enhanced our Equalizing algorithms (version 5.5 and above) such that specific priority for TOS/DiffServ bits will also be recognized.  DiffServ stands for “Differentiated Services Code Point (DSCP)” field and is analogous to the Type of Service (TOS) field.

The following FAQ addresses eleven common questions about our new TOS/DiffServ-aware technology:

1) Who can take advantage of this feature?
Anybody who needs to give priority to an incoming video or voice stream but does not know the source IP of the sender.

2) How do you control whether traffic coming into your network has a TOS/DiffServ bit enabled or not?
This is great mystery. Very little is written about this and how public Internet applications use the TOS bit. From experiments to-date, it seems that YouTube and VoIP providers are setting TOS bit(s) on their data streams.  This is the main reason why the initial NetEqualizer release 5.5 will be in beta test. It is an experimental release so our customers can turn on TOS/DiffServ priority and gather information on performance gains.

3) Who can set a TOS bit?
Almost any application that wants to can send out a stream with a TOS bit set; however, the typical home user does not have access to the TOS bit.

4) What are some of the Caveats with using the DiffServ/TOS Priority Feature?
In the initial beta release, we did not differentiate between types of TOS bits. There are several bits that can be set in this field by the sender that imply different types of quality. We decided to just treat this entire field as ON or OFF in our first release. Most networks that attempt multiple levels of priority are just not practical, as equipment lacks resolution in their processing to enforce different levels of priority. We decided to keep it simple; a stream either has priority or it doesn’t. Multiple levels of priority is more of an academic endeavor for wishful specifications.

5) How do you set the DiffServ/TOS Priority Feature from the NetEqualizer GUI?
Under “Modify Parameters” in the NetEqualizer set up screen:

TOS_ENABLED (on/off)

6) How do you know when a stream on your network has the DiffServ/TOS bit enabled?
From the “Active Connections” reporting screen on the NetEqualizer GUI, you will see a value of either on or off in the last column of the connection row.  “Off” indicates a TOS value of 0; “on” represents a TOS value greater than 0.

7) How does DiffServ/TOS bit priority compare with normal default equalizing?

To recap: A NetEqualizer bandwidth shaper naturally gives priority to VoIP and small web pages.

Now with the ability to provide priority specifically to streams with the TOS bit set, you can more tightly tune the NetEqualizer for VoIP priority, while at the same time provide priority for video.  The big variable will be just how much the TOS bit is used in public applications. On many of our field systems, we do have room to allow a little extra priority for the occasional video or Skype with video component. With the ability to honor TOS priority, your Internet link can grant priority to video without having to know the IP address of the sender or receiver.

8) What if an ISP allowed priority for a TOS bit and their users get wind of it?  Can they figure out a way to jump in front of the line giving ALL of their traffic priority?
We do not think this is likely at this time; the user would have to be aware of the practice of giving priority to TOS in a bandwidth controller to start, and they would then need a fairly sophisticated setup to change all of their applications to set this bit. A more realistic scenario is that video applications will by default already set this service.

9) With the lack of control over who can set a TOS bit, doesn’t this make this feature a little risky to turn on?
My analogy would be that we have a drug that promises to cure cancer and there might be some side effects (none of them will kill you, we promise), so give it a try and tell us what you find.

Note: An administrator has the ability to turn DiffServ/TOS priority on and off quickly, and take a look at the streams on the network. From our early tests over the Internet, we did see some public streams with this bit set, but it was only a small minority of them. We think the potential benefits far outweigh the risk.
Also, we will be working closely with all customers that participate in the Beta.  When Beta customers choose to turn  on DiffServ/TOS priority, we will be available to support them, and are happy to login and do some quick heuristics to assess results.  Our next release beyond the beta will make some sweeping optimizations.

10) Lets suppose all video from YouTube has the TOS bit set, would it be counter-productive to turn it on?
The worst case scenario here is that it would render your bandwidth shaping ineffective, which is no worse than running your network without your bandwidth shaper.  The best case scenario is that you have a mix of large downloads, BitTorrents, etc. that do not have the TOS bit set,  and so turning this feature on will make your video and VoIP better.

11) Many of the points discussed are specific to priority for video.  What about priority for VoIP – does it help with that?
Yes, it can, but for the most part normal equalizing already gives priority to VoIP.  In our next release, we expect to know if the VoIP providers and video providers are following guidelines for using different TOS bits. We could then give priority to VoIP all of the time, and especially on very tight networks, we could lower the HOGMIN threshold to further differentiate VoIP traffic. This point is rather technical, and if you have read this far it might be a good idea to pick up the phone and talk over these concepts with one of our network engineers.

Related Article
Other Solutions

Posted in Bandwidth Management. Tags: , , . 1 Comment »

You May Be the Victim of Internet Congestion

October 9, 2011 — netequalizer

Have you ever had a mysterious medical malady? The kind where maybe you have strange spots on your tongue, pain in your left temple, or hallucinations of hermit crabs at inappropriate times – symptoms seemingly unknown to mankind?

But then, all of a sudden, you miraculously find an exact on-line medical diagnosis?

Well, we can’t help you with medical issues, but we can provide a similar oasis for diagnosing the cause of your slow network – and even better, give you something proactive to do about it.

Spotting classic congested network symptoms:

You are working from your hotel room late one night, and you notice it takes a long time to get connected. You manage to fire off a couple emails, and then log in to your banking website to pay some bills. You get the log-in prompt, hit return, and it just cranks for 30 seconds, until… “Page not found.” Well maybe the bank site is experiencing problems?

You decide to get caught up on Christmas shopping. Initially the Macy’s site is a bit a slow to come up, but nothing too out of the ordinary on a public connection. Your Internet connection seems stable, and you are able to browse through a few screens and pick out that shaving cream set you have been craving – shopping for yourself is more fun anyway. You proceed to checkout, enter in your payment information, hit submit, and once again the screen locks up. The payment verification page times out. You have already entered your credit card, and with no confirmation screen, you have no idea if your order was processed.

We call this scenario, “the cyclical rolling brown out,” and it is almost always a problem with your local Internet link having too many users at peak times. When the pressure on the link from all active users builds to capacity, it tends to spiral into a complete block of all access for 20 to 30 seconds, and then, service returns to normal for a short period of time – perhaps another 30 seconds to 1 minute. Like a bad case of Malaria, the respites are only temporary, making the symptoms all the more insidious.

What causes cyclical loss of Internet service?

When a shared link in something like a hotel, residential neighborhood, or library reaches capacity, there is a crescendo of compound gridlock. For example, when a web page times out the first time, your browser starts sending retries. Multiply this by all the users sharing the link, and nobody can complete their request. Think of it like an intersection where every car tries to proceed at the same time, they crash in the middle and nobody gets through. Additional cars keep coming and continue to pile on. Eventually the police come with wreckers and clear everything out of the way. On the Internet, eventually the browsers and users back off and quit trying – for a few minutes at least. Until late at night when the users finally give up, the gridlock is likely to build and repeat.

What can be done about gridlock on an Internet link?

The easiest way to prevent congestion is to purchase more bandwidth. However, sometimes even with more bandwidth, the congestion might overtake the link. Eventually most providers also put in some form of bandwidth control – like a NetEqualizer. The ideal solution is this layered approach – purchasing the right amount of bandwidth AND having arbitration in place. This creates a scenario where instead of having a busy four-way intersection with narrow streets and no stop signs, you now have an intersection with wider streets and traffic lights. The latter is more reliable and has improved quality of travel for everyone.

For some more ideas on controlling this issue, you can reference our previous article, Five Tips to Manage Internet Congestion.

Posted in Bandwidth Management, Network Capacity, Network Speed. Tags: , . 1 Comment »

How Effective is P2P Blocking?

October 3, 2011 — netequalizer

This past week, a discussion about peer-to-peer (P2P) blocking tools came up in a user group that I follow. In the course of the discussion, different IT administrators chimed in, citing their favorite tools for blocking P2P traffic.

At some point in the discussion, somebody posed the question, “How do you know your peer-to-peer tool is being effective?” For the next several hours the room went eerily silent.

The reason why this question was so intriguing to me is that for years I collaborated with various developers on creating an open-source P2P blocking tool using layer 7 technology (the Application Layer of the OSI Model). During this time period, we released several iterations of our technology as freeware. Our testing and trials showed some successes, but we also learned how fragile the technology was and we were reluctant to push it out commercially. I had always wondered if other privately-distributed layer 7 blocking tools had found some magic key to perfection?

Sometimes, written words can be taken as fact even though the same spoken words might be dismissed as gossip; and so it was with our published open source technology. We started getting indications that it was getting picked up and integrated in other solutions and touted as gospel.

Our experience with P2P blocking:

Our free P2P blocking tool worked most of the time – maybe eighty percent. Eighty percent accuracy is fine for an experimental open-source tool. Intuitively, a blocking tool is expected to be 99.9 percent effective. Even though most customers would likely not conclusively measure our accuracy, eighty percent was too low to ethically sell this technology without disclosures.

The on-line discussion ended fairly quickly when the question of accuracy was brought up, and I think it is safe to assume the silence is an indication that nobody else was achieving better than eighty percent.

How do you validate the effectiveness of a P2P tool?

1) Brute force testing:

I am not aware of too many IT administrators that have the time to load up six or seven different P2P clients on their laptops, and download bootlegged Madonna videos all day.

In testing P2P clients, we infected several computers with just about every virus in circulation. Over time, you can get a rough idea of how deep you must go to expose weaknesses in your tool set. To be thorough, you can’t stop at the first P2P client tool. In the real world, users on your network will likely search for multiple P2P clients, especially if the first one fails. Once they find a kink in the armor, they will yap to others, exposing your Achilles heel.

2) Reduction of RIAA requests:

Most small-to-medium ISP’s don’t really think about P2P unless they get RIAA requests or their network is saturated.

RIAA requests seem to be a big motivator in purchasing technology to block P2P. If you are getting RIAA requests (these are letters from lawyers threatening to sue you for copyright infringement), you can install your P2P blocking tool, and if in the next week your notifications of copyright violations are way down, you can assume that you have put a good dent in your P2P downloading issue.

3) Reduced congestion:

Plug your P2P tool in and see if your network utilization drops.

4) Lower connection rates through your router:

One of the signatures of P2P is that clients will open up hundreds of connections per minute to P2P servers in order to download content. There are ways to measure and quantify these connection rates empirically.

Other observations:

Many times we’ll hear from an ISP/operator claiming they have P2P users run amok on their network, however analysis often shows most of their traffic is video – Netflix, YouTube, Hulu, etc.

Total P2P traffic has really dropped off quite a bit in the last three or four years. We attribute this decline to:

1) Legal iTunes. 99 cent songs have eliminated the need for pirated music.

2) RIAA enforcement and education of copyright laws.

3) The invention of the iPad and iPhone. These devices control the applications which run on them (they are not going to distribute P2P clients as readily).

One method to handle P2P problems is to control all the computers in your environment, scan them before granting network access, and then block access to P2P sites (the sites where the client utilities are loaded from).

Note: once a P2P client is loaded on a computer you cannot block any single remote site, as the essence of P2P is that the content is not centralized.

Summary:

Results of different P2P blocking techniques are often temporary, especially when you have an aggressive user base with motivation to download free content.

Posted in Bandwidth Management. Tags: , , . 3 Comments »

Speeding Up Your Internet Connection Using a TOS Bit

August 10, 2011 — netequalizer

A TOS bit (Type Of Service bit) is a special bit within an IP packet that directs routers to give preferential treatment to selected packets. This sounds great, just set a bit and move to the front of the line for faster service. As always there are limitations.

How does one set a TOS bit?

It seems that only very special enterprise applications, like VoIP PBX’s, actually set and make use of TOS bits. Setting the actual bit is not all that difficult if you have an application that deals with the Network layer, but most commercial applications just send their data on to their local host computer clearing house for data, which in turn, puts the data into IP packets without a TOS bit set. After searching around for a while, I just don’t see any literature on being able to set a TOS bit at the application level. For example, there are several forums where people mention setting the TOS bit in Skype but nothing definitive on how to do it.

However, not to be discouraged, and being the hacker that I am, I could, with some work, make a little module to force every packet leaving my computer or wireless device standard with the TOS bit set. So why not package this up and sell it to the public as an Internet accelerator?

Well before I spend any time on it, I must consider the following:

Who enforces the priority for TOS packets?

This is a function of routers at the edge of your network, and all routers along the path to wherever the IP packet is going. Generally, this limits the effectiveness of using a TOS bit to networks that you control end-to-end. In other words, a consumer using a public Internet connection cannot rely on their provider to give any precedence to TOS bits; hence this feature is relegated to enterprise networks within a business or institution.

Incoming traffic generally cannot be controlled.

The subject of when you can and cannot control a TOS bit does get a bit more involved (pun intended). We have gone over it in more detail in a separate article.

Most of what you do is downloading.

So assuming that your Internet provider did give special treatment to incoming data (which it likely does not), such as video, downloads, and VoIP, the problem with my accelerator idea is that it could only set the TOS bit on data leaving your computer. Incoming TOS bits would have to be set by the sending server.

The moral of the story is that TOS bits that traverse the public Internet don’t have much of a chance in making a difference in your connection speed.

In conclusion, we are going to continue to study TOS bits to see where they might be beneficial and complement our behavior-based shaping (aka “equalizing”) technology.

Posted in Bandwidth Management, Network Speed. Tags: , , , , . 1 Comment »

NetEqualizer Provides Unique Low-Cost Way to Send Priority Traffic over the Internet

July 26, 2011 — netequalizer

Quality of service, or QoS as it’s commonly known, is one of those overused buzz words in the networking industry. In general, it refers to the overall quality of online activities such as video or VoIP calls, which, for example, might be judged by call clarity. For providers of Internet services, promises of high QoS are a selling point to consumers. And, of course, there are plenty of third-party products that claim to make quality of service that much better.

A year ago on our blog, we broke down the costs and benefits of certain QoS methods in our article QoS Is a Matter of Sacrifice. Since then, and in part to address some of the drawbacks and shortcomings we discussed, we’ve developed a new NetEqualizer release offering a very unique and novel way to provide QoS over your Internet link using a type of service (ToS) bit. In the article that follows, we’ll show that the NetEqualizer methodology is the only optimization device that can provide QoS in both directions of a voice or video call over an Internet link.

This is worth repeating: The NetEqualizer is the only device that can provide QoS in both directions for a voice or video call on an open Internet link. Traditional router-based solutions can only provide QoS in both directions of a call when both ends of a link are controlled within the enterprise. As a result, QoS is often reduced and limited. With the NetEqualizer, this limitation can now be largely overcome.

First, let’s step back and discuss why typical routers using ToS bits cannot ensure QoS for an incoming stream over the Internet. Consider a typical scenario with a VoIP call that relies on ToS bits to ensure quality within the enterprise. In this instance, both sending and receiving routers will make sure there is enough bandwidth on the WAN link to ensure the voice data gets across without interruption. But when there is a VoIP conversation going on between a phone within your enterprise and a user out on the cloud, the router can only ensure the data going out.

When communicating enterprise-to-cloud, the router at the edge of your network can see all of the traffic leaving your network and has the ability to queue up (slow down) less important traffic and put the ToS-tagged traffic ahead of everybody else leaving your network. The problem arises on the other side of the conversation. The incoming VoIP traffic is hitting your network and may also have a ToS bit set, but your router cannot control the rate at which other random data traffic arrives.

The general rule with using ToS bits to ensure priority is that you must control both the sending and receiving sides of every stream.

With data traffic originating from an uncontrolled source, such as with a Microsoft update, the Microsoft server is going to send data as fast as it can. The ToS mechanisms on your edge router have no way to control the data coming in from the Microsoft server, and thus the incoming data will crowd out the incoming voice call.

Under these circumstances, you’re likely to get customer complaints about the quality of VoIP calls. For example, a customer on a conference call may begin to notice that although others can hear him or her fine, those on the other end of the line break up every so often.

So it would seem that by the time incoming traffic hits your edge router it’s too late to honor priority. Or is it?

When we tell customers we’ve solved this problem with a single device on the link, and that we can provide priority for VoIP and video, we get looks as if we just proved the Earth isn’t flat for the first time.

But here’s how we do it.

First, you must think of QoS as the science of taking away bandwidth from the low-priority user rather than giving special treatment to a high-priority user. We’ve shown that if you create a slow virtual circuit for a non-priority connection, it will slow down naturally and thus return bandwidth to the circuit.

By only slowing down the larger low-priority connections, you can essentially guarantee more bandwidth for everybody else. The trick to providing priority to an incoming stream (voice call or video) is to restrict the flows from the other non-essential streams on the link. It turns out that if you create a low virtual circuit for these lower-priority streams, the sender will naturally back off. You don’t need to be in control of the router on the sending side.

For example, let’s say Microsoft is sending an update to your enterprise and it’s wiping out all available bandwidth on your inbound link. Your VPN users cannot get in, cannot connect via VoIP, etc. When sitting at your edge, the NetEqualizer will detect the ToS bits on your VPN and VoIP call. It will then see the lack of ToS bits on the Microsoft update. In doing so, it will automatically start queuing the incoming Microsoft data. Ninety-nine out of one hundred times this technique will cause the sending Microsoft server to sense the slower circuit and back off, and your VPN/VoIP call will receive ample bandwidth to continue without interruption.

For some reason the typical router is not designed to work this way. As a result, it’s at a loss as to how to provide QoS on an incoming link. This is something we’ve been doing for years based on behavior, and in our upcoming release, we’ve improved on our technology to honor ToS bits. Prior to this release, our customers were required to identify priority users by IP address. Going forward, the standard ToS bits (which remain in the IP packet even through the cloud) will be honored, and thus we have a very solid viable solution for providing QoS on an incoming Internet link.

Related article QOS over the Internet is it possible?

Related Example: Below is an excerpt from a user that could have benefited from a NetEqualizer. In this comment below, taken from an Astaro forum, the user is lamenting on the fact that despite setting QoS bits he can’t get his network to give priority to his VoIP traffic:

“Obviously, I can’t get this problem resolved by using QoS functionality of Astaro. Phone system still shows lost packets when there is a significant concurring traffic. Astaro does not shrink the bandwidth of irrelevant traffic to the favor of VoIP definitions, I don’t know where the problem is and obviously nobody can clear this up.

Astaro Support Engineer said “Get a dedicated digital line,” so I ordered one it will be installed shortly.

The only way to survive until the new line is installed was to throttle all local subnets, except for IPOfficeInternal, to ensure the latter will have enough bandwidth at any given time, but this is not a very smart way of doing this.“

Posted in Bandwidth Management. Tags: , , , , . 4 Comments »

Shaping Bandwidth by VLAN under the NetEqualizer Hood

May 18, 2011 — netequalizer

As a followup to my recent commentary on  the history of VLAN tags, I decided to jump down into the guts of a bandwidth shaper and go over some of the techniques we use to set rate limits on a particular VLAN. When writing, I assumed the reader has a basic understanding of how data can be manipulated inside a computer program.

Let’s start with some background information. First off, the NetEqualizer bandwidth shaper is a transparent bridge.  A typical setup has two Ethernet cards —  one connected to your LAN and the other side connected to your WAN (Internet router). Before we added in our VLAN shaping, the Linux kernel bridging code would blindly transfer Ethernet packets from one side to the other, passing right through the NetEqualizer.

As these Ethernet packets pass through, they’re visible as data in the Linux kernel. Normally, they pass through unmolested — in one side out the other. However, the key to bandwidth shaping is what you do with them as they come through.

To give you a better idea of what goes inside the Linux kernel when data passes through, I’ve included a couple of snippets of C code below. This is actual Linux kernel code. I have also littered the code with some detailed explanations in line, so you don’t have to understand C to follow the logic.

Below is the C language data definition of the fields in an Ethernet header. When an Ethernet packet comes across the NetEqualizer, the contents of the Ethernet packet are put into data structures. The reason why we’re interested in the Ethernet header is that it’s where the VLAN tags are located.

Note: Code appears in italics while notes are in bold and non-italicized font.

struct vlan_ether_header {
char dst[6];     // This is six bytes for the destination MAC address.
char src[6];     // This is six bytes for the source MAC address.
short type;
short tci_vid;
short encapsulated_type;
} __attribute__ ((__packed__));

Below is the C function that finds the actual VLAN tag inside the Ethernet header in an Ethernet packet.

struct iphdr* findIph(struct sk_buff* skb, int *vlan_id) {
struct ethhdr* eh;    

// This is a pointer to a data structure of type ether net header. We first declare the pointer and will assign it later.
struct iphdr* iph = NULL;   

// This is a pointer to a data structure that contains the IP header of an IP packet (I did not show the definition of the structure).
*vlan_id = -1;                         

// Set the VLAN ID to something.

eh = (struct ethhdr*)(skb->mac_header);

/*  The SKB buffer is the standard structure for network data being passed around the kernel. It contains all the data related to IP data  including the Ethernet packet. Part of the Ethernet packet is the MAC header which is what we are interested in to find out the VLAN ID. FYI . . . SKB is the buffer that IP tables routinely use. To enforce firewall rules, they pass this buffer from rule-to-rule because everybody needs to look inside of it to decide what to do. I am not going to go into how it came into existence. Suffice to say the Ethernet packet is located in this buffer. The MAC header is a field in the SKB buffer and the above assignment copies this location to the variable eh, which is the pointer of an Ethernet header. We now have a data structure that we can access to see fields inside the Ethernet header as a packet passes through the NetEqualizer */

if (eh->h_proto == 0x0081) {
struct vlan_ether_header* veh = (struct vlan_ether_header*)(skb->mac_header);

if (veh->encapsulated_type == 0x0008) {
iph = (struct iphdr*)(skb->mac_header + sizeof(*veh));
*vlan_id = ((ntohs(veh->tci_vid)) & 0x0fff);
// BR_DEBUG_IP printk (KERN_INFO “got VLAN ID %d \n”, *vlan_id);
}
}

/* The above code snippet is where the actual VLAN ID gets put into the variable vlan_id. The FFF is a bit mask which slices the value of the VLAN ID out of the field tci_vid. It is a 12-bit number */
else {
if (eh->h_proto == 0x0008) {
iph = (struct iphdr*)(skb->mac_header + sizeof(*eh));
}
}
return iph;
}

Hopefully the code captured the spirit of the type of work that goes on in the Linux kernel to analyze packets. But, how does VLAN shaping work once you have the VLAN ID?

Well, once we have the VLAN ID of a packet, we check and see if there is a VLAN shaping rule in effect for that ID. There is a table in the Kernel with a list of all of the active VLAN shaping rules that have been specified by the user. If there is a rule for this VLAN, a counter is incrimented for the number of data bytes in the payload of the IP packet.

if (vlan_id > -1  && vlan_id < VLAN_MAX && hard_table[vlan_id + HARD_SIZE].ip == vlan_id && port_id ==2) {
                hard_table[vlan_id + HARD_SIZE].incount=hard_table[vlan_id +HARD_SIZE].incount +hsize;

The code snippet above checks to make sure the VLAN ID is valid and then it increments the byte count for that VLAN. hsize is a variable that contains the actual number of data bytes in the Ethernet packet.

The NetEqualizer keeps this counter for an entire second (it will reset it each second), and if the data coming in for the VLAN is coming in faster than the rate limit defined by a user rule for that particular VLAN ID, then the NetEqualizer will take action by actually slowing down the packet in the kernel. This in turn reduces the data rate of transfer for the VLAN.

Posted in Bandwidth Management. Tags: , . Leave a Comment »

QoS is a Matter of Sacrifice

April 7, 2011 — netequalizer

Usually in the first few minutes of talking to a potential customer, one of their requests will be something like “I want to give QoS (Quality of Service) to Video”, or “I want to give Quality of Service to our Blackboard application.”

The point that is often overlooked by resellers, pushing QoS solutions, is that providing QoS for one type of traffic always involves taking bandwidth away from something else.

The network hacks understand this, but for those that are not down in the trenches sometimes we must gently walk them through a scenario.

Take the following typical exchange:

Customer: I want to give our customers access to NetFlix and have that take priority over P2P.

NetEq Rep: How do you know that you have a p2p problem?

Customer: We caught a guy with Kazaa on his Laptop last year so we know they are out there.

NetEq rep (after plugging in a test system and doing some analysis): It looks like you have some scattered p2p users, but they are only about 2 percent of your traffic load. Thirty percent of your peak traffic is video. If we give priority to all your video we will have to sacrifice something, web browsing, chat, e-mail, Skype, and Internet Radio. I know this seems like quite a bit but there is nothing else out there to steal from, you see in order to give priority to video we must take away bandwidth from something else and although you have p2p, stopping it will not provide enough bandwidth to make a dent in your video appetite.

Customer (now frustrated by reality): Well I guess I will just have to tell our clients they can’t watch video all the time. I can’t make web browsing slower to support video, that will just create a new problems.

If you have an oversubscribed network, meaning too many people vying for limited Internet resources, when you implement any form of QoS, you will still end up with an oversubscribed network. QoS must rob Peter to pay Paul.

So when is QoS worth while?

QoS is a great idea if you understand who you are stealing from.

Here are some facts on using QoS to improve your Internet Connection:

Fact #1

If your QoS mechanism involves modifying packets with special instructions (ToS bits) on how it should be treated, it will only work on links where you control both ends of the circuit and everything in between.

Fact #2

Most Internet congestion is caused by incoming traffic. For data originating at your facility, you can certainly have your local router give priority to it on its way out, but you can’t set QoS bits on traffic coming into your network (we assume from a third party). Regulating outgoing traffic with ToS bits will not have any effect on incoming traffic.

Fact #3

Your public Internet provider will not treat ToS bits with any form of priority (the exception would be a contracted MPLS type network). Yes, they could, but if they did then everybody would game the system to get an advantage and they would not have much meaning anyway.

Fact #4

The next two facts address our initial question — Is QoS over the Internet possible? The answer is, yes. QoS on an Internet link is possible. We have spent the better part of seven years practicing this art form and it is not rocket science, but it does require a philosophical shift in thinking to get your arms around.

We call it “equalizing,” or behavior-based shaping, and it involves monitoring incoming and outgoing streams on your Internet link. Priority or QoS is nothing more than favoring one stream’s packets over another stream’s packets. You can accomplish priority QoS on incoming streams by queuing (slowing down) one stream over another without relying on ToS bits.

Fact #5

Surprisingly, behavior-based methods such as those used by our NetEqualizer do provide a level QoS for VoIP on the public Internet. Although you can’t tell the Internet to send your VoIP packets faster, most people don’t realize the problem with congested VoIP is due to the fact that their VoIP packets are getting crowded out by large downloads. Often, the offending downloads are initiated by their own employees or users. A good behavior-based shaper will be able to favor VoIP streams over less essential data streams without any reliance on the sending party adhering to a QoS scheme.

Please remember our initial point “providing QoS for one type of traffic always involves taking bandwidth away from something else,” and take these facts into consideration as you work on QoS for your network.

Posted in Bandwidth Management. Tags: , , . 4 Comments »

What Is Deep Packet Inspection and Why the Controversy?

February 8, 2011 — netequalizer

By Art Reisman

Art Reisman CTO www.netequalizer.com

Editor’s note: Art Reisman is the CTO of APconnections. APconnections designs and manufactures the popular NetEqualizer bandwidth shaper. APconnections removed all deep packet inspection technology from their NetEqualizer product over 2 years ago.

Article Updated March 2012

As the debate over Deep Packet Inspection continues, network administrators are often faced with a difficult decision: ensure network quality or protect user privacy. However, the legality of the practice is now being called into question, adding a new twist to the mix. Yet, for many Internet users, deep packet inspection continues to be an ambiguous term in need of explanation. In the discussion that follows, deep packet inspection will be explored in the context of the ongoing debate.

Exactly what is deep packet inspection?

All traffic on the Internet travels around in what is called an IP packet. An IP packet is a string of characters moving from computer A to computer B. On the outside of this packet is the address where it is being sent. On the inside of the packet is the data that is being transmitted.

The string of characters on the inside of the packet can be conceptually thought of as the “payload,” much like the freight inside of a railroad car. These two elements, the address and the payload, comprise the complete IP packet.

When you send an e-mail across the Internet, all your text is bundled into packets and sent on to its destination. A deep packet inspection device literally has the ability to look inside those packets and read your e-mail (or whatever the content might be).

Products sold that use DPI are essentially specialized snooping devices that examine the content (pay load inside) of Internet packets. Other terms sometimes used to describe techniques that examine Internet data are packet shapers, layer-7 traffic shaping, etc.

How is deep packet inspection related to net neutrality?

Net neutrality is based on the belief that nobody has the right to filter content on the Internet. Deep packet inspection is a method used for filtering. Thus, there is a conflict between the two approaches. The net neutrality debate continues to rage in its own right.

Why do some Internet providers use deep packet inspection devices?

There are several reasons:

1) Targeted advertising — If a provider knows what you are reading, they can display content advertising on the pages they control, such as your login screen or e-mail account.

2) Reducing “unwanted” traffic — Many providers are getting overwhelmed by types of traffic that they deem as less desirable such as Bittorrent and other forms of peer-to-peer. Bittorrent traffic can overwhelm a network with volume. By detecting and redirecting the Bittorrent traffic, or slowing it down, a provider can alleviate congestion.

3) Block offensive material — Many companies or institutions that perform content filtering are looking inside packets to find, and possibly block, offensive material or web sites.

4) Government spying — In the case of Iran (and to some extent China), DPI is used to keep tabs on the local population.

When is it appropriate to use deep packet inspection?

1) Full disclosure — Private companies/institutions/ISPs that notify employees that their Internet use is not considered private have the right to snoop, although I would argue that creating an atmosphere of mistrust is not the mark of a healthy company.

2) Law enforcement — Law enforcement agencies with a warrant issued by a judge would be the other legitimate use.

3) Intrusion detection and prevention– It is one thing to be acting as an ISP  and to eaves drop on a public conversation;  it is entirely another paradigm if you are a  private business examining the behavior of somebody  coming in your front door. For example in a private home it is within your right to look through your peep hole and not let shady characters into your home.  In a private business it is a good idea to use Deep packet inspection in order to block unwanted intruders from your network. Blocking bad guys before they break into and damage your network and is perfectly acceptable.

4) Spam filtering- Most consumers are very happy to have their ISP or email provider remove spam.  I would categorize this type of DPI as implied disclosure. For example, in Gmail you do have the option to turn Spam filtering off, and although most consutomers may not realize that google is reading their mail ( humans don’t read it but computer scanners do), their motives are understood. What consumers may not realize is that their email provider is also reading everything they do in order to set target advertising

Does Content filtering use Deep Packet Inspection ?

For the most part no. Content filtering is generally  done at the URL level. URL’s are generally considered public information, as routers need to look this up anyway. We have only encountered content filters at private institutions that are within their right.

What about spam filtering, does that use Deep Packet Inspection?

Yes many Spam filters will look at content, and most people could not live without their spam filter, however with spam filtering most people have opted in at one point or another, hence it is generally done with permission.

What is all the fuss about?

It seems that consumers are finally becoming aware of what is going on behind the scenes as they surf the Internet, and they don’t like it. What follows are several quotes and excerpts from articles written on the topic of deep packet inspection. They provide an overview not only of how DPI is currently being used, but also the many issues that have been raised with the practice.

For example, this is an excerpt from a recent PC world article:

Not that we condone other forms of online snooping, but deep packet inspection is the most egregious and aggressive invasion of privacy out there….It crosses the line in a way that is very frightening.

— Paul Stephens, director of policy and advocacy for the Privacy Rights Clearinghouse, as quoted in the E-Commerce Times on November 14, 2008. Read the full article here.

Recently, Comcast had their hand slapped for re-directing Bittorrent traffic:

Speaking at the Stanford Law School Center for Internet and Society, FCC Chairman Kevin Martin said he’s considering taking action against the cable operator for violating the agency’s network-neutrality principles. Seems Martin was troubled by Comcast’s dissembling around the BitTorrent issue, not to mention its efforts to pack an FCC hearing on Net neutrality with its own employees.

— Digital Daily, March 10, 2008. Read the full article here.

Later in 2008, the FCC came down hard on Comcast.

In a landmark ruling, the Federal Communications Commission has ordered Comcast to stop its controversial practice of throttling file sharing traffic.

By a 3-2 vote, the commission on Friday concluded that Comcast monitored the content of its customers’ internet connections and selectively blocked peer-to-peer connections.

— Wired.com, August 1, 2008.Read the full article here.

To top everything off, some legal experts are warning companies practicing deep packet inspection that they may be committing a felony.

University of Colorado law professor Paul Ohm, a former federal computer crimes prosecutor, argues that ISPs such as Comcast, AT&T and Charter Communications that are or are contemplating ways to throttle bandwidth, police for copyright violations and serve targeted ads by examining their customers’ internet packets are putting themselves in criminal and civil jeopardy.

— Wired.com, May 22, 2008. Read the full article here.

However, it looks like things are going the other way in the U.K. as Britain’s Virgin Media has announced they are dumping net neutrality in favor of targeting bittorrent.

The UK’s second largest ISP, Virgin Media, will next year introduce network monitoring technology to specifically target and restrict BitTorrent traffic, its boss has told The Register.

— The Register, December 16, 2008. Read the full article here.

Canadian ISPs confess en masse to deep packet inspection in January 2009.

With the amount of attention being paid to Comcast recently, a lot of people around the world have begun to look at their ISPs and wonder exactly what happens to their traffic once it leaves. This is certainly true for Canada, where several Canadian ISPs have come under the scrutiny of the CRTC, the regulatory agency responsible for Canada. After investigation, it was determined that all large ISPs in Canada filter P2P traffic in some fashion.

— Tech Spot, January 21, 2009. Read the full article here.

In April 2009, U.S. lawmakers announced plans to introduce legislation that would limit the how ISPs could track users. Online privacy advocates spoke out in support of such legislation.

In our view, deep packet inspection is really no different than postal employees opening envelopes and reading letters inside. … Consumers simply do not expect to be snooped on by their ISPs or other intermediaries in the middle of the network, so DPI really defies legitimate expectations of privacy that consumers have.

— Leslie Harris, president and CEO of the Center for Democracy and Technology, as quoted on PCWorld.com on April 23, 2009. Read the full article here.

The controversy continues in the U.S. as AT&T is accused of traffic shaping, lying and blocking sections of the Internet.

7/26/2009 could mark a turning point in the life of AT&T, when the future looks back on history, as the day that the shady practices of an ethically challenged company finally caught up with them: traffic filtering, site banning, and lying about service packages can only continue for so long before the FCC, along with the bill-paying public, takes a stand.

— Kyle Brady, July 27, 2009. Read the full article here.

[February 2011 Update] The Egyptian government uses DPI to filter elements of their Internet Traffic, and this act in itself becomes the news story. In this video in this news piece, Al Jazeera takes the opportunity to put out an unflattering piece on the company Naurus that makes the DPI technology and sold it to the Egyptians.

While the debate over deep packet inspection will likely rage on for years to come, APconnections made the decision to fully abandon the practice over two years ago, having since proved the viability of alternative approaches to network optimization. Network quality and user privacy are no longer mutually exclusive goals.

Created by APconnections, the NetEqualizer is a plug-and-play bandwidth control and WAN/Internet optimization appliance that is flexible and scalable. When the network is congested, NetEqualizer’s unique “behavior shaping” technology dynamically and automatically gives priority to latency sensitive applications, such as VoIP and email. Click here for a full price list.

Posted in Bandwidth Management, Educational Articles, Net Neutrality. Tags: , , , , , , , , , . 16 Comments »

Pros and Cons of Using Your Router as a Bandwidth Controller

January 30, 2011 — netequalizer

So, you already have a router in your network, and rather than take on the expense of another piece of equipment, you want to double-up on functionality by implementing your bandwidth control within your router. While this is sound logic and may be your best decision, as always, there are some other factors to consider.

Here are a few things to think about:

1. Routers are optimized to move packets from one network to another with utmost efficiency. To do this function, there is often minimal introspection of the data, meaning the router does one table look-up and sends the data on its way. However, as soon as you start doing some form of bandwidth control, your router now must perform a higher-level of analysis on the data. Additional analysis can overwhelm a router’s CPU without warning. Implementing non-routing features, such as protocol sniffing, can create conditions that are much more complex than the original router mission. For simple rate limiting there should be no problem, but if you get into more complex bandwidth control, you can overwhelm the processing power that your router was designed for.

2. The more complex the system, the more likely it is to lock up. For example, that old analog desktop phone set probably never once crashed. It was a simple device and hence extremely reliable. On the other hand, when you load up an IP phone on your Windows PC,  you will reduce reliability even though the function is the same as the old phone system. The problem is that your Windows PC is an unreliable platform. It runs out of memory and buggy applications lock it up.

This is not news to a Windows PC owner, but the complexity of a mission will have the same effect on your once-reliable router. So, when you start loading up your router with additional missions, it is increasingly more likely that it will become unstable and lock up. Worse yet, you might cause a subtle network problem (intermittent slowness, etc.) that is less likely to be identified and fixed. When you combine a bandwidth controller/router/firewall together, it can become nearly impossible to isolate problems.

3. Routing with TOS bits? Setting priority on your router generally only works when you control both ends of the link. This isn’t always an option with some technology. However, products such as the NetEqualizer can supply priority for VoIP in both directions on your Internet link.

4. A stand-alone bandwidth controller can be  moved around your network or easily removed without affecting routing. This is possible because a bandwidth controller is generally not a routable device but rather a transparent bridge. Rearranging your network setup may not be an option, or simply becomes much more difficult, when using your router for other functions, including bandwidth control.

These four points don’t necessarily mean using a router for bandwidth control isn’t the right option for you. However, as is the case when setting up any network, the right choice ultimately depends on your individual needs. Taking these points into consideration should make your final decision on routing and bandwidth control a little easier.

Posted in Bandwidth Management, Commentary and Editorials. Tags: , . 1 Comment »

How to Determine a Comprehensive ROI for Bandwidth Shaping Products

January 9, 2011 — netequalizer

In the past, we’ve published several articles on our blog to help customers better understand the NetEqualizer’s potential return on investment (ROI). Obviously, we do this because we think we offer a compelling ROI proposition for most bandwidth-shaping decisions. Why? Primarily because we provide the benefits of bandwidth shaping at a a very low cost — both initially and even more so over time. (Click here for the NetEqualizer ROI calculator.)

But, we also want to provide potential customers with the questions that need to be considered before a product is purchased, regardless of whether or not the answers lead to the NetEqualizer. With that said, this article will break down these questions, addressing many issues that may not be obvious at first glance, but are nonetheless integral when determining what bandwidth shaping product is best for you.

First, let’s discuss basic ROI. As a simple example, if an investment cost $100, and if in one year that investment returned $120, the ROI is 20 percent.  Simple enough. But what if your investment horizon is five years or longer? It gets a little more complicated, but suffice it to say you would perform a similar calculation for each year while adjusting these returns for time and cost.

The important point is that this technique is a well-known calculation for evaluating whether one thing is a better investment than another — be it bandwidth shaping products or real estate. Naturally and obviously the best financial decision will be determined by the greatest return for the smallest cost.

The hard part is determining what questions to ask in order to accurately determine the ROI. A missed cost or benefit here or there could dramatically alter the outcome, potentially leading to significant unforeseen losses.

For the remainder of this article, I’ll discuss many of the potential costs and returns associated with bandwidth shaping products, with some being more obscure than others. In the end, it should better prepare you to address the most important questions and issues and ultimately lead to a more accurate ROI assessment.

Let’s start by looking at the largest components of bandwidth shaping product “costs” and whether they are one-time or ongoing. We’ll then consider the returns.

COSTS

  • The initial cost of the tool
    • This is a one-time cost.
  • The cost of vendor support and license updates
    • These are ongoing costs and include monthly and annual licenses for support, training, software updates, library updates, etc…  The difference from vendor to vendor can be significant — especially over the long run.
  • The cost of upgrades within the time horizon of the investment
    • These upgrades can come in several different forms. For example, what does it cost to go from a 50Mbs tool to 100Mbs? Can your tool be upgraded, or do you have to buy a whole new tool? This can be a one-time cost or it can occur several times. It really depends on the growth of your network, but it’s usually inevitable for networks of any size.
  • The internal (human) cost to support the tool
    • For example, how many man hours do you have to spend to maintain the tool, to optimize it and to adapt it to your changing network? This could be a considerable “hidden” cost and it’s generally recurring. It also usually increases in time as the cost of salaries/benefits tend to go up. Because of that, this is a very important component that should be quantified for a good ROI analysis. Tools that require little or no ongoing maintenance will have a large advantage.
  • Overall impact on the network
    • Does the product add latency or other inefficiencies? Does it create any processing overhead and how much? If the answer is yes, costs such as these will constantly impact your network quality and add up over time.

RETURNS

  • Savings from being able to delay or eliminate buying more bandwidth
    • This could either be a one-time or ongoing return. Even delaying a bandwidth upgrade for six months or a year can be highly valuable.
  • Savings from not losing existing revenue sources
    • How many customers did you not lose because they did not get frustrated with their network/Internet service? This return is ongoing.
  • Ability to generate new revenue
    • How many new customers did you add because of a better-maintained network?  Were you able to generate revenue by adding new higher-value services like a tiered rate structure? This will usually be an ongoing return.
  • Savings from the ability eliminate or reduce the financial impact of unprofitable customers
    • This is an ongoing savings. Can you convert an unprofitable customer to a profitable one by reducing their negative impact on the network? If not, and they walk, do you care?
  • Avoidance of having to buy additional equipment
    • Were you able to avoid having to “divide and conquer” by buying new access points, splitting VLANs, etc..? This can be a one-time or ongoing return.
  • Savings in the cost of responding to technical support calls
    • How much time was saved by not having to receive an irate customer call, research it and respond back? If this is something you typically deal with on a regular basis, the savings will add up every day, week or month this is avoided.

Overall, these issues are the basic financial components and questions that need to be quantified to make a good ROI analysis. For each business, and each tool, this type of analysis may yield a different answer, but it is important to note that over time there are many more items associated with ongoing costs/savings than those occurring only once. Thus, you must take great care to understand the impact of these for each tool, especially those issues that lead to costs that increase over time.

Posted in Bandwidth Management, Educational Articles, ROI. Tags: , , , . 2 Comments »

The 10-Gigabit Barrier for Bandwidth Controllers and Intel-Based Routers

December 5, 2010 — netequalizer

By Art Reisman

Editor’s note: This article was adapted from our answer to a NetEqualizer pre-sale question asked by an ISP that was concerned with its upgrade path. We realized the answer was useful in a broader sense and decided to post it here.

Any router, bandwidth controller, or firewall that is based on Intel architecture and buses will never be able to go faster than about about 7 gigabits sustained. (This includes our NE4000 bandwidth controller. While the NE4000 can actually reach speeds close to 10 gigabits, we rate our equipment for five gigabits because we don’t like quoting best-case numbers to our customers.) The limiting factor in Intel architecture is that to expand beyond 10-gigabit speeds you cannot be running with a central clock. Therefore, with a central clock controlling the show, it is practically impossible to move data around much faster than 10 gigabits.

The alternative is to use a specialized asynchronous design, which is what faster switches and hardware do. They have no clock or centralized multiprocessor/bus. However, the price point for such hardware quickly jumps to 5-10 times the Intel architecture because it must be custom designed. It is also quite limited in function once released.

Obviously, vendors can stack a bunch of 10-gig fiber bandwidth controllers behind a switch and call it something faster, but this is no different from dividing up your network paths and using multiple bandwidth controllers yourself.  So, be careful when assessing the claims of other manufacturers in this space.

Considering these limitations, many cable operators here in the US have embraced the 10-gigabit barrier. At some point you must divide and conquer using multiple 10-gig fiber links and multiple NE4000 type boxes, which we believe is really the only viable plan — that is if you want any sort of sophistication in your bandwidth controller.

While there are some that will keep requesting giant centralized boxes, and paying a premium for them (it’s in their blood to think single box, central location), when you think about the Internet, it only works because it is made of many independent paths. There is no centralized location by design. However, as you approach 10-gigabit speeds in your organization, it might be time to stop thinking “single box.”

I went through this same learning curve as a system architect at AT&T Bell Labs back in the 1990s.  The sales team was constantly worried about how many telephone ports we could support in one box because that is what operators were asking for.  It shot the price per port through the roof with some of our designs. So, in our present case, we (NetEqualizer) decided not to get into that game because we believe that price per megabit of shaping will likely win out in the end.

Art Reisman is currently CTO and co-founder of APconnections, creator of the NetEqualizer. He  has worked at several start-up companies over the years and has invented and brought several technology products to market, both on his own and with the backing of larger corporations. This includes tools for the automotive industry.

Posted in Bandwidth Management, Educational Articles, Network Capacity. Tags: , . 1 Comment »

Analyzing the cost of Layer 7 Packet Shaping

November 16, 2010 — netequalizer

November, 2010

By Eli RIles

For most IT administrators layer 7 packet shaping involves two actions.

Action 1:  Involves inspecting and analyzing data to determine what types of traffic are on your network.

Action 2: Involves taking action by adjusting application  flows on your network .

Without  the layer 7 visibility and actions,  an administrator’s job would degrade into a quagmire of random guesswork. Or would it?

Layer 7 monitoring and shaping is intuitively appealing , but it is a good idea to take a step back and break down examine the full life cycle costs of your methodology .

In an ironic inverse correlation, we assert that costs increase with the complexity of the monitoring tool.

1) Obviously, the more detailed the reporting tool (layer 7 ) , the more expensive its initial price tag.

2)  The kicker comes with part two. The more expensive the tool, the more  detail  it will provide, and the more time an administrator is likely to spend adjusting and mucking, looking for optimal performance.

But, is it a fair to assume higher labor costs with  more advanced monitoring and information?

Well, obviously it would not make sense to pay more for an advanced tool if there was no intention of doing anything with the detailed information it provides. Why have the reporting tool in the first place if the only output was to stare at reports and do nothing? Typically, the more information an admin has about a network, the more inclined he might be to spend time making adjustments.

On a similar note, an oversight often made with labor costs is the belief  that when  the work needed to adjust the network comes to fruition, the associated adjustments can remain statically in place. However, in reality, network traffic changes constantly, and thus the tuning so meticulously performed on Monday may be obsolete by Friday.

Does this mean that the overall productivity of using a bandwidth tool is a loss? Not at all. Bandwidth monitoring and network mucking can certainly result in a cost-effective solution. But, where is the tipping point? When does a monitoring solution create more costs than it saves?

A review of recent history reveals that technologies with a path similar to bandwidth monitoring have become commodities and shunned the overhead of most human intervention.  For example, computer operators disappeared off the face of the earth with the invention of cheaper computing in the late 1980′s.  The function of a computer operator did not disappear completely, it just got automated and rolled into the computer itself. The point is, anytime the cost of a resource is falling, the attention and costs used to manage it should be revisited.

An effective compromise with many of our customers is that they are stepping down from expensive complex reporting tools to a simpler approach. Instead of trying to determine every type of traffic on a network by type, time of day, etc., an admin can spot trouble by simply checking overall usage numbers once a week or so. With a basic bandwidth control solution in place (such as a NetEqualizer), the acute problems of a network locking up will go away, leaving what we would call only “chronic” problems, which may need to be addressed eventually, but do not require immediate action.

For example, with a simple reporting tool you can plot network usage by user.  Such a report, although limited in detail, will often reveal a very distinct bell curve of usage behavior. Most users will be near the mean, and then there are perhaps one or two percent of users that will be well above the mean. You don’t need a fancy tool to see what they are doing; abuse becomes obvious just looking at the usage (a simple report).

However, there is also the personal control factor, which often does not follow clear lines of ROI (return on investment).

What we have experienced when proposing a more hands-off model to network management is that a customer’s comfort depends on their bias for needing to know, which is an unquantifiable personal preference. Even in a world where bandwidth is free, it is still human nature to want to know specifically what bandwidth is being used for, with detailed information regarding the type of traffic. There is nothing wrong with this desire, but we wonder how strong it might be if the savings obtained from using simpler monitoring tools were converted into a trip to Hawaii.

In our next article, we’ll put some real world numbers to the test for actual break downs, so stay tuned. In the mean time, here are some other articles on bandwidth monitoring that we recommend. And, don’t forget to take our poll.

List of monitoring tools compiled by Stanford

Top five free monitoring tools

Planetmy
Linux Tips
How to set up a monitor for free

Posted in Bandwidth Management, Commentary and Editorials, ROI. Tags: , , , . Leave a Comment »

Bandwidth Control Return on Investment (ROI) Calculator

October 13, 2010 — netequalizer

Are you looking to justify the cost of purchasing a bandwidth control device for your Internet or WAN link? Our ROI calculator is Industry neutral, click here to see custom results based on your network.

Aside from our customers’ comments about the overall improvement in their network performance, one of the most common remarks we hear from NetEqualizer users concerns the technology’s positive return on investment (ROI).

However, it’s also one of the most common questions we get from potential customers – How will the NetEqualizer benefit my bottom line?

To better answer this question, we recently interviewed NetEqualizer customers from across several verticals to get their best estimates of the cost savings and value associated with their NetEqualizer. We compiled their answers into a knowledge base that we now use to estimate reasonable ROI calculations.

Our calculations are based on real data and were done conservatively as not to create false promises. There are plenty of congested Internet links suffering out there every day, and hence there is more than enough value with the NetEqualizer. So, we did not need to exaggerate.

ROI calculations were based on the following:

  1. Savings in Bandwidth Costs – Stay at your current bandwidth level or delay future upgrades.
  2. Reduced Labor and Support Costs – Avoid Internet congestion issues that lead to support calls during peak usage times.
  3. Retention of Customers – Stop losing customers, clients, and guests because of unreliable or unresponsive Internet service (applies to ISPs and operators such as hotels and executive suites).
  4. Addition of New Customers – Put more users on your link than before while keeping them all happy.

To see what the NetEqualizer can do for you, visit http://www.netequalizer.com

Other ROI calculators

Posted in Bandwidth Management, Educational Articles, ROI. Tags: , . Leave a Comment »

QoS Over The Internet – Is it possible? Five Must-Know Facts

August 29, 2010 — netequalizer

I had an inquiry from a potential customer yesterday asking if we could monitor their QoS. I was a bit miffed as to what to tell them. At first, the question struck me as if they’d asked if we can monitor electrons on their power grid. In other words, it was a legitimate question in a sense, but of what use would it be to monitor QoS? I then asked him why he had implemented QoS in the first place. How did he know he needed it?

After inquiring a bit deeper, I also found out this customer was using extensive VPNs to remote offices over DSL internet circuits. His WAN traffic from the remote offices was sharing links with regular Internet data traffic, and all of it was traversing the public Internet. Then it hit me – he did not realize his QoS mechanisms were useless outside of his internal network.

Where there is one customer with confusion there are usually others. Hence, I’ve put together a quick fact sheet on QoS over an Internet link. Below, you’ll find five quick facts that should help clarify QoS and answer the primary question of “is it possible over the Internet?”.

Fact #1

If your QoS mechanism involves modifying packets with special instructions (ToS bits) on how it should be treated, it will only work on links where you control both ends of the circuit and everything in between.

Fact #2

Most Internet congestion is caused by incoming traffic. For data originating at your facility, you can certainly have your local router give priority to it on its way out, but you cannot set QoS bits on traffic coming into your network (We assume  from a third party). Regulating outgoing traffic with ToS  bits will not have any effect on incoming traffic.

Fact #3

Your public Internet provider will not treat ToS bits with any form of priority (The exception would be a contracted MPLS type network). Yes, they could, but if they did then everybody would game the system to get an advantage and they would not have much meaning anyway.

Fact #4

The next two facts address our initial question — Is QoS over the Internet possible? The answer is, yes, QoS on an Internet link is possible. We have spent the better part of seven years practicing this art form and it is not rocket science, but it does require a philosophical shift in thinking to get your arms around it.

We call it “equalizing,” or behavior-based shaping, and it involves monitoring incoming and outgoing streams on your Internet link.  Priority or QoS is nothing more than favoring one stream’s packets over another stream’s. You can accomplish priority QoS on incoming streams by queuing (slowing down) one stream over another without relying on ToS bits.

Fact #5

Surprisingly, behavior-based methods such as those used by our NetEqualizer do provide a level QoS for VoIP on the public Internet. Although you can’t tell the Internet to send your VoIP packets faster, most people don’t realize the problem with congested VoIP is due to the fact that their VoIP packets are getting crowded out by large downloads. Often, the offending downloads are initiated by their own employees or users. A good behavior-based shaper will be able to favor VoIP streams over less essential data streams without any reliance on the sending party adhering to a QoS scheme.

For more information, check out Using NetEqualizer To Ensure Clean Clear VOIP.

Posted in Bandwidth Management, Educational Articles. Tags: , , , . 22 Comments »
« Older posts
Newer posts »