Shaping Bandwidth by VLAN under the NetEqualizer Hood


As a followup to my recent commentary on  the history of VLAN tags, I decided to jump down into the guts of a bandwidth shaper and go over some of the techniques we use to set rate limits on a particular VLAN. When writing, I assumed the reader has a basic understanding of how data can be manipulated inside a computer program.

Let’s start with some background information. First off, the NetEqualizer bandwidth shaper is a transparent bridge.  A typical setup has two Ethernet cards —  one connected to your LAN and the other side connected to your WAN (Internet router). Before we added in our VLAN shaping, the Linux kernel bridging code would blindly transfer Ethernet packets from one side to the other, passing right through the NetEqualizer.

As these Ethernet packets pass through, they’re visible as data in the Linux kernel. Normally, they pass through unmolested — in one side out the other. However, the key to bandwidth shaping is what you do with them as they come through.

To give you a better idea of what goes inside the Linux kernel when data passes through, I’ve included a couple of snippets of C code below. This is actual Linux kernel code. I have also littered the code with some detailed explanations in line, so you don’t have to understand C to follow the logic.

Below is the C language data definition of the fields in an Ethernet header. When an Ethernet packet comes across the NetEqualizer, the contents of the Ethernet packet are put into data structures. The reason why we’re interested in the Ethernet header is that it’s where the VLAN tags are located.

Note: Code appears in italics while notes are in bold and non-italicized font.

struct vlan_ether_header {
char dst[6];     // This is six bytes for the destination MAC address.
char src[6];     // This is six bytes for the source MAC address.
short type;
short tci_vid;
short encapsulated_type;
} __attribute__ ((__packed__));

Below is the C function that finds the actual VLAN tag inside the Ethernet header in an Ethernet packet.

struct iphdr* findIph(struct sk_buff* skb, int *vlan_id) {
struct ethhdr* eh;    

// This is a pointer to a data structure of type ether net header. We first declare the pointer and will assign it later.
struct iphdr* iph = NULL;   

// This is a pointer to a data structure that contains the IP header of an IP packet (I did not show the definition of the structure).
*vlan_id = -1;                         

// Set the VLAN ID to something.

eh = (struct ethhdr*)(skb->mac_header);

/*  The SKB buffer is the standard structure for network data being passed around the kernel. It contains all the data related to IP data  including the Ethernet packet. Part of the Ethernet packet is the MAC header which is what we are interested in to find out the VLAN ID. FYI . . . SKB is the buffer that IP tables routinely use. To enforce firewall rules, they pass this buffer from rule-to-rule because everybody needs to look inside of it to decide what to do. I am not going to go into how it came into existence. Suffice to say the Ethernet packet is located in this buffer. The MAC header is a field in the SKB buffer and the above assignment copies this location to the variable eh, which is the pointer of an Ethernet header. We now have a data structure that we can access to see fields inside the Ethernet header as a packet passes through the NetEqualizer */

if (eh->h_proto == 0x0081) {
struct vlan_ether_header* veh = (struct vlan_ether_header*)(skb->mac_header);

if (veh->encapsulated_type == 0x0008) {
iph = (struct iphdr*)(skb->mac_header + sizeof(*veh));
*vlan_id = ((ntohs(veh->tci_vid)) & 0x0fff);
// BR_DEBUG_IP printk (KERN_INFO “got VLAN ID %d \n”, *vlan_id);
}
}

/* The above code snippet is where the actual VLAN ID gets put into the variable vlan_id. The FFF is a bit mask which slices the value of the VLAN ID out of the field tci_vid. It is a 12-bit number */
else {
if (eh->h_proto == 0x0008) {
iph = (struct iphdr*)(skb->mac_header + sizeof(*eh));
}
}
return iph;
}

Hopefully the code captured the spirit of the type of work that goes on in the Linux kernel to analyze packets. But, how does VLAN shaping work once you have the VLAN ID?

Well, once we have the VLAN ID of a packet, we check and see if there is a VLAN shaping rule in effect for that ID. There is a table in the Kernel with a list of all of the active VLAN shaping rules that have been specified by the user. If there is a rule for this VLAN, a counter is incrimented for the number of data bytes in the payload of the IP packet.

if (vlan_id > -1  && vlan_id < VLAN_MAX && hard_table[vlan_id + HARD_SIZE].ip == vlan_id && port_id ==2) {
                hard_table[vlan_id + HARD_SIZE].incount=hard_table[vlan_id +HARD_SIZE].incount +hsize;

The code snippet above checks to make sure the VLAN ID is valid and then it increments the byte count for that VLAN. hsize is a variable that contains the actual number of data bytes in the Ethernet packet.

The NetEqualizer keeps this counter for an entire second (it will reset it each second), and if the data coming in for the VLAN is coming in faster than the rate limit defined by a user rule for that particular VLAN ID, then the NetEqualizer will take action by actually slowing down the packet in the kernel. This in turn reduces the data rate of transfer for the VLAN.

VLAN tags made simple


By Art Reisman, CTO, www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Why am I writing a post on VLAN tags ?

VLAN tags and Bandwidth Control are often intimately related, but before I can post on the relationship I thought it prudent to comment on VLAN tags, I definitly think they are way over used and hope to comment on that also in a future post.

I generally don’t like VLAN tags, the original idea behind them was to solve the issue with  Ethernet broadcasts saturating network segment. Wikipedia explains it like this…

After successful experiments with voice over Ethernet from 1981 to 1984, Dr. W. David Sincoskie joined Bellcore and turned to the problem of scaling up Ethernet networks. At 10 Mbit/s, Ethernet was faster than most alternatives of the time; however, Ethernet was a broadcast network and there was not a good way of connecting multiple Ethernets together. This limited the total bandwidth of an Ethernet network to 10 Mbit/s and the maximum distance between any two nodes to a few hundred feet.

What does that mean and why do you care?

First lets address how an Ethernet broadcast works and then we can discuss Dr Sincoskies solution and make some sense of it.

When a bunch of computers share a single Ethernet segment of a network separated by switches everybody can hear each other talking

Think of 2 people in a room yelling back and forth to communicate, that might work if one person pauses after each yell to give the other person a chance to yell back.  Now if you had three people in a room they can still yell at each other and pause and listen for other people yelling and that might still work, but if you had 1000 people in the room and they are trying to talk to people on the other side of the room the pausing technique waiting for other people to talk does not work very well.  And that is exactly the problem with Ethernet as it grows everybody is trying to talk on the same wire at once.  VLAN tags work by essentially creating a bunch of smaller virtual  rooms where only the noise and yelling from the people in the virtual room can be heard at one time.

Now when you set up a VLAN tag (virtual room ) you have to put up the dividers. On a network this is done by having  the switches, the things the computers plug into,  be aware of what virtual room each computer is in. The Ethernet tag specifies the identifier for the virtual room and so once set up you have a bunch of virtual rooms and everybody can talk.

This sort of begs the question

Does everybody attached to the Internet live in a virtual room ?

No virtual rooms  (VLANs) were needed so a single organization like a company can put a box around their network segments to protect them with a common set of access rules ( firewall router), the Internet works fine without VLAN tags.

So a VLAN tag is only appropriate when a group of users sit behind a common router ?

Yes that is correct , Ethernet broadcasts ( yelling  as per our analogy) do not cross cross router boundaries on the Internet.

Routers handle public IP addresses to figure out where to send things. A router does not use broadcast (yelling), it is much more discrete , it only sends on data to another router if it knows that the data is supposed to go there.

So why do we have two mechanisms one for  local computers sending Ethernet broadcasts and another for routers using point to point routing ?

This post was supposed to be about VLAN tags….. I’ll take it one step further to explain the difference.

Perhaps you have heard about the layers of networking, layer 2 is Ethernet and Layer 3 is IP.

Answers.com gave me the monologue below, which is technically correct, but does not really make much sense unless you already had a good understanding of networking in the first place , so I’ll finish by breaking down this into something a little more relevant with some in-line comments.

Basically a layer 2 switch operates utilizing Mac addresses in it’s caching table to quickly pass information from port to port. A layer 3 switch utilizes IP addresses to do the same.

What this means is that an Ethernet switch looks at MAC addresses which are used by your router for local addressing to a computer on your network. Think back to people shouting in the room to communicate, the MAC address would be a Nick name that only their closest friends would use when they shout at each other. At the head end of your network is a router, this is where you connect to the Internet, and other Internet users send data to you from your IP address and this is essentially the well known public address at your router. The IP address could be thought of as the address of the building where everybody is inside shouting at each other. The routers job is to get information,sent by IP address  destined for some body inside the room to the door. If you are a Comcast home user you likely have a Modem where you cable plugs in the Modem is the gateway to your house and is addressed by IP address by the outside world.


Essentially, A layer 2 switch is essentially a multiport transparent bridge. A layer 2 switch will learn about MAC addresses connected to each port and passes frames marked for those ports.

The above paragraph is referring to how an Ethernet switch sends data around, everybody in room registers their Nick-Name to the switch so it can shout in the direction of the person in the room when new data comes in.

It also knows that if a frame is sent out a port but is looking for the MAC address of the port it is connected to and drop that frame. Whereas a single CPU Bridge runs in serial, todays hardware based switches run in parallel, translating to extremly fast switching.


I left this paragraph in because it is completely unrelated to the question I asked that Answers.com responded to, so ignore it. This is  a commentary about how modern switches can be reading and sending from multiple interfaces at the same time.

Layer 3 switching is a hybrid, as one can imagine, of a router and a switch. There are different types of layer 3 switching, route caching andtopology-based. In route caching the switch required both a Route Processor (RP) and a Switch Engine (SE). The RP must listen to the first packet to determine the destination. At that point the Switch Engine makes a shortcut entry in the caching table for the rest of the packets to follow.

More random stuff unrelated to the question “What is the difference between layer 3 and layer 2 ”

Due to advancement in processing power and drastic reductions in the cost of memory, today’s higher end layer 3 switches implement a topology-based switching which builds a lookup table and and poputlates it with the entire network’s topology. The database is held in hardware and is referenced there to maintain high throughput. It utilizes the longest address match as the layer 3 destination.

This is talking about how a Router translates between the local address Nick-Name of people yelling in the room and the public address of data leaving the building.
Now when and why would one use a l2 vs l3 vs a router? Simply put, a router will generally sit at the gateway between a private and a public network. A router can performNAT whereas an l3 switch cannot (imagine a switch that had the topology entries for the ENTIRE Internet!!).

NetEqualizer Software Update Improves VLAN Shaping, NTOP


Editor’s Note: The following blog entry explains the newest NetEqualizer features available with our most recent software update. While minor bug fixes are often included in these updates, they will not always be detailed.

We recently released our newest NetEqualizer software update, further improving on our existing technology. The following fixes have been implemented from the the previous 2.43k version to the latest 3.32a.

  1. Upgraded internal disk memory caching. This feature remedied an issue with NTOP that was causing disk corruptions on the CF drive.
  2. Subnet masking was modified such that masked traffic will not count against your license level. Prior to this change, a customer with a 10-meg license who ran 100 meg local transfers across their NetEqualizer would experience a license violation. You can now mask that traffic (make it invisible to the NetEqualizer and hence not violate your license).
  3. A bug fix was put in for customers who run asymmetric pools. Bandwidth pools with different upload and download speeds were not working correctly prior to this fix.
  4. VLAN shaping fix. There was an issue on cold restarts.
  5. Support for multi-core CPU
  6. More efficient connection limit processing

This software update is available without charge for NetEqualizer customers with a current NetEqualizer Software Subscription (NSS). For more information on this update, or the NSS, contact us at admin@apconnections.net.


%d bloggers like this: