NetEqualizer News Blog

What is URL-based Shaping?

URL shaping is the ability to specify the URL, normally a popular site such as YouTube or NetFlix, and set up a fixed-rate limit for traffic to that specific URL.

Is URL shaping just a matter of using a reverse lookup on a URL to get the IP address and plugging it into a bandwidth controller?

In the simplest case, yes, but for sites such as YouTube, the URL of http://www.youtube.com will have many associated IP addresses used for downloading actual videos. Shaping exclusively on the base URL would not be effective.

Is URL shaping the same thing as application shaping?

No. Although similar in some ways, there are significant differences:

1. URL shaping is essentially the same as shaping by a known IP address. The trick with URL shaping is to discover IP addresses associated with a well-known URL.
2. Application shaping uses Deep Packet Inspection (DPI). URL shaping does not. It does not inspect or open customer data.

How to set up URL-based shaping on your NetEqualizer

The following specifications are necessary:

1. NetEqualizer version 4.0 or later
2. A separate Linux-based client such that the client must access the Internet through the NetEqualizer
3. The Perl source code for client URL shaping (listed below) loaded onto a client
4. You will also need to set up your client so that it has permissions to run RSH (remote Shell) commands on your NetEqualizer without requiring a password to be entered. If you do not do this, your Perl discovery routine will hang. The notes for setting up the RSH permissions are outlined below.

How it works…

Save the Perl source code into a .pl file we suggest urlfinder.pl

Make sure to make this file executable

chmod 777 urlfinder.pl

Run the perl command with the following syntax from the command line, where domain.com will be replaced with the specific URL you wish to shape:

./urlfinder.pl http://www.domain.com pool# downlimit uplimit x.x.x.x y.y.y.y

• Pool# is an unused bandwidth pool on your NetEqualizer unit
• Downlimit is the rate in bytes per second incoming for the URL
• Uplimit is the rate bytes per second outgoing to the Internet for the URL
• x.x.x.x is the IP address of your NetEqualizer
• y.y.y.y is the IP address of the client

The script will attempt an http request using http://www.domain.com. It will then continue to do recursive Web accesses on subsequent links starting on the main domain URL. It will stop when there are no more links to follow or when 150 pages have been accessed. Any foreign IP’s found during the access session will be put into the given bandwidth pool as CLASS B masks, and will immediately be forever shaped until you remove the pool.

Notes:

In our beta testing, the script did well in finding YouTube subnets used for videos.  We did not confirm whether the main NetFlix home page URL shares IP subnets with their download sites.

Notes for setting up RSH

Begin Notes

These notes  assume you are either logged in on the Client as root or you use sudo -i and are acting as root.

192.168.1.143 is used in the example as the Server (NetEq) IP.

On your Client machine, do:

• ssh-keygen -t rsa -b 4096
• ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.143
• nano -w /etc/ssh/ssh_config

Make sure that these are as follows:

• RhostsRSAAuthentication yes
• RSAAuthentication yes
• EnableSSHKeysign yes
• HostbasedAuthentication yes

The next line is all one line to the ssh_known_hosts

• scp /etc/ssh/ssh_host_rsa_key.pub root@192.168.1.143:/etc/ssh/ssh_known_hosts

The next line is all one line to the ssh_known_hosts2

• scp /etc/ssh/ssh_host_rsa_key.pub root@192.168.1.143:/etc/ssh/ssh_known_hosts2

Now, find out your HOSTNAME on the Client:

• echo $HOSTNAME

On the Server machine, do:

• nano -w /etc/hosts.equiv
• harry-lin root
• my $HOSTNAME of the Client was harry-lin
• nano -w /etc/ssh/sshd_config

Check the following:

• PermitRootLogin yes
• StrictModes yes
• RSAAuthentication yes
• PubkeyAuthentication yes
• AuthorizedKeysFile %h/.ssh/authorized_keys
• IgnoreRhosts no
• RhostsRSAAuthentication no
• HostbasedAuthentication yes

Now do:

• chown root:root /root

Then:

• /etc/init.d/ssh reload

Now you can try something like this from your Client:

• ssh root@192.168.1.143

If it doesn’t work, then do the following, which gives you details if possible:

• ssh -v root@192.168.1.143

Final Notes: While support for this utility is NOT currently included with your NetEqualizer, we will assist any customers with a current Network Software Subscription for up to one hour. For additional support, consulting fees may apply.