What is URL-based Shaping?
URL shaping is the ability to specify the URL, normally a popular site such as YouTube or NetFlix, and set up a fixed-rate limit for traffic to that specific URL.
Is URL shaping just a matter of using a reverse lookup on a URL to get the IP address and plugging it into a bandwidth controller?
In the simplest case, yes, but for sites such as YouTube, the URL of http://www.youtube.com will have many associated IP addresses used for downloading actual videos. Shaping exclusively on the base URL would not be effective.
Is URL shaping the same thing as application shaping?
No. Although similar in some ways, there are significant differences:
- URL shaping is essentially the same as shaping by a known IP address. The trick with URL shaping is to discover IP addresses associated with a well-known URL.
- Application shaping uses Deep Packet Inspection (DPI). URL shaping does not. It does not inspect or open customer data.
How to set up URL-based shaping on your NetEqualizer
The following specifications are necessary:
- NetEqualizer version 4.0 or later
- A separate Linux-based client such that the client must access the Internet through the NetEqualizer
- The Perl source code for client URL shaping (listed below) loaded onto a client
- You will also need to set up your client so that it has permissions to run RSH (remote Shell) commands on your NetEqualizer without requiring a password to be entered. If you do not do this, your Perl discovery routine will hang. The notes for setting up the RSH permissions are outlined below.
How it works…
Save the Perl source code into a .pl file we suggest urlfinder.pl
Make sure to make this file executable
chmod 777 urlfinder.pl
Run the perl command with the following syntax from the command line, where domain.com will be replaced with the specific URL you wish to shape:
./urlfinder.pl http://www.domain.com pool# downlimit uplimit x.x.x.x y.y.y.y
- Pool# is an unused bandwidth pool on your NetEqualizer unit
- Downlimit is the rate in bytes per second incoming for the URL
- Uplimit is the rate bytes per second outgoing to the Internet for the URL
- x.x.x.x is the IP address of your NetEqualizer
- y.y.y.y is the IP address of the client
The script will attempt an http request using http://www.domain.com. It will then continue to do recursive Web accesses on subsequent links starting on the main domain URL. It will stop when there are no more links to follow or when 150 pages have been accessed. Any foreign IP’s found during the access session will be put into the given bandwidth pool as CLASS B masks, and will immediately be forever shaped until you remove the pool.
Notes:
In our beta testing, the script did well in finding YouTube subnets used for videos. We did not confirm whether the main NetFlix home page URL shares IP subnets with their download sites.
Notes for setting up RSH
Begin Notes
These notes assume you are either logged in on the Client as root or you use sudo -i and are acting as root.
192.168.1.143 is used in the example as the Server (NetEq) IP.
On your Client machine, do:
- ssh-keygen -t rsa -b 4096
- ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.143
- nano -w /etc/ssh/ssh_config
Make sure that these are as follows:
- RhostsRSAAuthentication yes
- RSAAuthentication yes
- EnableSSHKeysign yes
- HostbasedAuthentication yes
The next line is all one line to the ssh_known_hosts
- scp /etc/ssh/ssh_host_rsa_key.pub root@192.168.1.143:/etc/ssh/ssh_known_hosts
The next line is all one line to the ssh_known_hosts2
- scp /etc/ssh/ssh_host_rsa_key.pub root@192.168.1.143:/etc/ssh/ssh_known_hosts2
Now, find out your HOSTNAME on the Client:
- echo $HOSTNAME
On the Server machine, do:
- nano -w /etc/hosts.equiv
- harry-lin root
- my $HOSTNAME of the Client was harry-lin
- nano -w /etc/ssh/sshd_config
Check the following:
- PermitRootLogin yes
- StrictModes yes
- RSAAuthentication yes
- PubkeyAuthentication yes
- AuthorizedKeysFile %h/.ssh/authorized_keys
- IgnoreRhosts no
- RhostsRSAAuthentication no
- HostbasedAuthentication yes
Now do:
- chown root:root /root
Then:
- /etc/init.d/ssh reload
Now you can try something like this from your Client:
- ssh root@192.168.1.143
If it doesn’t work, then do the following, which gives you details if possible:
- ssh -v root@192.168.1.143
Final Notes: While support for this utility is NOT currently included with your NetEqualizer, we will assist any customers with a current Network Software Subscription for up to one hour. For additional support, consulting fees may apply.
Leave a Reply