NetEqualizer Testing and Integration of Squid Caching Server

Editor’s Note: Due to the many variables involved with tuning and supporting Squid Caching Integration, this feature will require an additional upfront support charge. It will also require at minimum a NE3000 platform. Contact for specific details.

In our upcoming 5.0 release, the main enhancement will be the ability to implement YouTube caching from a NetEqualizer. Since a squid-caching server can potentially be implemented separately by your IT department, the question does come up about what the difference is between using the embedded NetEqualizer integration and running the caching server stand-alone on a network.

Here are a few of the key reasons why using the NetEqualizer caching integration provides for the most efficient and effective set up:

1. Communication – For proper performance, it’s important that the NetEqualizer know when a file is coming from cache and when it’s coming from the Internet. It would be counterproductive to have data from cache shaped in any way. To accomplish this, we wrote a new utility, aptly named “cache helper,” to advise the NetEqualizer of current connections originating from cache. This allows the NetEqualizer to permit cached traffic to pass without being shaped.

2. Creative Routing – It’s also important that the NetEqualizer be able to see the public IP addresses of traffic originating on the Internet. However, using a stand-alone caching server prevents this. For example, if you plug a caching server into your network in front of a NetEqualizer (between the NetEqualizer and your users), all port 80 traffic would appear to come from the proxy server’s IP address. Cached or not, it would appear this way in a default setup. The NetEqualizer shaping rules would not be of much use in this mode as they would think all of the Internet traffic was originating from a single server. Without going into details, we have developed a set of special routing rules to overcome this limitation in our implementation.

3. Advanced Testing and Validation – Squid proxy servers by themselves are very finicky. Time and time again, we hear about implementations where a customer installed a proxy server only to have it cause more problems than it solved, ultimately slowing down the network. To ensure a simple yet tight implementation, we ran a series of scenarios under different conditions. This required us to develop a whole new methodology for testing network loads through the Netequalizer. Our current class of load generators is very good at creating a heavy load and controlling it precisely, but in order to validate a caching system, we needed a different approach. We needed a load simulator that could simulate the variations of live internet traffic. For example, to ensure a stable caching system, you must take the following into consideration:

  • A caching proxy must perform quite a large number of DNS look-ups
  • It must also check tags for changes in content for cached Web pages
  • It must facilitate the delivery of cached data and know when to update the cache
  • The squid process requires a significant chunk of CPU and memory resources
  • For YouTube integration, the Squid caching server must also strip some URL tags on YouTube files on the fly

To answer this challenge, and provide the most effective caching feature, we’ve spent the past few months developing a custom load generator. Our simulation lab has a full one-gigabit connection to the Internet. It also has a set of servers that can simulate thousands of simultaneous users surfing the Internet at the same time. We can also queue up a set of YouTube users vying for live video from the cache and Internet. Lastly, we put a traditional point-to-point FTP and UDP load across the NetEqualizer using our traditional load generator.

Once our custom load generator was in place, we were able to run various scenarios that our technology might encounter in a live network setting.  Our testing exposed some common, and not so common, issues with YouTube caching and we were able to correct them. This kind of analysis is not possible on a live commercial network, as experimenting and tuning requires deliberate outages. We also now have the ability to re-create a customer problem and develop actual Squid source code patches should the need arise.

4 Responses to “NetEqualizer Testing and Integration of Squid Caching Server”

  1. NetEqualizer News: March 2011 « NetEqualizer News Blog Says:

    […] Top Posts The Dark Side of Net NeutralityNetEqualizer YouTube Caching FAQSupport ArchivesNine Tips And Tricks To Speed Up Your Internet ConnectionNetEqualizer Testing and Integration of Squid Caching Server […]

  2. Mario Crespo Says:

    Hi, i have neteq betwen squid cache and users for a few years.
    My squid cache is setup to do NAT and trasnparent proxy.
    My neteq see connections betwen private ip addres and internet address.
    I can control connections, speed and have mask private squid proxy address in neteq for cache served content don´t be penanlized never.
    What kind of issues i will be there. What thing i can do for improve this setup.

  3. NetEqualizer News: April 2011 « NetEqualizer News Blog Says:

    […] previously discussed, we’ve developed an embedded caching capability for YouTube videos that runs directly on your NetEqualizer (NE3000 or above only). When the NCO is […]

  4. Setting up a Squid Proxy Caching co-resident with bandwidth controller. « NetEqualizer News Blog Says:

    […] The Firewall has a table called ConnTrack  not be confused with NetEqualizer connection tracking but similar.  The connection tracking table on the firewall tends to fill up and crash the firewall denying new requests for re-direction if you are not careful.  If you just go out and make the connection table randomly enormous that can also cause your system to lock up , so you must measure and size this table based on experimentation. Another reason to build our simulation lab. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: