Editor’s Note: It was a long road to get here, and for those following in our footsteps or just curious on the intricacies of YouTube caching, we have laid open the details.
This evening, I’m burning the midnight oil. I’m monitoring Internet link statistics at a state university with several thousand students hammering away on their residential network. Our bandwidth controller, integrated with Squid caching, has been running continuously for several days and all is stable. From the stats I can see, about 1,000 YouTube videos have been played out of the local cache over the past several hours. Without the caching feature installed, most of the YouTube videos would have played anyway, but there would be interruptions as the Internet link coughed and choked with congestion. Now, with caching running smoothly, the most popular videos will run without interruptions.
Getting to this stable product was a long and winding road. Here’s how we got there.
First, some background information on the initial problem.
To use a Squid proxy server, your network administrator must put hooks in your router so that all Web requests go the Squid proxy server before heading out to the Internet. Sometimes the Squid proxy server will have a local copy of the requested page, but most of the time it won’t. When a local copy is not present, it sends your request on to the Internet to get the page (for example the Yahoo! home page) on your behalf. The squid server will then update a local copy of the page in its cache (storage area) while simultaneously sending the results back to you, the original requesting user. If you make a subsequent request to the same page, the Squid will quickly check it to see if the content has been updated since it stored away the first time, and if it can, it will send you a local copy. If it detects that the local copy is no longer valid (the content has changed), then it will go back out to the Internet and get a new copy.
Now, if you add a bandwidth controller to the mix, things get interesting quickly. In the case of the NetEqualizer, it decides when to invoke fairness based on the congestion level of the Internet trunk. However, with the bandwidth controller unit (BCU) on the private side of the Squid server, the actual Internet traffic cannot be distinguished from local cache traffic. The setup looks like this:
Internet->router->Squid->bandwidth controller->users
The BCU in this example won’t know what is coming from cache and what is coming from the Internet. Why? Because the data coming from the Squid cache comes over the same path as the new Internet data. The BCU will erroneously think all the traffic is coming from the Internet and will shape cached traffic as well as Internet traffic, thus defeating the higher speeds provided by the cache.
In this situation, the obvious solution would be to switch the position of the BCU to a setup like this:
Internet->router->bandwidth controller->Squid->users
This configuration would be fine except that now all the port 80 HTTP traffic (cached or not) will appear like it is coming from the Squid proxy server and your BCU will not be able to do things like put rate limits on individual users. Fortunately, with the our NetEqualizer 5.0 release, we’ve created an integration with NetEqualizer and co-resident Squid such that everything works correctly. (The NetEqualizer still sees and acts on all traffic as if it were between the user and the Internet. This required some creative routing and actual bug fixes to the bridging and routing in the Linux kernel. We also had to develop a communication module between the NetEqualizer and the Squid server so the NetEqualizer gets advance notice when data is originating in cache and not the Internet.)
At this point, you may be wondering if Squid caching is so great, why not just dump the BCU and be done with the complexity of trying to run both? Well, while the Squid server alone will do a fine job of accelerating the access times of large files such as video when they can be fetched from cache, a common misconception is that there is a big relief on your Internet pipe with the caching server. This has not been the case in our real world installations.
The fallacy for caching as panacea for all things congested assumes that demand and overall usage is static, which is a unrealistic. The cache is of finite size and users will generally start watching more YouTube videos when they see improvements in speed and quality (prior to Squid caching, they might have given up because of slowness), including videos that are not in cache. So, the Squid server will have to fetch new content all the time, using additional bandwidth and quickly negating any improvements. Therefore, if you had a congested Internet pipe before caching, you will likely still have one afterward, leading to slow access for many e-mail, Web chat and other non-cacheable content. The only solution is to include a bandwidth controller in conjunction with your caching server. This is what NetEqualizer 5.0 now offers.
In no particular order, here is a list of other useful information — some generic to YouTube caching and some just basic notes from our engineering effort. This documents the various stumbling blocks we had to overcome.
1. There was the issue of just getting a standard Squid server to cache YouTube files. It seemed that the URL tags on these files change with each access, like a counter, and a normal Squid server is fooled into believing the files have changed. By default, when a file changes, a caching server goes out and gets the new copy. In the case of YouTube files, the content is almost always static. However, the caching server thinks they are different when it sees the changing file names. Without modifications, the default Squid caching server will re-retrieve the YouTube file from the source and not the cache because the file names change. (More on caching YouTube with Squid)
2. We had to move to a newer Linux kernel to get a recent of version of Squid (2.7) which supports the hooks for YouTube caching. A side effect was that the new kernel de-stabalized some of the timing mechanisms we use to implement bandwidth control. These subtle bugs were not easily reproduced with our standard load generation tools, so we had to create a new simulation lab capable of simulating thousdands of users accessing the Internet and YouTube at the same time. Once we built this lab, we were able to re-create the timing issues in the kernel and have them patched.
3. It was necessary to set up a firewall re-direct (also on the NetEqualizer) for port 80 traffic back to the Squid server. This configuration, and the implementation of an extra bridge, were required to get everything working. The details of the routing within the NetEqualizer were customized so that we would be able to see the correct IP addresses of Internet sources and users when shaping. (As mentioned above, if you do not take care of this, all IPs (traffic) will appear as if they are coming from the Proxy server.
4. The firewall has a table called ConnTrack (not be confused with NetEqualizer connection tracking but similar). The connection tracking table on the firewall tends to fill up and crash the firewall, denying new requests for re-direction if you are not careful. If you just go out and make the connection table randomly enormous that can also cause your system to lock up. So, you must measure and size this table based on experimentation. This was another reason for us to build our simulation lab.
5. There was also the issue of the Squid server using all available Linux file descriptors. Linux comes with a default limit for security reasons, and when the Squid server hit this limit (it does all kinds of file reading and writing keeping descriptors open), it locks up.
6. Some of the tuning changes we made to support YouTube caching were:
a. To limit the file size of a cached object of 2 megabits to 40 megabits
- minimum_object_size 2000000 bytes
- maximum_object_size 40000000 bytes
If you allow smaller cached objects it will rapidly fill up your cache and there is little benefit to caching small pages.
b. We turned off the Squid keep reading flag
- quick_abort_min 0 KB
- quick_abort_max 0 KB
This flag when set continues to read a file even if the user leave the page , for example when watching a video if the user aborts on their browser the Squid cache continues to read the file. I suppose this could now be turned back on but during testing it was quite obnoxious to see data transfers talking place to the squid cache when you thought nothing was going on.
c. We also explicitly told the Squid what DNS servers to use in its configuration file. There was some evidence that without this the Squid server may bog down, but we never confirmed it. However, no harm by setting these parameters
d. You have to be very careful to set the cache size not to exceed your actual capacity. Squid is not smart enough to check your real capacity, so it will fill up your file system space if you let it, which in turn causes a crash. When testing with small RAM disks less than four gigs of cache, we found that the Squid logs will also fill up your disk space and cause a lock up. The logs are refreshed once a day on a busy system. With a large amount pages being accessed, the log will use close to a gig of data quite easily, and then to add insult to injury, the log back up program makes a back up. On a normal-sized cacheing system there should be ample space for logs
e. The Squid has a short-term buffer not related to caching. It is just a buffer where it stores data from the Internet before sending it to the client. Remember all port 80 (HTTP) requests go through the squid, cached or not, and if you attempt to control the speed of a transfer between the Squid and the user, it does not mean that the Squid server slows the rate of the transfer coming from the Internet right away. With the BCU in line, we want the sender on the Internet to back off right away if we decide to throttle the transfer, and with the Squid buffer in between the NetEqualizer and the sending host on the Internet, the sender would not respond to our deliberate throttling right away when the buffer was too large (Link to Squid caching parameter).
f. How to determine the effectiveness of your YouTube caching statistics?
I use the Squid client cache statistics page. Down at the bottom there is a entry that lists hits verses requests.
TOTALS
- ICP : 0 Queries, 0 Hits (0%)
- HTTP: 21990877 Requests, 3812 Hits (0%)
At first glance, it may appear that the hit rate is not all that effective, but let’s look at these stats another way. A simple HTTP page generates about 10 HTTP requests for perhaps 80K bytes of data total. A more complex page may generate 500k. For example, when you go to the CNN home page there are quite a few small links, and each link increments the HTTP counter. On the other hand, a YouTube hit generates one hit for about 20 megabits of data. So, if I do a little math based on bytes cached we get, the summary of HTTP hits and requests above does not account for total data. But, since our cache is only caching Web pages from two megabits to 40 megabits, with an estimated average of 20 megabits, this gives us about 400 gigabytes of regular HTTP and 76 Gigabytes of data that came from the cache. Abut 20 percent of all HTTP data came from cache by this rough estimate, which is a quite significant.
Like this:
Be the first to like this post.
'
NetEqualizer News Special Feature: Technology and Other Predictions for 2012 and Beyond
July 13, 2011 — netequalizerAs we pass the midpoint of 2011, it’s time to start making a few predictions about the year to come. So keep an eye out for these developments over the next 18 months. If we’re right, be sure to give us credit. If we’re wrong, just act like this post never happened. Here are our thoughts…
Prediction #1: Apple or a new player will make a splash in the search engine market. Current search engine technology, although thorough and expansive, tends to be lacking in smarts. How many times have you searched for a page or link that you know for sure is out there, and despite your best efforts of crafting your key words, Google or Yahoo can’t find what you are looking for? Sometimes, unless you know the exact context of a sentence, in correct word order, you just can’t find it. And that leaves room for improvement.
This is not a knock on Google, Yahoo! or Bing, per se, but rather just an observation that there is room for another generation of search engine and somebody is going to do it sooner rather than later. However, we expect the next-generation search engine will sacrifice speed for intelligence. By this we mean that it is likely the newer generation may crank for 20 seconds to find what you are looking for, but the slower speeds will be more than compensated for by the better, more relevant results. New search engine technology will take the market by storm because of more useful content.
The reason why we suspect Apple might solve this puzzle is that Steve Jobs has a habit of leap frogging technology and bringing it to market. Google has grown by acquisition and not so much by innovation. If not Apple, then it might also come out of left field at some graduate research lab. Regardless, we think it will happen.
Prediction #2: There will be a tumble in the social networking and search engine stock bubble. The expectations for advertisement revenue will not pan out. Placement ads are just too easy to ignore on the Internet. These sites do not have the captive audience of the super bowl, and advertisers are starting to figure that out.
There will be price pressure on the content sites and search engine sites to lower costs to attract advertisers as they actually start to measure and go public with their returns on advertising investment. There will be quite a bit of pressure to hide this fact in the media, as there is now, but at some point content advertising revenues ROI will bare this out.
We are not predicting a collapse in this market, but just some major adjustments to valuations. This is based on our six years of experience placing online ads. Prices have gone up and results were never there to justify cost.
Related Article: Facebook Valuation Too High
Related Article: Demand Builds for TV Ad Time
Prediction #3: Fuel prices will plummet as the Chinese and Indian economies cool down.
Although oil production and exploration is flat in the US, every other country around the world is picking up exploration and exploiting new reserves. The market will be flooded with oil by mid or late 2012, sending the price of gasoline back down to $2 or below.
Prediction #4: There will be a new resurgence in urban mesh networks.
Why? These things really do enhance economic activity. The initial round of municipal mesh networks was a learning experiment with some limited success and way too much inexperience in sourcing providers.
The real reason for cities to invest in these networks will be the growing monthly fees with 4G devices that traditional providers are charging to cover the cost of their larger networks. Users will gravitate toward areas where they can switch over to free wireless. A well-covered downtown or small city with free wireless service will be a welcome island for business users and consumers alike. Think of it like a stepping inside a circle where you can make free unlimited long distance calls while the rest of the provider networks gouge you when step outside.
We’ll see how these predictions pan out. As always, feel free to share your thoughts on our predictions, or some predictions of your own, in the comments section below.
In a related article, the WSJ reports Wi-fi is the largest provider for Mobile Devices such as the iPhone.
Share this:
Like this: