Do We Need an Internet User Bill of Rights?


The Computers, Freedom and Privacy conference wraps up today in Washington, D.C., with conference participants having paid significant attention to the on-going debates concerning ISPs, Deep Packet Inspection and net neutrality.  Over the past several days, representatives from the various interested parties have made their cases for and against certain measures pertaining to user privacy. As was expected, demands for the protection of user privacy often came into conflict with ISPs’ advertising strategies and their defense of their overall network quality.

At the center of this debate is the issue of transparency and what ISPs are actually telling customers. In many cases, apparent intrusions into user privacy are qualified by what’s stated in the “fine print” of customer contracts. If these contracts notify customers that their Internet activity and personal information may be used for advertising or other purposes, then it can’t really be said that the customer’s privacy has been invaded. But, the question is, how many users actually read their contracts, and furhtermore, how many people actually understand the fine print? It would be interesting to see what percentage of Internet users could define deep packet inspection. Probably not very many.

This situation is reminiscent of many others involving service contracts, but one particular timely example comes to mind — credit cards. Last month, the Senate passed a credit card “bill of rights,” through which consumers would be both better protected and better informed. Of the latter, President Obama stated, “you should not have to worry that when you sign up for a credit card, you’re signing away all your rights. You shouldn’t need a magnifying glass or a law degree to read the fine print that sometimes doesn’t even appear to be written in English.”

Ultimately, the same should be true for any service contracts, but especially if private information is at stake, as is the case with the Internet privacy debate. Therefore, while it’s a step in the right direction to include potential user privacy issues in service contracts, it should not be done only with the intention of preventing potential legal backlash, but rather with the customer’s true understanding of the agreement in mind.

Editor’s Note: APconnections and NetEqualizer have long been a proponent of both transparency and the protection of user privacy, having devoted several years to developing technology that maintains network quality while respecting the privacy of Internet users.

Obama’s Revival of Net Neutrality Revisits An Issue Hardly Forgotten


Last Friday, President Obama reinvigorated (for many people, at least) the debate over net neutrality during a speech from the White House on cybersecurity. The president made it clear that users’ privacy and net neutrality would not be threatened under the guise of cybersecurity measures. President Obama stated:

“Let me also be clear about what we will not do. Our pursuit of cyber-security will not — I repeat, will not include — monitoring private sector networks or Internet traffic. We will preserve and protect the personal privacy and civil liberties that we cherish as Americans. Indeed, I remain firmly committed to net neutrality so we can keep the Internet as it should be — open and free.”

While this is certainly an important issue on the security front, for many ISPs and networks administrators, it didn’t take the president’s comments to put user privacy or net neutrality back in the spotlight.  In may cases, ISPs and network administrators constantly must walk the fine line between net neutrality, user privacy, and ultimately the well being of their own networks, something that can be compromised on a number of fronts (security, bandwidth, economics, etc.).

Therefore, despite the president’s on-going commitment to net neturality, the issue will continue to be debated and remain at the forefront of the minds of ISPs, administrators, and many users. Over the past few years, we at NetEqualizer have been working to provide a compromise for these interested parties, ensuring network quality and neutrality while protecting the privacy of users. It will be interesting to see how this debate plays out, and what it will mean for policy, as the philosophy of network neutrality continues to be challenged — both by individuals and network demands.

Further Reading

Top Six Fear-Driven Network Equipment Purchases


Fear is one of our most primal survival instincts.  But, as such, sales people around the world have made a business out of selling their products on fear and making  them out to be a necessity for survival. Below, we will highlight some of the current and historical fear-based triggers used to push oftentimes unneeded items with respect to the networking industry.

1) CALEA compliance — A little over a year ago, we were besieged by frantic inquiries from many of our ISP customers about the need to do something for the new CALEA laws.  Basically, these are laws that require data carriers to provide access to law enforcement agencies upon receipt of a judge’s order.

We spent the next few months researching what the intent of the CALEA laws were, and what that meant to our customers.   Yes, CALEA is a real law with teeth, but it was intended to help law enforcement agencies track criminals using data networks, not force ISPs into bankruptcy.

There are some low cost options available to operators wanting to conform, so before you break the bank, do some research.  But, also be aware, as somewhere along the line CALEA became the Next Y2k fear-driven windfall for unscrupulous networking sales reps. Familiarize yourself with what you need and then find a product that works for you. While we were more than happy to help users of our products comply, we felt than an informed customer was more important that one that was simply panicked and afraid.  More info on the NetEqualizer approach to CALEA compliance.

2) Secure credit card transmission over the Internet — In short, credit information becomes the most unsecured  once it reaches  a corporate database. A hacker or employee with bad intentions is many times more likely to lift credit card information from a fixed database rather than in transit over the Internet. Therefore, the paranoia that abounds over submitting a credit card to Web a site for fear of transmission piracy is way out of proportion to the actual risk.

Consumers will gladly hand their credit card off to a random strangers behind the cash register at a brick and mortar establishment, but for some reason, submitting your credit card to a Web site creates an unacceptable risk for many. This fear has given rise to a cottage industry around secure Internet transmission. The bottom line is that stealing a credit card in transit over the Internet would take extreme patience and inside help from a carrier. To top it off, the credit card issuers have mastered the art of shutting off your card at the first sign of any anomaly (at great inconvenience to their customers in many cases, but worth it in a true emergency).  However despite the relative lack of risk, there is a significant amount of money and technology spent on securing merchant sites.

Related article “Do we really need SSL

3) Y2k — This is an old one, and yes, there were some critical systems out there that might have suffered. My firsthand personal experience from that  time was just a wake-up call. My employer had me doing Y2k upgrades to our product line and the scare pushed our sales to their biggest year ever.  However, within 3 years revenue had dropped 65 percent. Perhaps we should have been doing real product improvements?

4) Virus protection for your laptop — Yes, viruses are real and they attack all the time, but I simply just save off my critical files daily and re-load my windows box when I get a virus.  I prefer this method over being a slave to a Norton pop-up  box.  You can also convert to MAC or Linux desktop, which seem to carry some form of natural immunity. New York Times writer Paul Boutin agrees in this recent article.

5)  Lack of technology for our schools — Yes, there is some level of computer literacy required in the work force today, however, with the billions (trillions?) spent by schools today, you’d think there might be some increase in standardized test scores. I’d much rather see the money spent on increasing teacher salaries and smaller class sizes, even if it meant learning to calculate on an abacus. Training the mind to think and reason critically is a skill for life that transcends technology and requires encouragement and challenge from teachers.

6) Uninterruptable Power Supply (UPS) — I almost gagged when I read the blurb  below from a UPS sales VP from a trade rag. Originally, I was thinking of including UPS power supplies on my list, but I had no evidence that they were being miss represented. And, yes, in many situations a good UPS will save your computer and computer center from crashing, so please understand they are important pieces of equipment for a data center. But, the context below confirmed my suspicion.  The lead touts ways to speed up network performance, essentially implying that if your network is slow, you need UPS servers to correct it!

Are their desktops locking up every time someone runs the microwave oven? “If VARs aren’t selling UPSs [uninterruptible power supplies] with each new server or desktop, they are doing their customers an injustice, and they may be leaving money on the table,” says ….. name and company omitted.

This quote and full  article is written to infer that your desktop computer and network may run “slow” because of a lack of power. The fact is, your computer will crash hard if  power drops below a fixed tolerance. It is not an electric motor that winds down slowly. It is either on or off. A UPS prevents crashes due to lack of power, but it will not make your network faster or more efficient.

The point of this article isn’t to completely discount the six issues discussed above, but rather to provide some context. In many cases, fear is based on a lack of knowledge and understanding. Therefore, the problems mentioned here may not necessarily be best solved with one tech product or another, but instead could be remedied by a little bit of research. As a consumer, doing your homework goes a long way.

An Easy Way to Block/Interfere with Skype?


Art Reisman CTO www.netequalizer.com By Art Reisman

I got a call from a customer the other day who claimed that their NetEqualizer was working great except that it was interfering with their Skype calls, and he wanted us to make it stop.

Upon further investigation, we determined the NetEqualizer was  not interfering with his Skype calls at all. And then it hit me…His upstream ISP must be interrupting them. I can’t be sure of this, but there really was no other explanation. His access was good and we checked a couple of Skype calls and their bandwidth load was well below the threshold of anything the NetEqualizer would touch by design.

Then I had another “Aha!” moment while looking at their Skype streams on our built-in sniffer. The calls seemed to stay fairly steady in a tight range around 16kbs. It would be very easy and low cost to target streams in this range and periodically drop some packets, enough to make the call sound horrific while leaving any non-streaming media in that bandwidth range alone. I have no intention of tweaking our NetEqualizer to fill this mission,  however I did some quick research on the subject and did not come up with anything to make me think it would not work. If you are a Skype geek feel free to comment.

To add some context, here’s a link to an article I wrote a while back on the subject of blocking Skype: Blocking Skype Won’t Be Easy.

Using a Load Generator/Emulator to Test Your Network


By Art Reisman, CTO, APconnections (www.netequalizer.com)

One of the most challenging aspects of technology development has always been the process of bridging the gap between theory and application.  What may seem to work on paper, and even in limited trials, was never guaranteed when dealing with real-world scenarios and often unforeseen problems.

Several members of our engineering team just returned from a week of  testing with Candela Technologies’ network load emulator, and once again, we’ve not been dissapointed.  At the touch of a button, we were able to create unbelievably realistic worst-case load scenarios. Candela’s LANforge equipment not only stressed our network elements, but did so with variation, creating an environment that successfully simulated the challenges our technology will face on a regular basis in the field.

Judging by the numerous trials we’ve run, it’s become clear that simply driving a fixed load across a network is not enough to ensure reliability. Instead, you need a simulation with a multitude of elements (different packet sizes, UDP , TCP, broadcast traffic, etc.) and traffic streams, including those that refuse to back down such as with a bad denial of service attack or virus.  Fortunatley, this is exactly the quality of service that Candela Tech offers.

In addition to giving you peace of mind, this type of simulation can also save you and your company time and money.  When implementing a network upgrade, the normal method of operation goes a little like this:

  1. Work late at night and over the weekends
  2. Implement the change
  3. Put staff on standby for the next business day
  4. Have a fallback strategy to revert to a previously proven configuration should things go south

While these steps eventually may do the trick, they’re not without their costs — both financial and otherwise. Aside from the overtime you’ll end up paying your admin, perhaps more importantly, you also run the risk of negatively impacting the service of clients and customers during the hit-and-miss setup process.

Yet, the costs that come with this type of strategy can easily be reduced with a sophisticated load generation device. Network choke points can be stressed and limits determined before unwittingly making  guinea pigs out of your network users.  And, the staff from Candela Tech is more than knowledgeable and eager to help, which has allowed us to be up and running right out of the box on more than one occasion.

Ultimately, using Candela Technologies has been a lot like looking into a crystal ball. After the LANforge simulations, we’re able to identify and address any issues before they affect our customers. What was once a process of bringing our technology to the real world has now become a process of Candela bringing the real world to us.

Note: There are other competitive network load generators on the market, Fluke being the market leader.

Created by APconnections, the NetEqualizer is a plug-and-play bandwidth control and WAN/Internet optimization appliance that is flexible and scalable. When the network is congested, NetEqualizer’s unique “behavior shaping” technology dynamically and automatically gives priority to latency sensitive applications, such as VoIP and email. Click here for a full price list.

Five Key Marketing Tips for Entrepreneur and Tech Start-Up Companies


By Art Reisman, CEO and co-founder of APconnections, makers of the NetEqualizer

Updated April 25th , 2010

Aside from a few freakish start ups, very few products will take off with out some form of promotion. However, since founding APconnections in 2003,  we’ve learned that marketing can be a double-edged sword. Over time, we’ve been able to build upon both our successes as well as mistakes, coming up with a few dos and don’ts of marketing a tech start up along the way.  Here are a few of the key points:

1) Make sure your marketing company has skin in the game

Most marketing firms are staffed by people who went to college and took soft course work, meaning they were not into the black and white of the scientific method.  Perhaps they had a course or two with this emphasis, but it’s not likely to be as ingrained as perhaps a physics or psychology major whose course work included extensive lab experiments showing cause and effect.

Although some creative skill is necessary to be a good marketing person, the down side is most people in this industry tend to remain artsy and vague with how they can measure results. When negotiating with marketing companies  (or people), we came up with a simple formula to measure results and provide a metric which was easy to quantify — hits to the web site.

In our case it was very simple.  We had a baseline established already and we were only going to change one variable  — marketing.  Hence, it would be easy to tie any increase in web traffic to a marketing effort.  To make sure that  any benefit of doubt went to our marketing firm, we decided  any increase in web traffic, regardless of cause, would be credited to their efforts.

Once we tied marketing fees exclusively to a metric that could be measured, we were able to eliminate several marketing firms, many of which headed for the hills never to be seen from again.

2) Round two — good web traffic verses bad traffic

Having solved the problems of paying for results, we came across another hurdle. We’ll just call it good traffic verses bad traffic. The easiest way to describe this is by example.

Our product, the NetEqualizer, is meant to  be sold to commercial operators and businesses where there is Internet congestion.   An obvious catch-all key phrase to lead with in marketing literature would be, “speed up your Internet connection”.

If you throw this type of tag line into a generic advertisement to a broad audience,  perhaps 99 percent of the people who follow up on it will be home users, kids playing World of Warcraft, looking for some tool they can load up on their Windows machine for $25. In other words, the majority of these follow ups would certainly not be our target market.

What we found was that our consignment-based marketing people were not  screening this traffic out. We believe this disconnect  goes back to their inability to use the scientific method to control variables. So, as you can imagine, we initially had a flood of inquiries outside our target market which turned out to be a big waste of our time. The solution to ending this march of unqualified leads was to put a higher price in any literature or teasers and to emphasize our product was for commercial users etc. So, instead of just promoting the potential to speed up Internet traffic, we made NetEqualizer’s starting price clear from the beginning.

3) Make sure your marketing people understand  how your product is used and take an honest interest in it

We would spend hours explaining our target market and details about our product only to find out that this information would go in one ear and out the other. When we finally found somebody who had the capacity to understand our product we doubled their pay.  It was worth it in time saved.

4) Consignment ads only

Once you have decided on an effective message in an advertisement, follow this rule — Never pay a dime up front to bolster your ad’s exposure just to increase your market presence. Yes market presence is good, but unless you can measure this in terms of some metric,  just don’t do it.

Any advertisement you place should only charge you when somebody clicks on it. I am not talking about discount coupons for a local business here. I am talking about selling a product to a broad national or global audience. This edict pretty much rules out print advertising. To be fair, I have heard from other CEO’s of tech companies that if you stick with print ads and spend a good deal of cash, they will pay-off, but this shouldn’t be your first or only option. Until you have exhausted every conceivable outlet for consignment based advertising, why risk digging any dry wells?

Our experience with six weeks of continuous quarter- and half-page ads in tech magazines brought zero impact. Nothing. Nada. We measured no increase in web hits. Maybe we did increase awareness, but awareness has no value if you go bankrupt establishing it and don’t see any returns.

Of course, while there are no guarantees for successful marketing, these four tips have been tested and proven effective at APconnections over the past several years.

5) Avoid being strung out

Perhaps this tip should be number one as it is essential to understand how companies will string you out. As a start up, with an idea you will likely get conditional inquiries. Can you product do “this”, can we we customize it ? Often times the more questions, the more uncertain the customer is about their own business. You MUST establish the customers willingness to pay before getting  wrapped up in the promise of future sales. Obviously you cannot demand payment on a first consultation with a prospect, but this is a good time to  set some expecations that your time is valuable. I bring this up because at this very moment I am in talks with a large customer interested in our product that has been asking questions for over a month. This morning I basically told  them  (nicely)I will continue the conversation if and when they purchase their first unit from us. For all I know I am dealing with an underling that has time on his hands but no ability to influence a purchase decision. Getting them to purchase something is a big first step toward qualification. If you fail to master qualification you will find yourself borrowing money from relatives to pay your rent or out of business very quickly.

I’d also suggest you look at our tips for using google ads.

Another great site for start ups is entrepreneurship.org

Here is a marketing company (outspoken Media) that  I ran across last Friday , I have not talked to them yet but I really like their bio’s and attitude from their web site.

Good luck!

Looking for a new tech Job? You’ll need to embrace some fear and step out


By Art Reisman, CTO, http://www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Art Reisman

As the CEO of a  tech company and former employee of large stifling unimaginative telecom company here is the advice I recently provided a former colleague on how to land the job at a smaller tech firm.

Contrary to popular media  belief very few (if any employers) care  about your age and race when hiring for a tech job ; however if your former company was  large and stifling, that will create  serious baggage  with a more progressive company.

Nobody ever takes the blame for being part of the problem at a sputtering old line company.  The fact is if you worked there for a long time, then  you were part of the problem. Even if you had a million great ideas on how the company should run itself and the bureaucracy held you down. You will be viewed as complicit by association.

Perhaps it was the high salary that kept you there, or the lack of other opportunity in your region. It does not matter , the smaller start up company will view your past with suspicion.

I meet with several CEOs of other tech companies once a quarter and the consensus is that people with ambition go someplace where they can make an impact.

The perception on the street  is that your old company is still sputtering off its legacy channel left over from what it retained from days past,  and that all the abuse and mind games it uses to stifle employees leaves permanent scars on ingenuity etc.

The kind of people that settled for climbing the ladder in a larger organization are not what new tech companies are  looking for.

You can break that mold by saying hey look how confident I am that I can help you , I am an go getter stuck in an “old company” body type of thing.

So how do you compete and break out?


I’d throw out any expectation of a salary  based on your previous compensation. The company you previously worked for likely  paid you very well. High pay creates a false sense of market value. You’ll notice when times got tough they had to cut massive staff to make ends meet.  So evidentally, the high paying jobs were more of retainer in a good job market rather than a measure against your productivity. Please don’t misconstrue the point I am making,  you do have the potential to make a difference and perhaps make millions, its just that your drive and creativity has been misdirected toward internal corporate games and must be revitalized.

For example, if you are comfortable doing technical sales support and really like a  new company then:

Approach the  CEO (hiring manager) of an established small tech company:

Note: the following specific advice was geared toward an individual and company he was approaching.

Offer to work for $10 an hour  with  conditions:

A Large upside commission and a future salary based on some measurable metric.

I would only make this type  of offer to somebody with integrity that would follow through. Yes there are people with integrity out there , the media plays up the slime balls but most businesses do not function that way at the top.

He  (the CEO)  has long complicated sales cycles with large Fortune 1000  clients.  A mature person who could explain the technology, demonstrate value, could be a huge benefit to his business;  but in this market,  he would not want  the risk of hiring you unless you had skin in the game . On the other hand he might not be able to do something like that because it would be upsetting to his other sales staff who knows ? I am sure it would get his wheels turning as to how he could make it a win win.

This approach also allows you to have some control of where and what you do rather than just take the rescue job that some VP musters  at your current company.

My experience in the real world is that you have to scrap down in the dreggs to get anything going and over time work your  way up. For example when we  first got started there were quite a few super star sales people from the likes of  XXXX and XXXX that I would run into and put to work on commission.  They were completely and totally useless to a small tech startup.  Yes they were nice people but without a ready made channel and customer  base who calls them wanting a bake off between them and the competition they were lost. So we developed a model of guerilla web marketing  and slowly built my oour own direct channel very slowly , but it cost very little and now has very little overhead.

Does your ISP block Web Crawling?


By Art Reisman

Art Reisman CTO www.netequalizer.com

Editor’s note: Art Reisman is the CTO of APconnections. APconnections designs and manufactures the popular NetEqualizer bandwidth shaper.

About one year ago I got the idea to see if I could build a Web Crawler (robot) with the Specific mission of finding references to our brand name on the Internet.

I admit to being a complete amateur to the art of writing a Web Crawler, and certainly it might make more sense to  do Google search on “NetEqualizer” , but I wanted to see if any occurances were  out there,  in Cyber space, that Google ignored or missed.

If you are a hack and want to try this for yourself, I have included my beta Web Crawler source code below.

Back on topic, Does your ISP block Web Crawling?

First a little background on how my Web Crawler works.

1) It takes a seed , a set of web pages to start on

2) It systematically reads those seed Web Pages looking for URL’s amongst them

3) When it finds a URL, it reads it as text, looking for additional URLS within the text.

4) It ranks URLs as Interesting if it finds certain keywords ( a List I created) in the Text of the new URL

5) The more Interesting a URL the more likely it is to get read and so forth.

6) If no keywords are found at all on the searched page it tosses it out as not to be searched. (I think better check this)

7) Ultimately it will stop when it finds “NetEqualizer” or loops a whole bunch of times without finding any new keywords whichever comes first

So you can imagine when this thing is running it is sucking bandwidth as fast as it can read pages, and also hitting random web pages more than humanly possible, after all it is a crawler.

I only ran this script two or three times in its present form because each time I ran it within an hour or so my Internet service would crash and stop altogether. It may just be coincidence that I was having problems with my line at the time as within the next month I did have to have the external cable to the pole replaced by my provider. So honestly I am not postive if my Provider shut me down, but I think so.

At the time, I had not really given it much thought, but if my provider had any watch dog type big brother metric keeping tabs on me, surely this thing would have set off a code Red at the main office. I would assume that residential Internet accounts that start scanning the web at high speed are considered infected with a virus ? Is there a formal clause that by my provider that says they can shut me down if I write a crawler ? I don’t know , as I did not push the issue.

Below is the code. It did start with a perl program written by somebody else, but critical pieces seemed to be omitted (Specific Perl calls on the original) so I had stripped it way down and then built it back up to crawl. I honestly have no idea where I got the original code from as it was over a year ago. Apologies for not giving credit.

See also a generic flow diagram of a Web Crawler.

Sorry about the formatting in the blog.

Use at your won risk etc.

#!/usr/bin/perl -w
##
# spider.pl Set tabstops to 3.
#
$| = 1;

if(scalar(@ARGV) < 2){
print “Usage: $0 <fully-qualified- seed URL> <search-phrase> <keywords>\n”;
exit 1;
}

# Initialize.
%URLqueue = ();
chop($client_host=`hostname`);
$been = 0;
$search_phrase = $ARGV[1];
if (scalar(@ARGV) > 2 ) {
$kicker1 = $ARGV[2]; }
if (scalar (@ARGV) > 3 ) {
$kicker2 = $ARGV[3];
}
if(scalar (@ARGV) > 4 ) {
$kicker3 = $ARGV[4]; }

# Load the queue with the first URL to hit.
$URLqueue{$ARGV[0]} = 0;

# While there’s a URL in our queue which we haven’t looked at …
$total_sites=0;
while ($total_sites < 10000)
{
$x= `echo total sites loop $total_sites >> visited `;
# Progress report.
if ($total_sites > 1000) { exit 1; }
for ( $sites=0; $sites < 200; ) # keep looping hundred times in this beta version
{
$x= `echo sites loop $sites >> visited `;
while(($key,$value) = each(%URLqueue)){
if ( $URLqueue{$key} < 0 ){ if ($URLqueue{$key} == -1)
{ delete $URLqueue{$key}; } # garbage collection
next; } # already been there
if ($sites > 50 && $value < 1 ) {$sites ++; next; }
if ($sites > 100 && $value < 2 ) {$sites ++ ;next;}
if ($sites > 50)
{
$x=`echo primo sites $sites value $value site $key`;
}
($protocol, $rest) = $key =~ m|^([^:/]*):(.*)$|;

# If the protocol is http, fetch the page and process it.
if ( !defined ($protocol)) {next;}
if($protocol eq “http”){
$URLqueue{$key}=-1 ; # mark as visited
$sites++;
$total_sites++;
# Split out the hostname, port and document.
# ($server_host, $port, $document) =
# $rest =~ m|^//([^:/]*):*([0-9]*)/*([^:]*)$|;
print “getting $key \n”;
$x = `cd /tmp; wget -nd -Q 10000 –timeout=2 –tries=1 $key` ;
print “done wget \n”;
$x= `echo $key >> ./visited`;
$page_text = `cat /tmp/* 2> /dev/null`;
$x=`rm /tmp/* 2> /dev/null`;

$page_text =~ tr/\r\n//d;
$page_text =~ s|<!–[^>]*–>||g;
# Report if our search string is found here.
$kick=0;
if($page_text =~ m|$search_phrase|i){
print “found phrase $key $search_phrase ,total sites $total_sites \n”;
exit ;
}
if ( defined $kicker1) {
if($page_text =~ m|$kicker1|i){
#rank this page higher if it has this key word
$x= `echo found kicker $key $kicker1 total sites $total_sites >> visited`;
$kick++;
}
}
if ( defined $kicker2 ) {
if($page_text =~ m|$kicker2|i){
# rank this page higher if it has this key word
$x= `echo found kicker $key $kicker2 ,total sites $sites >> visited`;
$kick++;
}
}
if (defined $kicker3 ) {
if($page_text =~ m|$kicker3|i){
# rank this page higher if it has this key word
print “found kicker $key $kicker3 ,total sites $sites \n”;
$kick++;
}
}
else
{
delete $URLqueue{$key}; # not http
}

# Find anchors in the HTML and update our list of URLs..
(@anchors) = $page_text =~ m|<A[^>]*HREF\s*=\s*”([^
“>]*)”|gi;
foreach $anchor (@anchors){

$newURL = &fqURL($key, $anchor);

if ( exists $URLqueue{$newURL} )
{
$URLqueue{$newURL}= $URLqueue{$newURL} -1;
#don’t garbage collect low numbers
print “duplicate $newURL \n”;
}
else
{
print “new anchor $newURL \n”;
if ($kick > 0 ) {
$x=`echo kick $kick $key $newURL >> ./anchors`; }
$URLqueue{$newURL} =$kick; #new url added to queu
}
}
} #end of while URLqueue
} # end of sites
} #end of total sites
}

sub fqURL
{
local($thisURL, $anchor) = @_;
local($has_proto, $has_lead_slash, $currprot, $currhost, $newURL);

# Strip anything following a number sign ‘#’, because its
# just a reference to a position within a page.
$anchor =~ s|^.*#[^#]*$|$1|;

# Examine anchor to see what parts of the URL are specified.
$has_proto = 0;
$has_lead_slash=0;
$has_proto = 1 if($anchor =~ m|^[^/:]+:|);
$has_lead_slash = 1 if ($anchor =~ m|^/|);

if($has_proto == 1){

# If protocol specified, assume anchor is fully qualified.
$newURL = $anchor;

}
elsif($has_lead_slash == 1){

# If document has a leading slash, it just needs protocol and host.
($currprot, $currhost) = $thisURL =~ m|^([^:/]*):/+([^:/]*)|;
$newURL = $currprot . “://” . $currhost . $anchor;

}
else{

# Anchor must be just relative pathname, so append it to current URL.
($newURL) = $thisURL =~ m|^(.*)/[^/]*$|;
$newURL .= “/” if (! ($newURL =~ m|/$|));
$newURL .= $anchor;

}
return $newURL;
}
The disclaimers:

Use this code at your own risk. I am not even sure if it follows the moral and ethic standards that the major players who crawl the web for living abide by; but since I was only doing this as a weekend experiment I did not worry too much about the standard.

In other words it is experimental and not for commerical use. Do not walk away and leave it running unattended lest you get censured and black listed from the Internet.

Where are the safe tech jobs ?


By Art Reisman, CTO, http://www.netequalizer.com

Art Reisman CTO www.netequalizer.com

Art Reisman

Article Type: Opinion

As the CEO of a small (yet growing) tech company in the current recession I often get calls from former colleagues working at larger corporations. Amidst their companies insincere rhetoric, inaction, and falling revenues, good people wait around wondering who will be next to get the ax.

The underlying problem at most of these companies is that they are continue to push products into a stagnant or declining market. The only way to have any relative security is to get on board with an industry or niche with solid growth potential.

So if your wondering where to turn for potential job security here are some tips that might help

Look for a company that is doing something with real value for society and not just jumping on the latest bandwagon.

1) Renewable energy is hot , and certainly a job in renewable energy is better than selling steam engines running off coal. Renewable energy, although here to stay is being over hyped . Right now the success of renewable energy is dependent on battery technology. Fossil fuels are nothing more than the Suns energy stored up and retrieved at will when needed. For renewable (wind, solar) energy is to compete easily with traditional fossil fuels we must come up with a clean effective battery to store energy. My advice seek out a company that specializes in battery technology and then help them make a difference.

2) Network and Internet Optimization

Internet Infrastructure companies are being forced by their stock holders to turn a profit. The days of free falling bandwidth contracts are slowing down, hence the new hot market will be companies with products that optimize internet bandwidth. Bandwidth control , WAN optimization and compression although not on the front pages, are areas of value and are holding their own in the recession. Some companies to look at , are

APconnection (NetEqualizer)

Packeteer

Allot

RiverBed

Exinda

3) Medical Technology,

From newer and better and MRI machines to prosthetics , Americans will spare no expense for anything that will make their lives more comfortable. So when will this party end and the associated demand for jobs in the Medical Industry flatten out?

Although I do not expect a crash in this field as we might have seen in other boom and bust industries, I do expect a slowdown. Every bubble has its end, and the Medical technology industry is due for a slow down. As consumers push back on medical care pricing, high end technology research will slow down. Still a better prospect than steam engines though.

4) Auto Industry

If you are entering into the field of Mechanical Engineering or electronics controls now would be a good time to focus on the Auto Industry. For the next 5 to 10 years I expect that auto makers will be looking for new innovative ideas in their engineering departments. They will also be looking for new talent. Don’t let the down turn discourage you this is an opportunity.

The Pros and Cons of Bonded DSL and Load Balancing Multiple WAN links


Editor’s Note:We often get asked if our NetEqualizer bandwidth shapers can do load balancing. The answer is yes -maybe if we wanted to integrate into one of the public domain load balancing devices freely available. It seems that to do it correctly without issues is extremely expensive. 

In the following excerpt, we have reprinted some thoughts and experience from a user who has a wide breadth of knowledge in this area.  He gives detailed examples of the trade-offs involved in bonding multiple WAN connections.

When bonding is done by your provider, it is essentially seamless and requires no extra effort (or risks to the customer). It is normally done using bonded T1 links, but also can come in the form of a bonded DSL. The technology discussed below is applicable to users who are bonding two or more lines together without the knowledge (or help) of their upstream provider.

As for Linux freeware Load Balancing devices, they are NOT any sort of true bonding at all.  If you have 3 x 1.5 Mbit lines, then you do NOT have a 4.5 Mbit line with these products. If you really want a 4.5Mbit Bonded line, then I’m not aware of any way to do it without having BGP or some method of coordinating with someone upstream on the other side of the link.  However, what a multi-WAN-router will do is try to equally spread sessions out over the three lines, so that if your users are collectively doing 3Mbit of collective downloads, that should be about 1Mbit on each line. For the most part, it does a pretty good job.

It does this by using fairly dumb round-robin NATing.  So, it’s much like a regular NAT router – everyone behind it is a private 192.168 number (which is the 1st downside) – and it will NAT the privates to one of the 3 Public IP’s on the WAN ports. The side effect of that is broken sessions, where some websites (particularly SSL) will complain that your IP address has changed, for example, while you’re inside the shopping cart or whatever.

To counteract that problem, they have ‘session persistence’ which tries to track each ‘Session Pair’ and keep the same WAN IP in effect for that ‘Session Pair’. That means that the 1st time one of the private IP:port accesses some particular public ip:port, the router will remember that and use that same WAN port for that same public/private pair. The result of this is that ‘most’ of the time, we don’t have these broken sessions, but the downside of this is that the fairness of the load balancing is offset.

For example, if you had 2 lines connected:

  • User1 comes to speakeasy and does a speedtest – the router says ‘speakeasy is out WAN1 forevermore’.
  • User2 comes and looks up google, and the router says ‘google is out WAN2 forevermore’
  • User3 goes to Download.com and the router decides ‘Download.com is on WAN1′.
  • User4 goes to smalltextsite.com (WAN2)
  • User5 goes to YouTube (WAN1)

And so on. With session persistence turned on, User300 will get SpeakEasy, Download.com and YouTube across WAN1 because that’s what it originally learned to be persistent about.

So, the tradeoff is if you don’t use the session persistence, then you’ll have angry customers because things break. If you do use persistence, then there may be an unbalancing.

Also, there are still some broken sites, even with persistence on. For example, some online stores have the customer shopping at www.StoreSite.com and when they checkout it transfers their cart contents to www.PaymentProcessor.com, which may flag an IP security violation. Any time the router sees different IP’s out in the public side, it figures it can use a new WAN port and doesn’t know it’s the same user and application. There are a few game launchers that kids load a ‘launcher’ program and select a server to connect to, but when they actually click ‘connect’, the server complains because the WAN addresses have changed.

In all honesty, it works quite well and there are few problems. We also can make our own exception list, so in my shopping cart example, we can manually add ‘storesite.com‘ and ‘paymentprocessor.com‘ to the same WAN address and that will ensure that it always uses the same WAN for those sites. This requires that users complain first before you would even know that there is a problem, AND also requires some tricks to figure out what’s going on.  However, the exception list can ultimately handle these problems if you make enough exceptions.

Created by APconnections, the NetEqualizer is a plug-and-play bandwidth control and WAN/Internet optimization appliance that is flexible and scalable. When the network is congested, NetEqualizer’s unique “behavior shaping” technology dynamically and automatically gives priority to latency-sensitive applications, such as VoIP and email. Click here to request a full price list.

Additional articles

How to inexpensively increase internet bandwidth by bonding cable and dsl.

From BusinessPhoneNews.com a great guide to access bandwidth needs, Bandwidth Management Buyers Guide.

When is Deep Packet Inspection a Good Thing?


Commentary

Update September 2011

Seems some shareholders  of a company who over promised layer 7 technology are not happy.

By Eli Riles

As many of our customers are aware, we publicly stated back in October 2008 that we officially had switched all of our bandwidth control solutions over to behavior-based shaping. Consequently, we  also completely disavowed Deep Packet Inspection in a move that has Ars Technica described as “vendor throws deep packet inspection under the bus.”

In the last few weeks, there has been a barrage of attacks on Deep Packet Inspection, and then a volley of PR supporting it from those implementing the practice.

I had been sitting on an action item to write something in defense of DPI, and then this morning I came across a pro-DPI blog post in the New York Times. The following excerpt is in reference to using DPI to give priority to certain types of traffic such as gaming:

“Some customers will value what they see as low priority as high priority,” he said. I asked Mr. Scott what he thought about the approach of Plusnet, which lets consumers pay more if they want higher priority given to their game traffic and downloads. Surprisingly, he had no complaints.

“If you said to me, the consumer, ‘You can choose what applications to prioritize and which to deprioritize, and, oh, by the way, prices will change as a result of how you do this,’ I don’t have a problem with that,” he said.

The key to this excerpt is the phrase, “IF YOU ASK THE CONSUMER WHAT THEY WANT.” This implies permission. If you use DPI as an opt-in , above-board technology, then obviously there is nothing wrong with it. The threat to privacy is only an issue if you use DPI without consumer knowledge. It should not be up to the provider to decide appropriate use of DPI,  regardless of good intent.

The quickest way to deflate the objections  of the DPI opposition is to allow consumers to choose. If you subscribe to a provider that allows you to have higher priority for certain application, and it is in their literature, then by proxy you have granted permission to monitor your traffic. I can still see the Net Neutrality purist unhappy with any differential service, but realistically I think there is a middle ground.

I read an article the other day where a defender of DPI practices (sorry no reference) pointed out how spam filtering is widely accepted and must use DPI techniques to be effective. The part the defender again failed to highlight was that most spam filtering is done as an opt-in with permission. For example, the last time I checked my Gmail account, it gave the option to turn the spam filter off.

In sum, we are fully in support of DPI technology when the customer is made aware of its use and has a choice to opt out. However, any use of DPI done unknowingly and behind the scenes is bound to create controversy and may even be illegal. The exception would be a court order for a legal wiretap. Therefore, the Deep Packet Inspection debate isn’t necessarily a black and white case of two mutually exclusive extremes of right and wrong. If done candidly, DPI can be beneficial to both the Internet user and provider.

See also what is deep packet inspection.

Eli Riles, a consultant for APconnections (Netequalizer), is a retired insurance agent from New York. He is a self-taught expert in network infrastructure. He spends half the year traveling and visiting remote corners of the earth. The other half of the year you’ll find him in his computer labs testing and tinkering with the latest network technology.

For questions or comments, please contact him at eliriles@yahoo.com.

Is Your ISP Throttling Your Bandwidth?


Editor’s  Note: With all the recent media coverage about ISPs giving preferential treatment to VOIP, and the controversy over Net Neutrality, we thought it might be interesting to revisit this original article Art published in PC Magazine back in 2007.

Update August 2010 the FCC is not being fooled anymore.

Analysis: The White Lies ISPs Tell About Broadband Speeds

By Art Reisman, CTO, APconnections (www.netequalizer.com)

In a recent PC Magazine article, writer Jeremy Kaplan did a fantastic job of exposing the true Internet access speeds of the large consumer providers.

He did this by creating a speed test that measured the throughput of continuous access to popular Web sites like Google, Expedia, and many others. Until this report was published, the common metric for comparing ISPs was through the use of the numerous Internet speed test sites available online.

The problem with this validation method was that it could not simulate real speeds encountered when doing typical Web surfing and downloading operations. Plus, ISPs can tamper with the results of speed tests — more on this later.

When I saw the results of PC Magazine’s testing, I was a bit relieved to see that the actual speeds of large providers was somewhere between 150 Kbit/s and 200 Kbit/s. This is a far cry from the two, three or even four megabit download speeds frequently hyped in ISP marketing literature.

These slower results were more in line with what I have experienced from my home connection, even though online Internet speed tests always show results close, if not right on, the advertised three megabits per second. There are many factors that dictate your actual Internet speed, and there are also quite a few tricks that can be used to create the illusion of a faster connection.

Before I continue, I should confess that I make my living by helping ISPs stretch their bandwidth among their users. In doing this, I always encourage all parties to be honest with their customers, and in most cases providers are. If you read the fine print in your service contract, you will see disclaimers stating that “actual Internet speeds may vary”, or something to that effect. Such disclaimers are not an attempt to deceive, but rather a simple reflection of reality.

Guaranteeing a fixed-rate speed to any location on the Internet is not possible, nor was the Internet ever meant to be such a conduit. It has always been a best-effort mechanism. I must also confess that I generally only work with smaller ISPs. The larger companies have their own internal network staff, and hence I have no specific knowledge of how they deal with oversold conditions, if they deliberately oversell, and, if so, by how much. Common business sense leads me to believe they must oversell to some extent in order to be profitable. But, again, this isn’t something I can prove.

Editors update Sept 2009: Since this article was written many larger providers have come clean.

A Matter of Expectations

How would you feel if you pumped a gallon of gas only to find out that the service station’s meter was off by 10 percent in its favor? Obviously you would want the owners exposed immediately and demand a refund, and possibly even lodge a criminal complaint against the station. So, why does the consumer tolerate such shenanigans with their ISP?

Put simply, it’s a matter of expectations.

ISPs know that new and existing customers are largely comparing their Internet-speed experiences to dial-up connections, which often barely sustain 28 Kbit/s. So, even at 150 Kbits/s, customers are getting a seven-fold increase in speed, which is like the difference between flying in a jet and driving your car. With the baseline established by dial-up being so slow, most ISPs really don’t need to deliver a true sustained three megabits to be successful.

As a consumer, reliable information is the key to making good decisions in the marketplace. Below are some important questions you may want to ask your provider about their connection speeds. It is unlikely the sales rep will know the answers, or even have access to them, but perhaps over time, with some insistence, details will be made available.

Five Questions to Ask Your ISP

1.) What is the contention ratio in my neighborhood?

At the core of all Internet service is a balancing act between the number of people who are sharing a resource and how much of that resource is available.

For example, a typical provider starts out with a big pipe of Internet access that is shared via exchange points with other large providers. They then subdivide this access out to their customers in ever smaller chunks — perhaps starting with a gigabit exchange point and then narrowing down to a 10 megabit local pipe that is shared with customers across a subdivision or area of town.

The speed you, the customer, can attain is limited to how many people might be sharing that 10 megabit local pipe at any one time. If you are promised one megabit service, it is likely that your provider would have you share your trunk with more than 10 subscribers and take advantage of the natural usage behavior, which assumes that not all users are active at one time.

The exact contention ratio will vary widely from area to area, but from experience, your provider will want to maximize the number of subscribers who can share the pipe, while minimizing service complaints due to a slow network. In some cases, I have seen as many as 1,000 subscribers sharing 10 megabits. This is a bit extreme, but even with a ratio as high as this, subscribers will average much faster speeds when compared to dial-up.

2.) Does your ISP’s exchange point with other providers get saturated?

Even if your neighborhood link remains clear, your provider’s connection can become saturated at its exchange point. The Internet is made up of different provider networks and backbones. If you send an e-mail to a friend who receives service from a company other than your provider, then your ISP must send that data on to another network at an exchange point. The speed of an exchange point is not infinite, but is dictated by the type of switching equipment. If the exchange point traffic exceeds the capacity of the switch or receiving carrier, then traffic will slow.

3.) Does your provider give preferential treatment to speed test sites?

As we alluded to earlier, it is possible for an ISP to give preferential treatment to individual speed test sites. Providers have all sorts of tools at their disposal to allow and disallow certain kinds of traffic. It seems rather odd to me that in the previously cited PC Magazine test, which used highly recognized Web sites, the speed results were consistently well under advertised connection speeds. One explanation for this is that providers give full speed only when going to common speed test Web sites.

4.) Are file-sharing queries confined to your provider network?

Another common tactic to save resources at the exchange points of a provider is to re-route file-sharing requests to stay within their network. For example, if you were using a common file-sharing application such as BitTorrent, and you were looking some non-copyrighted material, it would be in your best interest to contact resources all over the world to ensure the fastest download.

However, if your provider can keep you on their network, they can avoid clogging their exchange points. Since companies keep tabs on how much traffic they exchange in a balance sheet, making up for surpluses with cash, it is in their interest to keep traffic confined to their network, if possible.

5.) Does your provider perform any usage-based throttling?

The ability to increase bandwidth for a short period of time and then slow you down if you persist at downloading is another trick ISPs can use. Sometimes they call this burst speed, which can mean speeds being increased up to five megabits, and they make this sort of behavior look like a consumer benefit. Perhaps Internet usage will seem a bit faster, but it is really a marketing tool that allows ISPs to advertise higher connection speeds – even though these speeds can be sporadic and short-lived.

For example, you may only be able to attain five megabits at 12:00 a.m. on Tuesdays, or some other random unknown times. Your provider is likely just letting users have access to higher speeds at times of low usage. On the other hand, during busier times of day, it is rare that these higher speeds will be available.

In writing this article, my intention was not to create a conspiracy theory about unscrupulous providers. Any market with two or more choices ensures that the consumer will benefit. Before you ask for a Congressional investigation, keep in mind that ISPs’ marketing tactics are no different from those of other industries, meaning they will generally cite best-case scenarios when promoting their products. Federal regulation would only thwart the very spirit of the Internet, which, as said before, has always been a best-effort infrastructure.

But, with the information above, it is your job as a consumer to comparison shop and seek answers. Your choices are what drive the market and asking questions such as these are what will point ISPs in the right direction.

Since we first published this article, Google and others have been trying to educate consumers on Net Neutrality. There is now a consortium called M-Lab which has put together a sophisticated speed test site designed to give specific details on what your ISP is doing to your connection. See the article below for more information.

Related article Ten things your internet provider does not want you to know.

Created by APconnections, the NetEqualizer is a plug-and-play bandwidth control and WAN/Internet optimization appliance that is flexible and scalable. When the network is congested, NetEqualizer’s unique “behavior shaping” technology dynamically and automatically gives priority to latency sensitive applications, such as VoIP and email. Click here for a full price list.

Looking for a Good Web Hosting Company, Here Are Some Tips


We get asked all of the time for references for good Web hosting companies. This Ubuntu blog post by Inventa technologies is a great place to start.

Tips to find the right web hosting company

This article outlines the most important considerations one should keep in mind while looking for a website hosting service provider. Let’s take a look at some of them.

There are many things to consider when choosing a web hosting plan which suits your requirements. The 10 most important are listed below.

1. Types of Servers 2. Disk Space/Storage 3. Bandwidth or data transfer 4. Database support 5. FTP Access 6. Easy to use Control Panel 7. E-mail services 8. Cost 9. Customer Support 10. Added Features

To see the full original article click here.

World Wide Web Founder Denounces Deep Packet Inspection


Editor’s Note: This past week, we counted  several  vendors publishing articles touting how their deep packet inspection is the latest and best. And then there is this…

Berners-Lee says no to internet ‘snooping’

The inventor of the World Wide Web, Sir Tim Berners-Lee, has attacked deep packet inspection, a technique used to monitor traffic on the internet and other communications networks.

Speaking at a House of Lords event to mark the 20th anniversary of the invention of the World Wide Web, Berners-Lee said that deep packet inspection (DPI) was the electronic equivalent of opening people’s mail.

To continue reading, click here.

We can understand how DPI devices are attractive as they do provide visibility into what is going on in your network.  We also understand that the intent of most network administrators is to keep their network running smoothly by making tough calls on what types of traffic to allow on their wires.  But, while DPI is perhaps not exactly the same as reading private mail, as Mr Berners-Lee claims, where should one draw the line ?

We personally believe that the DPI line is one that should be avoided, if at all possible. And, our behavior-based shaping allows you to shape traffic without looking at data. Therefore, effective network optimization doesn’t have to come at the expense of user privacy.

The pros and cons of Disk (Web) Caching


Eli Riles an independent consultant and former VP of sales for NetEqualizer has extensively investigated the subject of caching with many of  ISPs from around the globe. What follows are some useful observations on disk/web caching.

Effective use of Disk Caching

Suppose you are the administrator for a network, and you have a group of a 1000 users that wake up promptly at 7:00 am each morning and immediately go to MSNBC.com to retrieve the latest news from Wall Street. This synchronized behavior would create 1000 simultaneous requests for the same remote page on the Internet.

Or, in the corporate world, suppose the CEO of a multinational 10,000 employee business, right before the holidays put out an all points 20 page PDF file on the corporate site describing the new bonus plan? As you can imagine all the remote WAN links might get bogged down for hours while each and every employee tried to download this file.

Well it does not take a rocket scientist to figure out that if somehow the MSNBC home page could be stored locally on an internal server that would alleviate quite a bit of pressure on your WAN or Internet link.

And in the case of the CEO memo, if a single copy of the PDF file was placed locally at each remote office it would alleviate the rush of data.

Local Disk Caching does just that.

Offered by various vendors Caching can be very effective in many situations, and vendors can legitimately make claims of tremendous WAN speed improvement in some situations. Caching servers have built in intelligence to store the most recently and most frequently requested information, thus preventing future requests from traversing the WAN link unnecessarily .

You may know that most desktop browsers do their own form caching already. Many web servers keep a time stamp of their last update to data , and browsers such as the popular Internet Explorer will use a cached copy of a remote page after checking the time stamp.

So what is the downside of caching?

There are two main issues that can arise with caching:

1) Keeping the cache current. If you access a cache page that is not current then you are at risk of getting old and incorrect information. Some things you may never want to be cached, for example the results of a transactional database query. It’s not that these problems are insurmountable, but there is always the risk that the data in cache will not be synchronized with changes.

2) Volume. There are some 100 millions of web sites out on the Internet alone. Each site contains upwards of several megabytes of public information. The amount of data is staggering and even the smartest caching scheme cannot account for the variation in usage patterns among users and the likely hood they will hit an un-cached page. If you have a diverse set of users it is unlikely the Cache will have much effect on a given day

Formal definition of Caching