Network Provider Outages and DDOS Attacks Dwarf Local Hardware Failure Problems

My Internet Service went down yesterday and I had to revert my backup provider.

Network Outages due to upstream provider Failure are endless…

Comcast Outage for North Denver Fiber cut

Comcast hit with massive Internet outage

Forum discussion about wide spread Internet outage Des Moines Iowa

Spokane Washington 10,000 customers without Internet service

Wide spread Internet outage London , Virgin Media

And even if your provider is not to blame, there are endless hackers out there instigating DDOS attacks , some with an ax to grind others just for random entertainment.

DDOS attack brings down Web Drive Client New Zealand

DDOS attack brings down dutch government

DDOS attack interrupts tournament.

Although this sampling of news stories is not very scientific, I could literally spend a month clipping these articles. There are new ones every day , and that is just the major ones that get reported. If I informally poll our customers, almost every single one of them has seen a  DDOS attack of  some kind in the past year,  and all have had some sort of upstream Internet outages  within the last couple of years.

Now if I ask how many have had critical Network Equipment go down due to hardware failure, that list shrinks to maybe 1 or 2 percent of our customers.   Basically what this tells me , is you have a 100 percent chance of a Network outage for some period of time every year due to a problem upstream with your provider. You have  a 2 percent chance due to a hardware failure with your local core Router/ Firewall/Bandwidth/Switches.

To put that another way , for every 50 outages caused by external events at your provider beyond your control, you have 1 event due to internal hardware failure.

The solution is to have multiple distinct Internet Providers on hand at all times, so if one goes down you can switch over to the other.   As I said there is nothing wrong with the idea of sourcing redundant local equipment , but statistically it is much  more important to get a second Internet provider sourced  before investing in redundant equipment.


Although DDOS attacks are provider Independent, your chances of stopping or mitigating the attack are enhanced by having multiple providers.

Other causes of failures:

Yes wireless topologies are notoriously unstable, and so are applications running on Web Servers, both of which can cause service outages to local users. These types outages are usually not on the same order as catastrophic hardware failure problems or upstream failures.  Outages with wireless equipment and service are usually related to these products getting into a bad state, and are not associated with a complete loss of communication to the outside world.   You’ll still need to re-boot these systems to get them back into a good state.

QoS and Your Cloud Applications, the Must Know Facts

When you make the switch to the cloud, you will likely discover that the standard QoS techniques, from the days when services were hosted within your enterprise, will not work on traffic coming in from the public Internet.  Below we detail why, and offer some unique alternatives to traditional router-based QoS . Read on to learn about new QoS techniques designed specifically for the Cloud.

Any QoS designed for the Cloud must address incoming traffic not originating on your Network

Most Internet congestion is caused by incoming traffic. From downloads of  data not originating at your facility. Unlike  the pre-cloud days,  your local router cannot give priority to this data because it has no control over the sending server stream.  Yes you can still control the priority of outgoing data , but if recreational traffic coming into your network comes in at the same priority as let’s say a cloud based VOIP call , then when your download link is full all traffic will suffer.

Likely No Help from your service provider

Even if you asked your cloud hosting service to mark their traffic as priority, your public Internet provider likely will not treat ToS bits with any form of priority.  Hence all data coming from the Internet into your router from the outside will hit with equal priority.  During peak traffic times , important cloud  traffic will not be able to punch through the morass.

Is there any way to give priority to incoming cloud traffic?

Is QoS over the Internet  for Cloud traffic possible? The answer is, yes, QoS on an Internet link is possible. We have spent the better part of seven years practicing this art form and it is not rocket science, but it does require a philosophical shift in thinking to get your arms around it.

How to give priority to Cloud Traffic

We call it “equalizing,” or behavior-based shaping, and it involves monitoring incoming and outgoing streams on your Internet link.  Priority or QoS is nothing more than favoring one stream’s packets over another stream’s. You can accomplish priority QoS on incoming streams by queuing (slowing down) one stream over another without relying on ToS bits.

How do we determine which “streams” to slow down?

It turns out in the real world there are three types of applications that matter

1 ) Cloud based Business applications. Typically things like, data bases, accounting, sales force, educational, voip services.

2) Recreational traffic such as Netflix, YouTube

3) Downloads and updates

The kicker that we discovered and almost always holds true is that Cloud based applications will use a fraction of the bandwidth of the video recreational traffic and the downloads. If you can simply spot these non essential data hogs by size and slow them down a bit, there will be plenty of room for your Cloud applications during peak periods.

How do we ensure that cloud traffic has priority, if we can’t rely on QoS bits?

To be honest, we stumbled upon this technique about 12 years ago.  We keep track of all the streams coming into your network with what can best be described as a sniffing device. When we see a large stream of data, we know from experience that it can’t be cloud traffic, just too large of a stream. Cloud applications by design are rarely large streams , because if they were, the cloud application would likely be sluggish and not commercially viable. With our sniffing device, the NetEqualizer, we are able to slow down the non-cloud connections by adding in tiny bit of latency, while at the same time allowing the cloud application streams to pass through. The interesting result is that the sending servers (the same ones that ignore TOS bits) will actually sense that their traffic is being delayed in transport and they will back off their sending speeds on their own.

For more information or a demo feel free to contact us

Miracle Product Fixes Slow Internet on Trains, Planes, and the Edge of the Grid

My apologies for the cheesy lead in. Just having some lighthearted fun, after my return from a seminar in the UK, and seeing all their news stands with the sensational headlines.

A few years ago I got a call from an agency that maintained the Internet service for the National Train service of a European country. (Finland)
The scheme they used to provide internet access on their trains was to put a 4g wireless connection on every train, and then relay the data to a standard Wifi connection for customers on the train.  The country has good 4g access throughout, hence this was the most practical way to get Internet to a moving vehicle.

Using this method they were able to pipe “mobile” wifi into the trains running around the country.  When their trains got a bit crowded the service became useless during peak times. All the business travelers on the train were funneling through what was essentially a 3 or 4 megabit connection.

Fortunately, we were able to work with them to come up with a scheme to alleviate the congestion. The really cool part of the solution was that we were able to put a central Netequalizer at their main data center, and there was no need to put a device on each train. Many of the solutions to this type of problem, either developed internally by satellite providers or by airlines offering Wifi, require a local controller at the user end, thus the cost and the logistics of the solution are much higher than using the centralized NetEqualizer.

We have talked about the using a centralized NetEqualizer for MPLS networks, but sometimes it is hard to visualize using a central bandwidth controller for other hub and spoke type connections such as the train problem. If you would like more information on the details we would be more than happy to provide them.

Complimentary NetEqualizer Bandwidth Management Seminar in the UK

Press Release issued via BusinessWire.

April 08, 2015 01:05 AM Mountain Daylight Time

LAFAYETTE, Colo.–(BUSINESS WIRE)–APconnections, an innovation-driven technology company that delivers best-in-class network traffic management solutions, is excited to announce its upcoming complimentary NetEqualizer Technical Seminar on April 23rd, 2015, in Oxfordshire, United Kingdom, hosted by Flex Information Technology Ltd.

This is not a marketing presentation; it is run by and created for technical staff.

Join us to meet APconnections’ CTO Art Reisman, a visionary in the bandwidth management industry (check out Art’s blog). This is not a marketing presentation; it is run by and created for technical staff. The Seminar will feature in-depth, example-driven discussions of network optimization and provide participants with a first-hand look at NetEqualizer technology.

Seminar highlights include:

  • Learn how behavior-based shaping provides superior QoS for Internet traffic
  • Optimize business-critical VoIP, email, web browsing, SaaS & web applications
  • Control excessive bandwidth use by non-priority applications
  • Gain control over P2P traffic
  • Get visibility into your network with real-time reporting
  • See the NetEqualizer in action! We will demo a live system.

We welcome both customers and those just beginning to think about bandwidth shaping. The Seminar will take place at 14:30pm, Thursday, April 23rd, at Flex Information Technology Ltd in Grove Technology Park, Wantage, Oxfordshire OX12 9FF.

Online registration, including location and driving directions, is available here. There is no cost to attend, but registration is requested. Questions? Contact Paul Horseman at or call +44(0)333.101.7313.

About Flex Information Technology Ltd
Flex Information Technology is a partnership founded in 1993 to provide maintenance and support services to wide range of customers with large mission critical systems, particularly the Newspaper and Insurance sectors. In 1998 the company began focusing on support for small to medium businesses. Today we provide “Smart IT Solutions combined with Flexible and Quality Services for Businesses” to a growing satisfied customer base. We have accounts with leading IT suppliers and hardware and software distributors in the UK.

About APconnections
APconnections is a privately held company founded in 2003 and is based in Lafayette, Colorado, USA. Our flexible and scalable network traffic management solutions can be found at thousands of customer sites in public and private organizations of all sizes across the globe, including: Fortune 500 companies, major universities, K-12 schools, Internet providers, libraries, and government agencies on six continents.


APconnections, Inc.
Sandy McGregor, 303-997-1300 x104
Flex Information Technology Ltd
Paul Horseman, +44(0)333 101 7313

NetEqualizer News: April 2015

April 2015


Enjoy another issue of NetEqualizer News! This month, we update you on our upcoming NetEqualizer Tech Seminars and conferences, ask for your input on what are your most pressing IT problems, and preview more exciting features for our Spring Release (8.3). As always, feel free to pass this along to others who might be interested in NetEqualizer News.

A message from Art…
Art Reisman, CTO – APconnections

Early April is the time of year in North America when some of the early arriving bird migrants start their journey north. artI saw a couple of common Grackles and an Eastern Phoebe this last week, as well as an Osprey – just to name a few. Spring is also the time of year when I get out on the road and start visiting customers. London and Pennsylvania are on the docket, with more to come.
art photo for NL

I really enjoy meeting with customers around the world and hearing their experiences. It’s how some of our best products and features have come to light. Whether it’s a small change to the NetEqualizer interface, or a problem that needs solving, we thrive off of these conversations. Read more about opportunities to meet with me and share your thoughts in this month’s newsletter.

We love it when we hear back from you – so if you have a story you would like to share with us of how we have helped you, let us know. Email me directly at I would love to hear from you!

2015 Technical Seminars

neteq seminar logo with border
Spring 2015 – Coming to the UK on April 23rd

We are coming across the Pond this spring! If you are in or around the United Kingdom on April 23, 2015, come join our CTO, Art Reisman, for an informative and educational session hosted by Flex Information Technology Ltd.

Art will be at the Grove Technology Park in Wantage, Oxfordshire, United Kingdom OX12 9FF the afternoon of April 23rd. You can view the details and register for the event here.

Our Technical Seminars are great because they are not marketing sessions. They are run by our CTO and are technical briefings. The seminar includes discussion on bandwidth control and also a live demonstration of the NetEqualizer technology.

So, if you have always been curious about the NetEqualizer, and would like to learn more, stop by for an afternoon! Or if you are an existing customer, and would like to meet Art to pick his brain, join us in the UK!

Please contact Paul Horseman of Flex Information Technology Ltd with any questions:
+44 (0)333 101 7313

Summer or Fall 2015 – Location TBD

We are currently starting to plan our Summer/Fall 2015 Tech Seminar. For this seminar, we are looking stateside. If you are in the United States and would like to be considered as a host site, let us know by contacting us at:
(303) 997-1300

What is Your Better Mouse Trap?

As always, we are looking to expand our product line in ways that are useful to our customers. Our DDoS Monitoring and DDoS Firewall are a perfect example of an urgent need that came up last month, with several customers being caught off guard by attacks. We responded with a timely interim release, by integrating some existing technology from our IPS device into the NetEqualizer.

We are also interested in new ideas that will help make the life of an IT administrator easier. We want to help you to solve your most pressing IT problems, so please take a moment and think of what your better mouse trap would be, and tell us!

Even if you don’t think it is possible, just throw it out there! Your ideas are invaluable in helping us to create the next generation networking solutions.


Annual edACCESS Conference


June 24, 2015

We are looking forward to visiting with a few our customers at the annual edACCESS Conference hosted by Mercersburg Academy in Mercersburg, Pennsylvania. We will stop by for Vendor Day on Wednesday, June 24th.

If you have never been to an edACCESS Conference, you might not know that they are purposely run small (100 attendees maximum) and that they use the peer conference model.

Here is what they say on the edACCESS website:

“Each edACCESS conference is small, responsive, and participant-driven. Small, because edACCESS conferences are limited to one hundred attendees. Responsive, because half the conference is spent discussing topics chosen by attendees through a careful first-day process. Participant-driven, because we believe that, collectively, we are the experts.”

If you cannot attend this conference, but are in the area of South Central Pennsylvania, please let us know and perhaps we can stop by as we will be on the East Coast for a few days after the conference.

We are looking forward to seeing you there!


Expanded RTR: Spring Release (8.3) Update

Anticipated Release Date: May 2015

Wow! I got a chance to kick the tires on our 8.3 Release last week. My favorite new RTR report now gives you the ability to see real-time bar charts showing actual bandwidth usage on a per Pool basis (you can also see the Top IP or IP subnet users).

And the real icing on the cake was the red warning colors on the bar chart whenever a pool went into Equalizing, essentially a nice graphical indicator that the NetEqualizer is doing it’s job on your main screen. Here is a screenshot of this feature:


We’ve also added in a number of other exciting features! These include, but are not limited to:

1) Historical Penalty Reporting – see how many penalties were enforced at a given point in time on your NetEqualizer.

2) Connection Count Reporting – see connection counts by IP. Use this to find possible P2P issues on your network.

3) Export Data from RTR – export data from the RTR databases in CSV format to keep a history longer than 4 weeks or just analyze the data as you wish.

4) Active Penalties – see all connections that are currently being Equalized, as well as what type of penalty they are receiving.

For more details on Release 8.3 features, check out our March 2015 Newsletter.

The coding for Release 8.3 is now complete, and we are moving it into our testing process. We are currently anticipating a May 2015 release, and will announce it in the May 2015 Newsletter.

Once 8.3 reaches GA, these features will be free to customers with valid NetEqualizer Software and Support (NSS). You will need to upgrade first to version 8.0+. If you are not current with NSS, contact us today!


Best Of The Blog

The Technology Differences between a Web Filter and a Traffic Shaper

By Art Reisman – CTO – APconnections

First, a couple of definitions, so we are all on the same page:

A Web Filter is basically a type of specialized firewall with a configurable list of URLs. Using a Web Filter, a Network Administrator can completely block specific web sites, or block complete categories of sites, such as pornography.

A Traffic Shaper is typically deployed to change the priority of certain kinds of traffic. It is used where blocking traffic completely is not required, or is not an acceptable practice. For example, the mission of a typical Traffic Shaper might be to allow users to get into their Facebook accounts, and to limit their bandwidth so as to not overshadow other more important activities. With a Traffic Shaper, the idea is to limit (shape) the total amount of data traffic for a given category.

From a technology standpoint, building a Web Filter is a much easier proposition than creating a Traffic Shaper. This is not to demean the value or effort that goes into creating a good Web Filter. When I say “easier,” I mean this from a core technology point of view. Building a good Web Filter product is not so much a technology challenge, but more of a data management issue.

A Web Filter worth its salt must be aware of potentially millions of various websites that are ever-changing. To manage these sites, a Web Filter product must be constantly getting updated. The product company supporting the Web Filter must search the Web, constantly indexing new web sites and their contents, and then passing this information into the Web Filter product. The work is ongoing, but not necessarily daunting in terms of technology prowess. The actual blocking of a Web site is simply a matter of comparing a requested URL against the list of forbidden web sites and blocking the request (dropping the packets)…

Photo Of The Month
Tribute to Jack Miller
By Art Reisman
I had the honor to meet my neighbor Jack Miller and listen to his life stories over the past few years. Jack has the unique distinction of serving two WWII tours in North Africa. After the first tour, he was discharged to take care of his farm after the death of his mother, but then, through a clerical mix-up, he was shipped back, only this time he was taken POW by the Germans.I spent many summer evenings sitting on his front porch listening to his stories. My favorite one was set at a time when the Germans were in retreat, and his battalion was marching across Germany, extremely hungry and low on rations. He described walking past German farms with grazing cattle, and I asked him why they did not just take the cows and eat them. Without hesitation, Jack’s reply was “Why, those cows belonged to the German people.”

People like Jack were cut from a different mold.

We lost Jack at the age of 92 last November.

Rest in Peace, Jack.

So You Think you Have Enough Bandwidth ?

There are actually only two tiers of bandwidth , video for all , and not video for all. It is a fairly black and white problem. If you secure enough bandwidth such that 25 to 30 percent of your users can simultaneously watch video feeds, and still have some head room on your circuit, congratulations you have reached bandwidth nirvana.

Why is video the lynchpin in this discussion?

Aside from the occasional IOS/Windows update, most consumers really don’t use that much bandwidth on a regular basis. Skype, Chat, E-mail, Gaming, together, do not consume as much bandwidth as video. Hence the marker species for congestion is Video.

Below  I have presented some of the metrics to see if you can mothball your bandwidth shaper.

1) How to determine the future bandwidth demand. Believe it or not, you can outrun your bandwidth demand , if your latest bandwidth upgrade is large enough to handle the average video load per customer.  Then it is possible that no further upgrades will be needed, at least in the foreseeable future.

In the “Video for all” scenario the rule of thumb is you can assume 25 percent of your subscribers watching video at any one time . If you still have 20 percent of your bandwidth left over, you have reached the video for all threshold.

To put some numbers to this

Assume 2000 subscribers, and a 1 gigabit link. The average video feed will require about 2 megabits. (note some HD video is higher than this )  This would mean, to support video 25 percent of your subscribers would use the entire 1 gigabit and there is nothing left over anybody else, hence you will run out of  bandwidth.

Now if you have 1.5 gigabits for 2000 subscribers you have likely reached the video for all threshold, and most likely you will be able to support them without any advanced intelligent bandwidth control . A simple 10 megabit rate cap per subscriber is likely all you would need.

2) Honey Moon periods are short-lived  The reason why the reprieve  in congestion after a bandwidth upgrade is so short-lived is usually because the operator either does not have a good intelligent bandwidth control solution or they take their existing solution out thinking mistakenly they have reached the “video for all” level. When in reality they are still under the auspices of the video not for all . They are lulled into a false sense of security for a brief honeymoon period.  After the upgrade things are okay. It takes a while for a user base to fill the void of a new bandwidth upgrade.  Unless you have the numbers to support 25 to 30 percent of your user base running video you will need some kind of bandwidth control.

Application Shaping and Encryption on a Collision Course

Art Reisman, CTO APconnections

I have had a few conversations lately where I have mentioned that due to increased encryption, application shaping is really no longer viable.  This statement without context evokes some quizzical stares and thus inspired me to expound.

I believe that due to increased use of encryption, Application Shaping is really no longer viable…

Yes, there are still ways to censor traffic and web sites, but shaping it, as in allocating a fixed amount of bandwidth for a particular type of traffic, is becoming a thing of the past. And here is why.

First a quick primer in how application shaping works.

When an IP packet with data comes into the application shaper, the packet shaper opens the packet and looks inside.  In the good old days the shaper would see the data inside the packet the same way it appeared in context on a web page. For example, when you loaded up the post that you are a reading now, the actual text is transported from the WordPress host server across the internet to you, broken up in a series of packets.  The only difference between the text on the page and the text crossing the Internet would be that the text in the packets would be chopped up into segments (about 1500 characters per packet is typical).

Classifying traffic in a packet shaper requires intercepting packets in transport, and looking inside them for particular patterns that are associated with applications (such as YouTube, Netflix, Bittorrent, etc.).  This is what is called the application pattern. The packet shaping appliance looks at the text inside the packets and attempts to identify unique sequences of characters, using a pattern matcher. Packet shaping companies, at least the good ones, spend millions of dollars a year keeping up with various patterns associated with ever-changing applications.

Perhaps you have used HTTPS, ssh. These are standard security features built into a growing number of websites. When you access a web page from a URL starting with HTTPS, that means this website is using encryption, and the text gets scrambled in a different way each time it is sent out.  Since the scrambling is unique/different for every user accessing the site, there is no one set pattern, and so a shaper using application shaping cannot classify the traffic. Hence the old methods used by packet shapers are no longer viable.

Does this also mean that you cannot block a website with a Web Filter when HTTPS is used?

I deliberately posed this question to highlight the difference between filtering a site and using application shaping to classify traffic. A site cannot typically hide the originating URL, as the encryption will not begin until there is an initial handshake. A web filter blocks a site based on the URL, thus blocking technology is still viable to prevent access to a website. Once the initial URL is known, data transfer is often set up on another transport port, and there is no URL involved in the transfer. Thus the packet shaper has no idea of where the datastream came from, nor is there any pattern that can be discerned due to the encryption stream.

So the short answer is that you can block a website using a web filter, even when https is used.  However, as we have seen, the same does not apply to shaping the traffic with an application shaper.


Get every new post delivered to your Inbox.

Join 58 other followers

%d bloggers like this: