Thursday, July 11, 2013

Where Are The Posts About The Websense Outage?

For those who don't know, Websense is filtering software that companies and Universities use to monitor and control Internet access on their networks.  Yesterday morning Websense sent out an update to their software that essentially broke it, plunging hundreds of their customers into Internet darkness...

At my company, we noticed the outage about 3 minutes after the update occurred; we identified Websense as the cause about 10 minutes afterwards, and we had a fix in place within the hour, however we had company-wide Internet access problems for a total an hour and 15 minutes.  Websense sent a fix for the problem about 5 hours after it started, I can only assume that most of their customers had at least a couple of hours of downtime/problems.  So why aren't there any news posts about this?

According to Gartner, Websense is an industry leader in the field of Internet filtering.  With all the companies that were impacted by yesterday's outage, why is there no media coverage of this incredible SNAFU?  During the initial phase of the outage, the only evidence of this being a widespread problem that I could initially find was on social media, and I had to do hashtag searches to find them!

There is a post on the Internet Storm Center that references this issue... so it's not like people aren't aware that it happened.  Why isn't there more talk about this in the media?  You'd think that this outage would raise all kinds of questions from "How could it happen if Websense has proper QA?" to "Should Internet filtering be designed with a 'kill switch' so that downtime can be minimized?" I'm very surprised that there isn't more discussion of this issue today.

We have now implemented a spare web proxy that can be spun up and placed into production in minutes should an issue like this occur again.  We've also set up monitoring of the Websense gateway so we will be alerted should the symptoms of this type of problem show up in the future.  An open discussion of issues like this, along with coverage in the media, will only improve systems and allow people to prepare for future issues.  Hopefully posts like this one will lead to a broader discussion of this outage and the implications it has for Websense's customers.