On Tuesday, at about 2:00 a.m. PST, lasting about 14 hours, SonicWALL customers learned firsthand the power of DRM, when a server glitch on SonicWall’s licensing server caused thousands of customers to lose service, leaving critical services such as Firewalls, VPN, CDP, and Email wide open.
“Beginning 12/2/2008 approximately 2:00 AM Pacific time, some SonicWALL products contacting a particular SonicWALL licensing server began receiving erroneous responses. This issue may have caused the products' license keys to be reset, and in some cases may have affected the products' operation,” are the only details SonicWALL would post to the public and send to an email.
The reasoning given to customers is not enough some said.
“The email received [this] evening – ‘SonicWALL License Server Outage’ really does not cover this. I am getting my ... busted over the outages today. We need something more up front and honest - not some BS about 'license outages',” said one SonicWALL forum user, exsarcorp.
“What happened was a near complete shutdown of TZ180's (what I am running) that killed our internet access for more than eight hours. This isn't a 'license server outage' it's a liability that I can not justify, defend, or support any further.”
At the time of this article, there have been no explanations as to the actual cause of the outage or the resolution steps taken to prevent it from happening again.
The outage affected customers who use SonicWALL Firewall (UTM) / Network Security (NSA) Appliances, SonicWALL Email Security Appliances and Software (v.6 and up), SonicWALL Continuous Data Protection (CDP) Appliances, SonicWALL Global Management System (SGMS) Software, and SonicWALL SSLVPN Appliances (Model 200, 2000, and 4000).
Some of the comments on the forum expressed the stress many of the IT Administrators who use SonicWall’s products were under. The harsh fact of IT is that the bosses do not care about licensing issues, service outages, or why a product is broken.
They want it fixed, and the IT Administrator is the one who will be doused in gasoline held to a match until it is. Once fixed, there had better be a damn good reason for the outage to begin with.
“This is beyond belief...Security should never be compromised due to a ‘licensing server’. Security services of the firewall should always be Paramount. Exactly what am I paying for? SonicWall’s whole business is SECURITY...and now we all are holding our breath hoping we won't get a virus or Trojan etc. It's unbelievable,” wrote delta58 on the SonicWALL forum.
“I fully agree. I am absolutely livid at the moment and I know the problem isn't over – I guarantee that we were compromised during this window of opportunity.....and that translates to MORE WORK FOR ME. How can something like this happen? Why did it take so long to resolve? What transpired today negates your entire line of products and services. NOT a good move,” added user pcicanada.
“This is unacceptable. Changes will need to [be] made to your real-time licensing setup in order for me to continue doing business with SonicWALL. We are a 100 million dollar "technology" defense contractor with serious security considerations and I can't just have SPAM and VIRUSES [pouring] into my network for half a day because your license server went down. Where are the redundancies? I expect some explanations and some resolutions to prevent this from ever happening again,” wrote rhouseholder.
There are no reports that any of the affected customers were attacked, or suffered any sort of breach during the outage. No one is sure what SonicWALL will do to compensate any of their customers, if any compensation is offered.
Yet, during the outage, and the rush of angry comments and blast of questions on the forums, there was one SonicWALL employee stuck in the middle who took all the abuse and helped as best as she could. His name is Jody Spoor.
Jody started posting on the forums a few hours into the outage and stuck it out until the end.
“…your frustrations are understandable,” Jody wrote to one user. “Due to our Global organization, we were alerted by this issue as early as we could be. We also took immediate steps from having this issue spread further. That is one of the reasons, why License managers are currently unreachable.”
“If you are having issues with this, please feel free to PM me. I will be online as long as possible to assist you,” he added.
Sadly, he was the only one who stuck around to help online. Most of the posts reported that calls to Tech Support were being sent directly to voicemail. So hats off to Jody.
SonicWALL is reporting that everything is back to normal. They have a Support Bulletin with some details on the outage, and steps customers need to take to get things back on line.