DNS issues happen, sometimes restarting BIND helps, other times there is a serious problem and it takes a bit to work out. Thus is the way of IT. However, if there is a DNS issue with a well-known US government agency, then there is a serious problem, because the government doesn’t make mistakes right? Late last week the NSA (National Security Agency) fell offline, and if you believe the hype, the three to four hour down time caused the sky to fall in various parts of the world.
Yes Virginia even the NSA can make a mistake. (IMG:J.Anderson)
So why is this news? In short, it isn’t, not really. However, it serves to prove a point that no amount of money or technology can prevent a mistake. Planning and design when laying out the foundation for a network is great, but we humans will always mess something up; it’s a hard unpleasant fact. Sometimes when things go bad, a fix is seconds away, other times it takes a while to work out. That is why one of the key requirements for an IT person, no matter their level of operational experience, is troubleshooting.
The NSA lost internet connectivity at about 7 a.m. PST on Thursday last week. Danny McPherson over on Arbor Networks did some – yep you guessed it – troubleshooting, and discovered a DNS issue along with some interesting facts. The two authoritative DNS servers were in the same /16 prefix.
romulus.nscs.mil (144.51.5.2)topscale.nsa.gov (144.51.68.4)
You can do that with DNS, but it is not the best of practices. Most online web hosts will force users to do this, it can work, but the problem is that it can lead to issues down the road if you are not careful. The exact cause for the DNS outage is unknown. The NSA website was back online at 11 a.m. PST the same day. The hype that followed during and shortly after the outage is what movies are made of.
The NSA has made no statement other than that the techs were working to resolve the problem. After service was restored, the NSA went back to not talking to anyone. (They are a government intelligence agency, it’s what they do.) One thing is clear, it was not an attack on the ROOT DNS servers, nor was it an attack on the NSA. Some people will run and scream attack these days for no reason.
There were no exploits used on the Apache services on the server hosting the NSA website. The issue was DNS, and as McPherson said in an interview with IDG, “It's either an internal routing problem of some sort on their side or they've messed up some firewall or ACL policy. Or they've taken their servers off-line because something happened.”
He added, “I am certain that someone's going to send an e-mail at some point that's not going to get through. If it's related to national security and it's not getting through, then as a US citizen that concerns me.”
Danny McPherson is a smart man; he knows not to spread FUD like this. The NSA does not rely nor collect creditable intelligence from email. If there were a local or international threat that the NSA had to address, rest assured email is the last method used for communications. Simply put, email is too insecure to use.
The issue was likely configuration, as that is the cause for most DNS outages. There are countless stories of people forgetting to add a period (.) when working with BIND causing the website to just poof out of existence. After all, you have to hand it to the NSA, rarely have they ever gone down, and one of the times it happens and makes the news, they are only offline for four hours. Not bad really.
Add your comment (no registration required)
page: 1
scottyMay 23rd, 2008 - 17:41:15
well, they're off again today. early vacation?
Report this comment
Advertising
scottyMay 23rd, 2008 - 17:41:15
well, they're off again today. early vacation?
Report this comment