Apache serves up details on recent compromise

by Steve Ragan - Sep 3 2009, 21:10

The Apache Software Foundation (ASF) has posted a detailed report about the recent compromise that led to the temporary suspension of several apache.org services. In their incident report, Apache said they suspect local root exploits as the initial cause of the breach, and confirmed that an SSH key from a backup account was the final piece to the overall security issue.

When the compromise was announced, Apache was quick to note that at no time were any code repositories, downloads, or users placed in harms way. The same notice is what leads the incident report, with a note that “…we believe that providing a detailed account of what happened will make the internet a better place, by allowing others to learn from our mistakes.”

The server that hosted apachecon.com (dv35.apachecon.com)was compromised. According to the Apache report, the server is suspected to have been compromised using local root exploits against the CentOS installation. A patch for the root vulnerability was issued prior to the compromise, but the system itself was not patched.

“The attackers fully compromised this machine, including gaining root privileges, and destroyed most of the logs, making it difficult for us to confirm the details of everything that happened on the machine,” the incident report says.

“This machine is owned by the ApacheCon conference production company, not by the Apache Software Foundation. However, members of the ASF infrastructure team had accounts on this machine, including one used to create backups.”

Once they had dv35 server under their control, the attackers attempted to use passwords discovered on the compromised system to access the ASF production environment, those attempts failed. Yet, an SSH key for the backup account granted them access to people.apache.org (minotaur.apache.org), as an unprivileged user.

Minotaur “…acts as the staging machine for our mirror network. It is our primary shell account server, and provides many other services for Apache developers. None of our Subversion (version control) data is kept on this machine, and there was never any risk to any Apache source code,” Apache explained in the report.

With the newly acquired access to Minotaur, Apache noted that the attackers added CGI scripts to document root folders on several websites. A scheduled rsync process copied these scripts to the ASF production web server, eos.apache.org, where they became externally visible. The CGI scripts were used to gain remote shell access thanks to information sent with HTTP POST commands.

The incident report explains what worked and what didn’t with regard to implementation of processes and security.

However, Apache noted that from this point forward, users with elevated privileges would need to use OPIE or sudo on certain systems. In addition, they will implement a measure that was once used but fell to the wayside, as they will enforce IP banning and other measures after several failed login attempts.

While the VM that hosted the old apachecon.com website remains offline for further analysis, Apache noted that they will change the method that most of their public facing websites are deployed.

The entire incident report is online here.

Around the Web