As reported last week, The Tech Herald has taken Comodo’s CEO up on his challenge to test his Internet Security suite against Symantec’s. The aim was to see which product would protect a system better. In this review, we’ll explain our testing process, and list the performance for each vendor.
As mentioned on Friday, Comodo CEO Melih Abdulhayaglou issued a challenge to Symantec to see which of their products would “protect users better”. [Original story here.]
Comodo and Symantec offer solid products when it comes to protection. We’d like to make this point clear from the start.
Comparing them apples to apples, each vendor offers a product that attempts to detect and remove a system infection with more than a simple Malware signature. The names for their technology are different, but the intent is the same, they look at the big picture and watch the whole system.
Plenty of tests were run by the Comodo community comparing the two products, but depending on the reviewer, the tests weren’t as even or as fair as they could’ve been. The object of a fair review is to come as close to real world system usage as possible, while exposing the system to current threats via real world attack vectors.
Real world testing of security systems and Malware samples can be a risky proposition. It’s not as easy as one would think. To compare Comodo and Norton, we first had to determine how to do so with a level playing field. When you’re not a hundred million dollar testing lab, evening the playing field is easier said than done.
For our test, we created a master image and loaded it on a Dell OptiPlex GX280, with an Intel Pentium D 3.4GHz CPU (Dual Core) and 2048MB of RAM. Windows XP Professional SP3 was the OS, and it had all of the proper system patches and updates installed. To ensure that the master image was clean, we started with a new installation of Windows and updated from there.
When it comes to third-party software, these applications were installed after the fact on the master image. Also, we kept the Internet connection active for both products during testing, offering each one a chance to use every layer of protection available to them.
In the pages that follow, we will explain the test performed and then detail the performance of Comodo and Norton.
Installation - Usage - Scanning:
Both Norton and Comodo install rather painlessly. Comodo will use their cloud service to scan running processes, and offers a modular installation where the Comodo Firewall and anti-Virus engine can be installed separately. Norton installs in a single setting, the only task a user has to perform is the option to join the Norton Community, which is a global network of threat monitoring.
While there is more for the user to do when installing Comodo, the options are clearly explained, and the default settings are recommended for maximum protection. Likewise, once installed, Norton by default has all of its services and protection layers enabled. Out of the box, both products are good to go.
Each security suite needed a signature update. If left to themselves, they will do this automatically. However, before running a full system scan, our first mission was to update each program.
When it comes to usage, we found both Comodo and Norton easy to navigate. However, both have dark interfaces, and on a black or dark desktop, the contrast may bother some users. Yet this is really the only complaint, as the controls and software options for both products are clearly marked and visible.
One observation from the full system scans is Norton will whitelist known legit files, and as a result, their scanning will get quicker over time. Comodo, at least in our tests, showed no such patterns.
Where a full scan of Drive C: on Norton takes less than four minutes, a similar scan on Comodo dragged out to over an hour. While we didn’t alter default options, it’s worth noting that both programs have settings to increase both the speed of a scan, and the depth of scanning itself, for a fuller level of system coverage.
Detection - Sweep Scan and Access Scan:
The Sweep Scan is a test that takes a collection of known Malware samples and checks the signature detection prowess of the security application. The process starts by taking the Malware collection and loading it to a folder on the system. Next we’ll let the security suite scan the folder to see what it will pickup using signatures alone. This leads to the Access Scan test, which we'll explain later.
When it came to Sweep Scan, as expected, both products missed samples. Neither missed the exact same sample, with the exception of a few AutoRun files. However, as mentioned, this test is a simple sweep, and does not serve as an accurate accounting of a security vendor’s detection abilities.
Once the Sweep Scan is finished, the samples that remain are noted. From the original collection, we loaded the missed samples onto the system one at a time, and then executed them. The point of this Access test is to see if a given security suite will notice the Malware activating and take action.
Using the Sweep and Access tests together will show a strong picture of how well a security suite will protect a system. This is because you can see both the access detection and signature detection side of a program running in sync. At the same time, this is not a complete picture, as there are several vectors to introduce Malware to a system.
During these tests, Comodo and Norton shined. They were quick to act once a sample started digging into the system.
When it came to Comodo, the Sandboxing technology, known as Defense+ flourished, but Comodo’s Internet Security still had some issues. First, while Defense+ responded to every one of the tested samples, it did not prevent the Malware from activating in some cases. When it came to complete removal, there were issues as well.
Generic Adware [VirusTotal]
This Adware will run at startup and capture information, shipping it off to a remote server. It was missed during both the Sweep and Access test for Comodo, but removed during Norton’s initial sweep.
Buzus_A - Trojan [Virus Total]
This Malware was missed by Comodo on the Sweep scan. During the Access test, it was sandboxed instantly. While sandboxed, a TMP file (the actual payload) was created by the sample and properly flagged. However, the original sample remained, allowing us the chance to activate it at will to trigger the payload. Due to this behavior, we count this as a fail.
Norton flagged the Malware during the Access test, and removed both the container and the payload immediately.
Buzus_C - Trojan [Virus Total]
Like the other Buzus sample, this file yielded the same results exactly from both Comodo and Norton.
Trojan - KillAV [Virus Total]
This sample was missed by Comodo during the Sweep scan. During the Access test, it was sandboxed, but allowed to run and create VBS files. The fact it was still allowed to create VBS files is why it was marked as a miss.
Norton flagged and removed this sample during the Sweep test.
Trojan - Autoit.za [VirusTotal]
This sample was missed by both Norton and Comodo during the Sweep test. However, when the Access test was performed, Norton flagged and removed the file the moment it was executed.
When Comodo was tested, the Malware was sandboxed, but it spawned processes on the system, which led to an attempt to establish a local connection on port 80. The Comodo firewall warning, initiated when the Malware attempted to establish a connection, could confuse some users, who would likely just allow the process.
(When allowed, the connection to 127.0.0.1:80 remained until the Malware was removed by Malwarebytes AntiMalware.)
The fact the Malware was able to create folders and spawn processes led us to mark this as a failure. While confusing, we will not count the port 80 connection against Comodo, as the firewall did issue a warning. We just ignored it.
Detection - Access Scanning with USB drives:
During the Sweep Scan, both Norton and Comodo missed some Autorun.inf files associated with known Malware. However, to allow them a chance to legitimately detect and remove the malicious samples, we needed to do a variation of the Access test.
Autorun.inf files are part of an attack vector when a compromised USB drive is loaded onto a system. In our opinion, the only way to test Norton and Comodo fairly when it came to detecting these threats was to replicate the actual infection vector. We were looking for both false positives as well as exact detection.
For Comodo, when the INF file was loaded onto a USB and connected to the system, if the INF was coupled with a known malicious file, it was detected and blocked. This was the same for Norton.
If the INF file was by itself, it was blocked if it matched a cloud-based signature or a local signature, otherwise it was correctly ignored. Again, both Comodo and Norton acted the same way, blocking what was needed, and leaving the harmless files alone.
Unless the INF creates a malicious file or spawns a process, it is harmless without the executable it is supposed to be coupled with. So it was a strong positive to see both Norton and Comodo act by judging the file’s actions over its settings.
Detection - Web-based threats:
When it comes to Internet based threats, the only way to get a solid result from the tested security software is to visit known malicious sites and observe the security suite’s reaction. In a perfect world, the site won’t load. If it does and a payload is delivered, then the security software should remove the threat.
It’s important to note that a majority of Internet based threats come from compromised legitimate sites. Due to the fact that you never know when a legit site has been compromised, we were unable to test this vector of attack. Still, the payloads on our tested sites are similar to the ones from compromised hosts, so we felt this was a fair comparison.
Both Comodo and Norton were given a list of ten malicious sites to test. Due to the fact the average life expectancy of a malicious domain is less than a day, the only domain that appeared in both lists was greenbuddylandscaping.com. For this domain, both products prevented the Trojan delivered from infecting the system.
With regard to the nine other sites tested, neither Comodo nor Norton failed to prevent infection. Comodo used cloud-based detections, combined with sandboxing and heuristics, to prevent the domain from attacking the system. Comodo’s response to a threat was on par with Norton - instant detection and remediation.
It should also be noted that for some of the domains, even before Comodo or Norton could respond, Internet Explorer 8, as well as Firefox 3.6.10, warned against a malicious file or domain. We ignored the browser warnings, and in every case the security software protected us.
Removal and other issues:
When it came to disinfection once a system was attacked, Comodo had some issues. As seen in the Access test, while payloads were flagged, the actual Malware remained on the system. Norton too failed to remove something, as a registry key was left behind from a piece of Adware.
In addition, Comodo prevented Process Explorer and Process Monitor from accessing memory during testing, which is the downside to the Defense+ protection. While you can exempt software from being blocked by Defense+, we did not allow an exemption as to keep the stock settings in place. Either way, the two applications are established as known “goodware”, so there is no reason why they should be blocked.
Norton listed both applications as trusted and made no effort to block them when they ran.
Comodo Internet Security: Final score 96.41 / 100 points
Scanning and Detection: 7 / 10
Slow full system scans and system resource usage played a factor in the scanning part of the grade. At one point more than 200 connections from cmdagent.exe to Comodo’s services were noted during the test. This seriously lagged the system.
The samples that were blocked after access, but remained behind on the system played a critical role in the overall detection score. Adding to that was the Defense+ block of legit applications.
Malware testing (Sweep and Access): 79.41 / 80
Comodo missed the five samples mentioned previously. This is where the difference between the two security suites stood out. The tested samples were sandboxed immediately when accessed, while this is a positive, when they were not fully removed the test fell to pieces.
Malicious URL: 10 / 10
When it came to malicious domains, Comodo correctly prevented infection. There were times when the sandbox played a role in the detection, and other times when a cloud signature recognized the payload.
Symantec Internet Security 2011: Final Score 99 / 100 points
Scanning and Detection: 9 / 10
Norton flagged every sample tested, either during the Sweep or Access part of the test. Yet, secondary testing located a registry key that was missed.
Like Comodo, Norton was used for full system as well as quick scanning before a final secondary scan of the system. Given that Norton’s scanning checks the registry, there is no reason for this wayward key to be missed.
Malware testing (Sweep and Access): 80 / 80
If there wasn’t a signature available, once activated, all Malware samples were flagged and removed by Norton, aside from the one registry key.
Malicious URL: 10 / 10
Download Insight and File Insight were the main reasons none of the malicious pages were able to infect the test system
The Detection part of the final grade is where any traces of Malware left on the system were counted against the product. This is because points are awarded for detection and overall removal. The Sweep and Access tests are used to grade the detection and initial removal parts of the security software.
Even if the trace left on the system is harmless, it will count against the product simply because it remains. In Comodo’s case, four out of the five Malware containers were left on the system, even after their malicious payloads were flagged. Given that the containers remained, this is why they lost points in both areas.
For Norton, the samples were flagged and removed on access, but the registry was not cleaned fully. This means that while there were no Malware containers left to count against Norton during the Sweep and Access test, they did lose points for leaving the registry key.
Comodo’s Internet Security and Symantec’s Norton Internet Security 2011 are robust security applications offering multiple layers of protection. They both impressed us, and there is no question that they have what it takes to protect a user’s system.
The total package, including full removal of threats and application monitoring protections, is what separates the two security products. In our test, it was a close race, but overall Norton was the winner.
However, this does not mean Comodo Internet Security is any less valuable, it just needs some backup. When it came to detection, Comodo stood its ground admirably, but the failure to fully remove all traces of Malware is where Comodo had issues.
If paired with Malwarebytes AntiMalware or SUPERAntiSpyware as a secondary layer of defense, Comodo can help offer a solid level of protection to your system. The trick is to make sure that you run regular scans of both products to ensure your system is clean.
Still, many consumers would rather install a single security product and then move on with their lives. Here is where you see the commercial offerings come in to play, as they merge the layered the defenses from several products into a single offering.
It isn’t that they are better than free software, as some commercial offerings are awful when it comes to various types of protection, but some consumers prefer a “set it and forget it” approach to security software. They just want it to work. They don’t want manage it or use it in every aspect of their lives online.
While security experts and geeks alike are quick to offer their recommendations as to the best security software or vendor, the final decision is up to the consumer. Do you want our advice? Download Comodo and try it for a week. Then download Norton’s 30-day trial and use it for a week.
Once installed, use your computer as normal, and do very little to change the basic settings of either security application. After you have tried them both, use the one that best suits your needs.
If testing these two with others, avoid security software that prevents you from actually using your system, by draining it during background scans for example, and stick to one that simply sits there and protects you.
While performing this test was fun, no single review can ever give you all the answers or account for everyone’s computer usage. In the end, we know that when coupled with safe personal computing habits, the best security software is one that you picked on your own.
Feel free to share your thoughts and experiences if you try both products, even if you opt to run with another security offering.
There were 673 samples in total used during the test. They came from various sources including IRC trading rooms, Email attachments, Drive-by-Downloads, and other Malware related toolkits.
As a disclosure, Comodo offered 20 samples that were not used in this test. The reason for not using them is that Symantec policy means they could not offer testing samples. We expected that, given that normally vendors do not send samples over to use in official tests. We felt it best then to not use any submitted samples on the test, as per our normal practice.