Can you trust the NSS Labs report touting the benefits of IE8?

On Thursday, as Microsoft pushed out Internet Explorer 8 to the masses, a report by NSS Labs was making the rounds in the news. The report says that Internet Explorer 8 blocks two to four times as many malicious Web sites as any of the other browsers on the market today. Yet, the news online surrounding this report is misleading to say the least, and the report itself appears biased and questionable at best.

According to an IE8 press releases, Microsoft said: “...91 percent of adults in the U.S. are concerned about online fraud and identity theft in today’s economic climate, and 37 percent are less likely to shop online because they would have to give their personal information. Internet Explorer 8 offers the best security protections among leading browsers: a study released today by NSS Labs indicates that Internet Explorer 8 blocks two to four times as many malicious sites as other browsers on the market today.”

The NSS Labs report shows that IE8 is effective on its own 69 percent of the time when dealing with socially-engineered Malware. Firefox v3.07 earned a 30 percent effectiveness rate, while Safari (v.3) earned 24 percent, Google’s Chrome (v. 1.0.154) earned 16 percent, Opera (v. 9.64) earned 5 percent, and IE7 earned 4 percent.

However, the results are far from conclusive. There were only 492 sites tested in the first place, a rather small testing sample. In addition, the test relied on standard protections offered by the browser, depending on the browser used; no two will have the exact same set of default protections.

In addition, the report itself is suspect as it is funded by Microsoft. Funding aside, the timing of the report and the breakdown of testing is something that one needs to take a long hard look at.

NSS Labs started with a set of more than 1,779 URLs, which it ultimately narrowed down to a testing set of just 492 sites. These sites were confirmed, by manual testing, to be malicious. Yet, not a single tested URL is presented for comparison. This lack of peer-review opportunity leads some to argue the NSS test was completely biased.

The NSS test used two points to grade the tested browsers. A browser would pass if it blocked the Malware from being downloaded and issued a warning. Failure means that there was no warning and the Malware downloaded and installed. The sites tested, all 492 of them, were related to “socially engineered Malware,” meaning the site that hosted the Malware often finds its victims from Phishing or other Social Engineering attacks.

Yet, at no time does the NSS report take into account layered security, it does not mention the add-ons or options available for Opera, Firefox, or Safari, each of which scored poorly on the test. For example, Firefox alone offers various add-ons for layered security, Opera and Safari both have extensive options for layered defense that are configured by the user for protection. None of these extras were tested.

With regard to the size of the sample set, there is no clear answer as to why it was so small. Millions of Phishing sites host or link to “socially engineered Malware”. Relying on a test set as small as the one used by NSS offers a slanted test at worse, or at best inconclusive results. There is just no way to fairly test each of the browsers without including all of their features or, in the case of Firefox, the most popular and recommended add-ons, and letting the test access several thousand sites -- if not more.

Moreover, Safari’s score should not be counted; this is because a new version of Safari was released before the NSS Labs test. Safari 4 was released to the public on February 24, 2009. NSS Labs says that it conducted its testing from February 26 to March 10, 2009. Why then, would it intentionally pick the older browser when testing features against those of Internet Explorer 8?

Layered security, which was neither discussed nor tested in the NSS study, is the ultimate lesson to be learned. Each tested browser has a different set of default security features. Thanks to various control options and add-ons, each one of them will secure the end user differently. The only way to protect the victim from “socially engineered Malware” is to ensure they are covered with layers.

In most of the cases where a browser failed the test by NSS, anti-Virus software or other added features to a browser could have prevented exploitation. However, the lack of published URLs used in testing makes this point impossible to prove.

However, in the past, layered security has proven time and time again to be the best defense; things like NoScript for Firefox, the use of the block lists by Opera, anti-Virus integration in Safari, or sandboxing by Chrome, offer various layers of protection for the end user. None of these features are listed in the NSS report as being tested. If they were tested, why is there no breakdown on the results?

Perhaps it is unwise to use a test you paid for as proof of why your browser is better than those of your rivals. In an interview last week, Microsoft said it hopes we (the users) will look past the sponsorship issues. Yet, how can we? It is impossible not to question them, considering the testing methods and the hands-down victory that IE8 obtained as a result.

The test in itself would have been better if there had been more time spent testing malicious URLs, and the layered abilities of the browsers were taken into account with the results documented. Something else that would have added value is the ability to re-create the test outside of the environment used by NSS.

The published results, paid for and used by Microsoft, can be read here.

Like this article? Please share on Facebook and give The Tech Herald a Like too!

From our Other Sites

Man Makes Tiny Edible Pancakes with Tiny Kitchen Tools (Video)

This Japanese guy cooks up some pancakes…nothing special there right? Well he uses tiny implements to do it and makes perfect little pancakes. Kinda cool and they look tasty!

What Color is this Dress?

White and Gold or Blue and Black?
Well this one has been trending all over the web, just what color is this dress? It all started in Scotland when the mother of a bride-to-be sent a picture to her daughter asking what she thought of the dress. The bride and groom each saw the image differently, this then got posted online and picked up by some viral sites. The lighting in photo is probably  causing different people to see it as either white and gold or blue and black. Prof Stephen Westland, chair of color science and technology at a University in the UK told the BBC that it was impossible to see what other people see but that it was most […]

McLaren 675LT Pictures

Some great shots of the forthcoming McLaren 675LT. This coupe will get you to 60mph in less than 2.9 second and go all the way to 205mph.

McLaren 675LT Details

McLaren’s 675LT will debut at this year’s Geneva show and promises some eye-popping performance. The coupe only 675LT has a 3.8 liter V8 that will get you from 0-60mph in less than 2.9 seconds and to 124mph in less than 7.9 secondsMore than a third of the parts have been changed compared with its stable mate […]

McLaren 675LT Wallpaper

Some cool McLaren 675LT Wallpaper. The McLaren 675LT is the latest coupe to come from the supercar maker and has a top speed of 205mph.Click on an image to open a page with multiple sizes that you can download to use as wallpaper for your mobile or desktop.More McLaren Wallpaper.

Octopus hunts on land, grabs crab (Video)

This crab is minding its own business searching the rock pools for food when suddenly an octopus leaps out of the water and grabs it. The amazing thing is that the octopus does not just jump on the crab it actually pulls it all the way back to the rock pool it came from. If you check the second video you will see it is not unknown for octopus to come out of the water and the one in the second video has a crab with it, though is not hunting one! Octopus Walks on Land at Fitzgerald Marine Reserve The video was taken by Porsche Indrisie in Yallingup, Western […]

Stunning Mars Rover Selfie

This image by the Curiosity Mars rover is not exactly your typical selfie. It is made up of a bunch of images taken by the rover during January 2015 by the Mars Hand Lens Imager. This (MAHLI) camera is at the end of the robot’s arm. For a sense of scale the rover’s wheels are about 20 inches diameter and 16 inches wide. Check the annotated image below for more information on the surroundings. Also if you really want to see some detail click this very large image, 36mb, at NASA.  

How the Sahara Helps Feed the Amazon (Video)

Sahara to Amazon
This cool video from NASA shows how dust is transferred across the Atlantic to the Amazon rainforest and helps nourish the plants growing there. For the first time scientists have measured the amount of dust and the amount of phosphorus in the dust. The later acts like a fertiliser and helps replenish the phosphorus the rainforest loses each year, around 22,000 tons. Amazing how something we perceive as being desolate like a desert actually has an important role in sustaining somewhere we see as teeming with life. Image and video from NASA’s Goddard Space Flight Center.

Bouncing Laser Guided Bomb (Video)

This amazing video shows a laser guided bomb bouncing back up after hitting its target. We actually think this is a non-explosive bomb designed to test guidance systems but it is still pretty remarkable and somewhat scary.

South Koreans Swallowed by Sinkhole (Video)

Thankfully the couple survived their adventure.
This amazing footage taken from the CCTV on a passing bus shows the moment two pedestrians in South Korea fall down a sinkhole in the street! Rescue workers managed to save the pair, who were treated in a nearby hospital for minor injuries. According to reports the city authorities and the Korean Geotechnical Society are looking into the cause.