AV Vendors Flip Over Consumer Report's 'Dummy Viruses' 5,000 Dummy Viruses Created for Testing Purposes Adny Patrizio
The antivirus community is crying foul over a consumer magazine's tests of their products, which included creating 5,500 dummy viruses to see how well the anti-virus (AV) programs handle the unknown.
Consumers Union, the well-respected consumer product testing organization and publisher of Consumer Reports, decided it was pointless to test antivirus programs against already known viruses.
"If signatures were updated instantaneously, you would never need prediction. You'd always be protected. But it doesn't always work that way, obviously," said Evon Beckford, senior director of electronics operations for the Consumer's Union.
The aim of the tests was to see if the program is capable of recognizing a variance of known viruses. The company licensed a third-party lab to create 5,500 "test" viruses; the vast majority are variants of known viruses for testing in its report.
Consumer Reports managed to do something no one else has done. It got antivirus vendors to all agree on one thing: They hated the idea.
"The AV community has always been very strongly opposed to the creation of new malware for any purpose," said John Hawes of Virus Bulletin, in a blog entry. "There's just no need for it — plenty of new viruses are being written all the time, why would anyone in a responsible position want to add to the glut?"
"Creating new viruses for the purpose of testing and education is generally not considered a good idea — viruses can leak and cause real trouble," said Igor Muttik of McAfee (Quote, Chart) in his own blog.
"This is a really unwise thing to do. There are plenty of 'real' viruses, worms and Trojans around without well-meaning organizations generating more of them, for whatever reason," said David Emm, senior technology consultant at Kaspersky Labs.
You would think CR had been playing with Ebola strains in a buffet line of the Bellagio Hotel, judging by the industry's reaction.
Only Symantec (Quote, Chart) held its tongue, declining to comment when contacted by internetnews.com.
Peter Firstbrook, research director for information security and privacy at Gartner, is not very sympathetic. "The AV guys are being ridiculous," he said. "The biggest problem with the AV vendors is they are totally reactive to new viruses. They all do well on the known virus list. Big deal, so you can catch a known virus."
The reason for creating variants of known viruses is because most viruses are just modifications of existing viruses. "If you're a virus writer, particularly an inexperienced one, that's what you do, modify an existing one. You don't try to create a new exotic virus," said Beckford.
Firstbrook applauded CU for its approach. "Antivirus vendors need to come to terms with dealing with the unknown. They all have a predictive capability but when someone puts it to the test they scream bloody murder?" he said.
Indeed, tests done by AV Comparatives show that while antivirus programs are very good at catching the known viruses — around 98 percent for most vendors — they stumble badly when it comes to unknowns. The best performer was ESET Software's NOD32 at 58 percent recognition, which wasn't on the CU test list.
"They're telling you they have all this heuristic capability, but the best they can do is 50 percent. That's nothing, that's terrible," said Firstbrook.
Microsoft's antivirus entry, OneCare, came too late to be a part of the tests. But Beckford said in the future, the company will do an evaluation of all-in-one computer protection suites, and OneCare will be a part of those tests.
