Traditional AV Testing: File under ‘Irrelevant’

ZDNet recently posted a video interview with me about the current state of the threat environment and the way forward for security.


 
I explained that Trend Micro had previously declined to participate in some high-profile AV tests. We felt that these tests didn’t match the reality of how threats infiltrate organisations and arguably give a false sense of security.
 
Typically, what happens in these traditional tests is that a file repository is loaded up with a collection of different viruses, Trojans and other malware. The security software is then installed and updated, disconnected from the Internet and set to work trying to detect malware. The headline scores are then generated according to the percentage of those malicious files that are successfully identified.
 
Testers would argue, I suppose, that this creates a level playing field in which to compare different software solutions. I can understand that, but it really doesn’t reflect the threat environment in real organisations, or for consumers. The most common threat vector now is the Internet; the second most common is malware downloading other malware via the Internet. Infected web pagesPDFssocial networking sites and cloud-based services represent just some of the significant real or potential threats that aren’t replicated in the traditional lab-based test environment. Traditional tests focus on the file – can this security software correctly identify this file?
 
A more holistic approach is necessary. Malware and other threats arrive through various channels and to be honest, once they have arrived then some part of your security solution has already failed. And it’s not necessarily through people breaking the rules. An email arrives from your CEO asking you to check out a web site. I’d suggest that most people will click on that link. What a good security solution should be doing is asking a series of questions on your behalf, questions that aren’t just about viruses but your security as a whole:
 

  • Is this email really from your CEO?
  • Is the link it contains hosted in a bad neighbourhood or does it contain suspicious elements?
  • Have we seen other examples of this same mail elsewhere recently?
  • Is it trying to deliver files or prompting to change settings?
  • Are those files bad?

 
The list can be almost endless, but traditional testing looks at what happens at the last line of defence. It asks one question: a bit like leaving your doors and windows open and unwatched but attaching a burglar alarm to the jewelry in your sock drawer. We believe that a security system should kick-in at the first link in this chain of events, not the last. No solution is 100% reliable at any level, but if you have multiple levels of control, each of which informs the others, then so much the better your chances of avoiding any compromise. Prevention is significantly better than a cure in such situations.
 
Going forward, a move to holistic protection networks and the centralisation of threat signatures is inevitable – new threats are detected every one-and-a-half seconds and as this trend continues, a solution based on signatures downloaded to client machines could neither keep pace, nor allow your machine to continue operating at the performance level you would expect while it’s attempting to do so.

One thought on “Traditional AV Testing: File under ‘Irrelevant’

  1. Rob Lambert

    This is absolutely a perfect approach to any type of testing, not just related to the security/AV industry. A holistic approach is the key to successful testing where we can look at the whole chain of events and not just single in on a specific requirements or area.

    It shows real professionalism to ignore any testing that offers false security and skips the bigger picture process.

    One problem that many people miss when testing is the user interaction with the system and the context the system will be used.

    I spent many years testing in the AV market and soon came to realise that the holistic approach is the only way to ensure a more comprehensive understanding of how threats evolve and how the software caters for this.

    Great article, superb interview and some perfect advice. Thanks

    Rob..

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*