Transparency Wars: Exposing Hidden Biases in Testing
At the 2016 Virus Bulletin Conference in Denver, USA, we presented “Anti-malware testing undercover,” arguing that anti-malware testing had long been controversial, and was likely to remain so. Nearly a decade later, this paper revisits that premise but in a testing landscape that is broader, more complex, and more commercially consequential.
Anti-malware testing continues to grow in complexity as new defensive technologies emerge, involving numerous stakeholders with differing objectives and often significant financial interests. While the technical challenges of testing are well understood and frequently discussed, this paper intentionally shifts focus to less explored dimensions of the testing landscape.
Instead of focusing on modern testing techniques, we examine the less discussed forces that shape testing outcomes: incentives, influence, and transparency.
Independent testing is intended to be impartial, but how independent is it in practice? In practice, “independent” testing often operates inside a web of financial dependencies and competing objectives, where vendors fund evaluations, influence test scope and methodology, and use results for competitive and marketing purposes. These dynamics create predictable pressure points, even when all parties act in good faith.
Concerns about bias are both inevitable and consequential and we will map where bias most commonly enters the process, including (1) who selects what gets tested, (2) how methodologies evolve and how changes are disclosed, (3) what is simplified or omitted in public reporting, and (4) how results are packaged and amplified. We will also examine vendor-side behaviors that can distort testing, and propose guardrails that apply to vendors, testing organizations, and sponsors.
Ultimately, this paper critically evaluates whether current transparency practices are sufficient to foster trust among consumers, media, and regulators, or whether the industry must raise its standards to maintain credibility in an increasingly scrutinized ecosystem.
Mr Luis Corrons – Gen Digital
Luis Corrons is a cybersecurity expert with more than 25 years of experience in malware analysis, threat research and security industry practices. Since 1999, he has helped people and organisations understand cyber threats and how to protect themselves against them. He works at Gen, the company behind Norton, Avast, AVG, Avira and other cyber safety brands, where he focuses on threat trends, consumer cyber safety, scams, malware and the broader evolution of the threat landscape. He regularly acts as a spokesperson for the company, providing expert commentary to media and presenting research at international cybersecurity conferences. Luis is also active in industry collaboration and testing standards. He is Chairman of the Board of AMTSO (Anti-Malware Testing Standards Organization), and a Board member of MUTE. Over the years, he has spoken at conferences including Virus Bulletin, AVAR and CARO Workshop, often addressing the intersection of malware, security testing, transparency and public trust. His work combines technical analysis with a strong focus on how cybersecurity evidence is produced, interpreted and communicated.
Mr. Righard Zwienenberg – ESET
Zwienenberg began his work with computer viruses in 1988 after encountering his first virus issues at the Technical University of Delft. This experience sparked his interest in virus behavior, leading him to study and present solutions and detection methods ever since. Over nearly four decades, he has worked for various companies, including CSE Ltd., ThunderBYTE, Norman, and ESET. He has also held or continues to hold positions in several industry organizations, such as AMTSO, AVAR, the WildList, IEEE ICSG, and serves on the Advisory Board for Europol’s European Cyber Crime Center (EC3) and Virus Bulletin. He also runs his own computer security consultancy company (RIZSC).
Zwienenberg has been a member of CARO since late 1991. He is a frequent speaker at conferences, including Virus Bulletin, EICAR, AVAR, FIRST, APWG, RSA, InfoSec, SANS, CFET, ISOI, SANS Security Summits, IP Expo, government symposia, SCADA seminars, and other general security events. Beyond his professional work in security, his hobbies include playing drums, performing magic, modeling balloons, restoring ancient computers, and much more.