NSS Labs has issued the first test results of Breach Detection Systems (BDS). Breach Detection, sometimes called Advanced Malware Defense, is usually a gateway device that inspects downloaded executables by detonating them in virtualized environments and inspecting them for behavior that indicates the presence of malware. Command and Control communications is a key indicator that an executable is malicious.
Participating vendors with sandbox solutions were:
Fidelis (General Dynamics) XPS v7.4.2
FireEye Web MPS 4310 v6.2 and Email MPS 5300 v6.2
Fortinet FortiSandbox 3000 v1.2
Trend Micro Deep Discovery v3.5
Based on a copy of NSS Labs’ report shown to securitycurrent, SourceFire (Cisco) and Trend Micro were the clear winners in terms of detection rates. FireEye and Ahnlabs were the worst in terms of detection rates and the highest cost based on NSS Labs’ calculations of list prices over three years vs. megabit/sec of protection. Fortinet and Fidelis earned a recommended rating falling between the two extremes.
The reactions from the vendors were predictable with SourceFire, Trend Micro, and Fidelis making immediate use of the favorable results. The loudest protestations came from FireEye who issued a vehement blog post penned by Manish Gupta, Senior Vice President of Products. His post begins “We declined to participate in this test because we believe the NSS methodology is severely flawed.” He went on to make several more claims, all of which were countered by Bob Walder, Founder and Chief Research Officer of NSS Labs in a post titled “Don’t Shoot The Messenger.” Walder effectively responded to Gupta’s objections, in particular refuting the claim that FireEye did not participate willingly in the test. He claims that FireEye engineers worked with NSS Labs to configure their products. Perhaps Gupta was mistaken because the 2014 report was the result of tests begun in 2013.
Palo Alto Networks made the strategic blunder of refusing to participate in the test.
Testing of network security products by independent labs is an extremely valuable service if only to shed light on vendor claims of effectiveness and throughput. Vendors should take advantage of such results to improve their products. Trend Micro told securitycurrent that the testing alerted them to an issue in Deep Discovery’s ability to unpack compressed executables, improvements that are already in their most current release.