MITRE Corporation released findings Wednesday from its latest round of ATT&CK evaluations, assessing the capabilities of enterprise cybersecurity solutions against some of the most prevalent ransomware tactics and North Korean malware.

The sixth such evaluation from the nonprofit research organization measured 19 different vendors’ ability to protect enterprise systems by evaluating them against two prominent ransomware strains -—Cl0p and LockBit — as well as North Korean-linked malware targeting macOS systems. For the latter, MITRE’s evaluation used advanced multi-stage malware emulations that highlighted sophisticated tactics, such as exploiting legitimate macOS utilities and stealthily exfiltrating sensitive data.

According to William Booth, the general manager of MITRE’s ATT&CK evaluations, the results revealed significant disparities between vendors’ detection rates and their ability to accurately distinguish malicious activity from benign system behavior.

“Some vendors had higher false-positive rates than detection rates, which indicates a need to better distinguish legitimate activity from malicious activity,” Booth told CyberScoop.�

How the tests were conducted�

The evaluation is conducted in multiple stages.

First, MITRE runs an initial emulation plan to assess the vendors’ baseline detection capabilities. This means they execute a series of malicious activities and see which ones the vendors can detect without any prior knowledge.

After this initial detection test, MITRE gives vendors a day to make configuration changes to their products. This could involve things like adding new detection logic, updating user interfaces, or making other adjustments to improve product performance.

The purpose of this configuration change period is to allow the vendors to enhance their products based on the initial test results. MITRE wants to see if the vendors can improve their detection and protection capabilities by making targeted changes.

In the second phase of testing, MITRE runs a separate emulation plan focused on the protection capabilities of the vendors’ products, complete with a new set of malicious activities that the vendors haven’t seen before.

By separating the detection and protection tests, and allowing the configuration changes in between, MITRE can assess how well the vendors can adapt and improve their security controls in response to new threats.

What the results show�

The organization explicitly states that “the evaluations do not rank vendors and their solutions, but instead provide insights” for organizations to make their own decisions based on their unique IT systems and threat models. However, Booth told CyberScoop there were surprising findings from the evaluation’s data.�

One of the most striking discoveries was that some vendors had higher false-positive rates than actual detection rates. Booth explained that this indicates a significant need for vendors to improve the specificity of their detection and blocking capabilities.

“There are certain vendors where you’ll see, yes, they had 100% detections, but their false-positive rate was also 90%,” Booth said. “That’s really interesting when you start to look at, OK, how can [vendors] determine what needs to be detected versus what is just noise?”

Another surprising finding was the difficulty vendors faced in protecting against threats in the post-compromise stage. Booth noted that MITRE’s evaluation placed a strong emphasis on assessing vendors’ ability to detect and mitigate ransomware activities after the initial breach, rather than just the initial infection.

“The assumption that you’re always going to block on the first piece of activity is not the case,” Booth said. “We’re focused on what happens after that initial compromise.”

Many vendors seemed to struggle with this post-compromise focus, as ransomware can often mimic normal system and file encryption behaviors.�

Booth also highlighted the varied approaches vendors are taking when it comes to detection, noting some key differences between machine learning-based methods and more heuristic-based techniques.

“There’s certainly some that are using AI, applying the language models on the raw data, and then there’s others that are using more of a heuristic approach,” Booth explained.

The evaluation revealed that these differing detection strategies can lead to vastly different results, both in terms of detection rates and false-positive rates.

A first for Mac

Booth told CyberScoop the inclusion of macOS in this latest evaluation round presented some unique challenges, noting that evaluating Mac-based threats required a different approach compared to previous Windows-focused assessments.

“MacOS was a bit tougher because there’s not a lot of public CTI [Cyber Threat Intelligence] on that,” Booth said.

That lack of public threat intelligence on Mac-targeted malware campaigns made it more challenging for MITRE to construct realistic, evidence-based emulation scenarios for the evaluation.

“There’s a lot that goes into formulating [the evaluation], in terms of our discussions with many different groups and organizations to get input into doing that. But Mac was hard because there’s not a lot of public CTI,” Booth acknowledged.

Despite these difficulties, MITRE included macOS in this round of testing to better reflect the evolving threat landscape. As more organizations adopt Apple devices, understanding the security capabilities of products against Mac-based attacks has become increasingly important.

Full list of vendors

The full cohort of products that MITRE evaluated included:�

AhnLab

Bitdefender

Check Point

Cisco Systems

Cybereason

Cynet

ESET

HarfangLab

Microsoft

Palo Alto Networks

Qualys

SentinelOne

Sophos

Tehtris

ThreatDown

Trellix

Trend Micro

WatchGuard

WithSecure

The evaluation results are publicly available on MITRE’s ATT&CK evaluation website.�

The post Latest round of MITRE ATT&CK evaluations put cybersecurity products through rigors of ransomware� appeared first on CyberScoop.