What the MITRE Engenuity ATT&CK® Evaluations Means to SOC Teams

When the infamous Carbanak cyberattack rattled an East European bank three years ago this month few would have guessed it would later play a starring role in the MITRE Engenuity enterprise evaluations of cybersecurity products from ourselves and 28 other vendors. We recently shared the results of this extensive testing and in a SOCwise discussion we turn to our SOCwise experts for insights into what this unprecedented exercise may mean for SOC teams assessing both strategy concerns and their tactical effectiveness.

SOCwise Weighs In

The first thing to realize about this exercise is few enterprises could ever hope to match its scope. What do you get when you match up red and blue teams? “I have not been through an exercise like that in an organization with both the red team and blue teams operationally trying to determine what their strengths and weaknesses are,” said Colby Burkett, McAfee XDR architect, a participant in the event, on our recent SOCwise episode. “And that was fantastic.”

Carbanak is a clever opponent known for innovative attacks on banks. FIN7 uses the similar malware and strategy of effective espionage and stealth to target U.S. retail, restaurant and hospitality sectors, according to MITRE Engenuity, and both were highlighted in this emulation. These notorious actors have reportedly stolen more than $1 billion worldwide over the past five years. An annual event, the four-day ATT&CK Evaluation spanned 20 major steps and 174 sub-steps of the MITRE framework.

While our SOCwise team raved about the value of conducting broad scale purple-team exercises, they expressed concern that the emphasis on “visibility” is no more valuable than “actionability.” McAfee, which scored 87% on visibility, one of the industry’s best, turned in a remarkable 100% on prevention in the MITRE Engenuity evaluations.

A lot of SOC teams conduct vulnerability assessments and penetration testing, but never emulate these types of behaviors, noted Ismael Valenzuela, McAfee’s Sr. Principal Engineer and co-host of SOCwise. And, he adds that many organizations lack the resources and skills to do purple-teaming exercises.

A well-informed SOC finds a sweet spot on an axis where the number of false positives is low enough and the true positives are high enough “where you can actually do something about it,” added Netz.

Illuminating Visibility

When we think about visibility, we think about how much useful information we can provide to SOC analysts when an attack is underway. There may be a tsunami of attack data entering SOCs, but it’s only actionable when the data that’s presented to analysts is relevant, noted Jesse Netz, Principal Engineer at McAfee.

For example, in the evaluation we saw McAfee’s MVISION EDR preserve actionability and reduce alert fatigue. We excelled in the five capabilities that matter most to SOC teams: time-based security, alert actionability, detection in depth, protection, and visibility.

He believes that for SOC practitioners, visibility is only part of the conversation. “How actionable is the data you’re getting? How usable is the platform in which that data is being presented to you?”

Expanding the Scope

The SOCwise team overall applauded the progressively sophisticated approach taken by the MITRE Engenuity enterprise evaluations of cybersecurity products—now in its third year. However, our panel of experts noted that this round of testing was more about defending endpoints, rather than cloud-based operations, which are fairly central to defending today’s enterprise. They expect that focus may change in the future.

If you can’t do anything about the information you obtain, your results aren’t really useful in any way. In this regard, prevention also trumps visibility. “It’s great that we can see and gain visibility into what’s happening,” explained Netz. “But it’s even better at the end of the day as a security practitioner to be able to prevent it.”

For instance, Carbanak and FIN 7 attacks may not be relevant to your particular organization, especially if they’re centered on Cloud-based operations.

The MITRE Engenuity enterprise evaluations provide a lot of useful data, but they should never be the single deciding factor in a cybersecurity product purchase decision. “Use it as a component of your evaluation arsenal,” advises Netz. “It’ll help to provide kind of statistics around visibility capabilities in this latest round, including some detection capabilities as well, but be focused on the details and make sure you’re getting your information from multiple sources.”

While no emulation can perfectly replicate the experience of battling real-time, zero-day threats, McAfee’s Valenzuela believes these evaluations deliver tremendous value to both our customers and our threat content engineers.