SOC vs MITRE APT29 evaluation – Racing with Cozy Bear
MITRE just released the results of the APT 29 evaluation of 21 commercial cybersecurity products today, including McAfee MVISION EDR. This evaluation, conducted in the form of a collaborative attack and defense exercise, is based on ATT&CK®, a freely available and open source knowledge base of adversary tactics and techniques that is widely used by blue teamers (the defenders) to find gaps in visibility, defensive tools, and processes.
In this evaluation, MITRE, played the role of a red team (the attacker), using its ATT&CK knowledge base to examine MVISION EDR and MVISION Endpoint’s ability to detect the tactics and techniques used by APT29 (also known as Cozy Bear, The Dukes and Cozy Duke among others). APT29, is the group believed to operate on behalf of the Russian government that compromised the Democratic National Committee starting in 2015. This evaluation took place over a period of two days. On each day a different version of the attack comprised of 10 steps was executed using several techniques attributed to APT29.
While it’s important to note that the goal of these evaluations is not to rank or score products, our analysis of the results found that McAfee’s blue team was able to use MVISION EDR to obtain a significant advantage over the adversary, achieving:
- 100% visibility of the attack steps on Day 1, and 89% on Day 2
- 90% detection of the attack steps on Day 1, and 67% on Day 2
During the assessment we also installed MVISION Endpoint in observe, non-blocking mode. This allowed us to determine that the blue team would have automatically blocked 40% of all the attack steps performed by the red team on Day 1 and 33% on Day 2.
However, as all practitioners know, cyber defense is more complicated than what raw data can express, especially when dealing with sophisticated threat actors. Years of warfare both in the physical and cyber space have taught us that observing and analyzing raw data is useless until it is framed in a way that provides context to both attackers and defenders.
While attacker actions and behaviors can be modeled effectively using MITRE ATT&CK, models[1] like Time Based Security (TBS) or OODA loop (Observe, Orient, Decide, Act) provide the context that blue teamers and security operations teams need to make tactical defensive decisions.
Time Based Security – Protection, Detection & Response in context
Time Based Security[2] (TBS), was introduced in 1999 by Winn Schwartau and is still one of the most relevant, effective and yet terribly simple security models any defender can apply today. The principles enumerated in Schwartau’s book are essential for any blue teamer, regardless of whether you are a CISO, a SOC analyst, a security architect or an incident responder. TBS provides a systematic and reproducible method to answer questions like, how much ‘security’ a product or technology provides, or in this case, how secure your systems are against an adversary that behaves like APT29.
TBS provides a methodological, quantitative, mathematically proven method, that merges information security and risk management to support security budget decision making. For example, when evaluating how much ‘security’ a product or technology like EDR provides, security operations teams and CISOS need to find answers for these questions:
- How long are my systems exposed?
- How long before we detect a compromise?
- How long before we respond?
To illustrate, in the physical world, you can buy a safe to protect any asset, and you would know how long it would take for somebody to break through that safe. These performance ratings are generally ranked by the amount of time your valuables are safe when under attack by either burglary or fire[3]. But we would never think of just putting the safe and waiting for the bad guys to break in it, sitting idle, right? That is why we put detection mechanisms around it, motion sensors, heat sensors, window alarms, vibration sensors, cameras, and security guards to monitor them. Can we measure how long it takes for an attacker to trip any of those sensors? Absolutely! Once that alarm goes off, what do we do? We react, we call the police and they show up to limit the impact. Can we measure that reaction time? Of course! Everything in the physical security world is about time.
TBS establishes that in the cybersecurity world, just like in the physical one, protection runs parallel to detection and reaction (see Figure 1). If the intruder is willing to dedicate resources to bypass the protection mechanisms, and in the absence of any detection or reaction, the attacker can always win. In the end, compromising a system is just a matter of time.
Racing with APT29 – It’s All About Time
While many vendors focus solely on the raw data and statistics, our approach is focused on modeling how a blue teamer, a SOC analyst or a cyber defender would do against this attack, considering the TBS model. For this evaluation, our blue team used our products as follows:
- Endpoint Protection – Protection was not the focus of this MITRE ATT&CK, therefore, we assumed worst case scenario and installed McAfee MVISION Endpoint disabled, in monitoring mode. Regardless, the alarms triggered by the McAfee protection mechanism can be considered as a HIGHLY tactical detection mechanism. As every SOC analyst knows, a block is never a “block and forget”, but a “block and investigate”.
- Endpoint Detection through McAfee MVISION EDR (focus of MITRE ATT&CK).
While MVISION EDR reaction capabilities were not considered as part of this evaluation, it’s evident that a fast reaction is a key element in the TBS equation (P > D+R) for a reduced exposure and therefore to a limited impact against any adversary [4].
Using the results of the evaluation, we modeled the data following an attack timeline, grouping the techniques executed by the MITRE ATT&CK red team for Days 1 and 2 into each of the steps (attack milestones) they employed. As a SOC, our objective would be to block, detect and react as early as possible in the attack timeline, knowing that once the attacker has stolen credentials and started lateral movement, their advantage and the impact of the attack grows exponentially. For this reason, we draw a line right before the ‘lateral movement’ step. We call this the ‘breakout point’.
To represent the data for each evaluation day, we list the detection categories used by MITRE[5] in addition to:
- Block: Detections triggered by MVISION Endpoint that would have resulted in a blocked activity. These alarms would have slowed down the attacker as well as provided a highly tactical detection to the SOC.
- Host interrogation: Represents data that is manually pulled from an endpoint. In MVISION EDR this data can reside in the Cloud or on the endpoint itself, and can be retrieved through real time searches, the collection engine, or through automated investigations.
Observing Figures 2 & 3 below, the results show:
- Had prevention been enabled on the endpoints (the default configuration for McAfee MVISION Endpoint), the defenders would have blocked 29% of the steps performed by the attacker before the breakout point on Day 1, and 40% on Day 2. As a SOC, this would have met our objective of disrupting the attacker several times, slowing down the attack to extend our protection time (P).
- The blue team was able to detect 86% of the steps performed by the attacker before the breakout point on Day 1, and 60% on Day 2. The early detections (D) of these tactics and techniques, augmented with additional context provided by telemetry and host interrogation allow the SOC to reduce exposure and speed up response and remediation efforts (R).
- The blue team was able to see 100% of the steps performed by the attacker before the breakout point on Day 1, and 80% on Day 2. This visibility was readily available to the SOC without the need to use additional tools and therefore saving time.
Conclusion
On both Day 1 and Day 2, the blue team would have been able to receive early indication of an attack multiple times before the breakout point. The protection capabilities would have also disrupted the attacker several times. All this give defenders time to respond using EDR’s capabilities to triage, scope, investigate, contain, and eradicate the threat, including the isolation of the affected systems. Additionally, MVISION EDR capabilities like threat clustering and machine learning assisted investigations would have helped to accelerate the response, resulting in reduced exposure time (Exposure=Detection+Reaction) which would have allowed the SOC to manage the risk of this intrusion, reducing the impact of a compromise.
In summary, security solutions cannot be evaluated by raw data without putting them into context and into the right defensive framework. The MITRE APT29 evaluation shows how McAfee provides effective time-based security by combining protection, as well as early detection and fast response across critical points along the attack chain, enabling Security Operation teams and cyber defenders to reduce exposure and limit impact of attacks, even sophisticated ones.
* MVISION Endpoint is part of our McAfee endpoint protection technology, optimized for Windows 10.
[1] https://www.sans.org/cyber-security-summit/archives/file/summit-archive-1524596581.pdf
[2] https://winnschwartau.com/wp-content/uploads/2019/06/TimeBasedSecurity.pdf
[3] http://www.keyfinders.org/burglary.htm
[4] https://www.mcafee.com/blogs/enterprise/endpoint-security/response-required-why-identifying-threats-with-your-edr-isnt-enough/
[5] https://attackevals.mitre.org/APT29/detection-categories.html
The post SOC vs MITRE APT29 evaluation – Racing with Cozy Bear appeared first on McAfee Blogs.