The Validated Canary: Unearthing changes in our detection engine with Coalmine
– George Allen
We use several existing tools for detection validation, including the well known Atomic Red Team for reproducing TTPs tied to the ATT&CK framework, Atomic Test Harnesses for reproducing many variations of a technique, and Vuvuzela, our internal tool and test oracle for validating discrete activities, event types, detections, and data flow across numerous EDR sensors and cloud systems.Following the success of that initial playbook, we later added additional playbooks, reusing our Atomic Red Team role to run subsets of TTPs that generated an âinteresting test setâ of dataâincluding some filemods, regmods, etc.âthat were guaranteed to trigger our detections.This will be described in detail in a future blog post, but daily runs of Vuvuzela tests executed via Coalmine have already provided early warnings of bugs such as: – regressions in new integrations or the regular development cycle before they are released to customers – additional data sources that should be integrated into our detection engine to enhance threat coverage – gaps in alert correlation to EDR telemetry Automation AWS hosts the execution for the Terraform and Ansible we just described with scheduling and logging.
Link: https://redcanary.com/blog/coalmine/