Observability and incident response
Project details
Why it was created
A suspected compromise crossed many stores and heterogeneous terminals. The team needed to distinguish infection, legitimate operational drift and incomplete collection across billions of records without interrupting sales or destroying evidence.
What I created
I created the monitoring platform and forensic toolkit, connected authorized acquisition paths to Codex and Claude CLI under human supervision, and turned the response into a repeatable acquire–triage–contain–eradicate–recover workflow.
How it is structured
- 01Collect
Active TCP checks measure reachability, latency and packet loss; remote WMI/DCOM collectors acquire Windows inventory, services and system events.
- 02Correlate
Fastify and SQLite deduplicate state transitions; forensic acquisition jobs record completeness, manifests, SHA-256 hashes and chain of custody.
- 03Act
AI-assisted analysis under human supervision compares peer devices and narrows hypotheses; only a reviewed workflow can simulate, approve, quarantine and restore.
Constraints and trade-offs
+
Constraints that shaped the system
- 01Stores kept serving customers while the investigation ran, so collection and containment needed a narrow blast radius.
- 02Network access and artifact coverage varied; incomplete evidence could never be labeled clean.
- 03Customer topology, credentials and raw evidence had to remain private while operator actions stayed auditable.
Trade-offs
- 01Targeted forensic acquisition was chosen over full-disk imaging to reach the fleet quickly while preserving the artifacts required by the investigation.
- 02Human gates made response slower than autonomous remediation, but prevented an AI or script from expanding the incident through a mistaken action.
- 03Only verified indicators enabled removal; uncertain cases remained visible for manual review instead of being force-classified.
Additional architecture decisions
- Targeted forensic acquisition was chosen over full-disk imaging to reach the distributed environment quickly while preserving the required artifacts.
- Incomplete acquisition never produces a clean classification; only validated indicators of compromise can authorize eradication.
Business value
Billions of records were triaged into comparable evidence, supporting phased containment, eradication and recovery across the affected environment.
Engineering principle
“AI accelerates an investigation when evidence, authority and rollback remain explicit; it must not replace the incident commander's judgment.”