Local diagnosis for POS systems
Project details
Why it was created
Support saw one generic symptom — “the POS platform is down” — while the real cause could live in dozens of processes, ports, log patterns, databases, HTTP interfaces or a wrong node address. Manual diagnosis depended on specialist memory.
What I created
I mapped the platform dependencies from production incidents, designed the health model and implemented the Rust agent, collectors, local console, approved actions, installer and operational documentation.
How it is structured
- 01Collect
TOML-configured collectors inspect processes, TCP ports, network identity, logs, database integrity, HTTP health checks and Windows host resources without hard-coding site-specific rules.
- 02Correlate
Persistent state correlates PID changes, log read positions and fiscal sequencer progress to detect crash loops and regressions between collection cycles.
- 03Act
A local HTTP API serves the web console, JSON endpoints and Prometheus metrics; audited allowlisted actions support recovery without arbitrary command execution.
Constraints and trade-offs
+
Constraints that shaped the system
- 01The agent had to run on Windows hosts with minimal dependencies and survive reboots unattended.
- 02Diagnosis needed to remain useful when the central network path was unavailable.
- 03Recovery controls could not become a generic remote shell inside the store.
Trade-offs
- 01A compact Rust binary and a small dependency set favored predictable deployment over a larger observability framework.
- 02Local-first operation reduced central visibility initially, but kept diagnosis available during connectivity incidents.
- 03Configured actions limit flexibility by design; safety and auditability matter more than arbitrary control.
Additional architecture decisions
- A compact Rust binary was favored over a larger observability framework to keep Windows deployment predictable and dependency-light.
- Local-first operation preserved diagnosis during central connectivity failures, accepting initially reduced fleet-wide visibility.
Business value
A generic outage became a precise component, probable cause and allowed next action in a five-second diagnostic loop.
Engineering principle
“Good observability does not produce more logs; it preserves state and explains which dependency changed before the symptom appeared.”