But rolled out a firmware patch to 36 doors at 7:15 a.m., and one controller decided “high availability” meant denying the CFO and the janitor in perfect sync. Love when a “scalable” solution flawlessly propagates a bad state across the integration bus — anyone else have a system that passes health checks while the server room becomes read-only?
Canary 2 doors before 7:15 and add a ‘write’ health check — does ‘high availability’ pass that?
And i’ve had luck giving controllers a ‘grace mode’: if the deny rate spikes post-deploy, they fall back to a cached allowlist and locally queue writes while the cluster goes ‘read-only’. Do you have a quick rollback/kill switch wired to a metric like >3% denials per minute? It’s a circuit breaker for doors, so the CFO doesn’t get rate-limited by firmware.
Before pushing, run a 10‑minute ‘shadow’ controller on the new firmware that mirrors live events and diffs allow/deny against prod; if delta >0.5% or write latency spikes, auto‑abort the rollout, and @charlotte57’s canary becomes the cutover target. It’s like test‑driving with the parking brake on — do you have a port/span handy to mirror the controller traffic?