Checks a release or launch: rollout, rollback, alerts, and load. Use it for this input: release / launch. Report each check as PASS, FAIL, or N/A with file:line evidence. A FAIL carries the card's severity. Any blockFAIL means request changes. A should-fix or discussFAIL goes in the review but does not block.
FAIL [block] chg-expand-contract - src/a.py:42 - renames the column and drops the old one in one migration
FAIL [should-fix] chg-one-concern - src/b.py:7 - a logging rewrite rides along with the bug fix
PASS chg-breaking-change-taxonomy - src/c.py:15 - no signature or return shape changes
N/A chg-perf-claim-evidence - the description makes no performance claim
prod-canary-staged-rolloutblockreliabilityRoll out in stages behind a canary, and abort by rule, not by judgment.
prod-kill-switch-fallbackshould-fixreliabilityMake a risky feature switchable off without a deploy, and exercise every fallback path in test or in production on a schedule.
prod-latency-slo-percentileshould-fixperformanceobservabilityDefine latency SLOs and alerts on high percentiles, never on the mean.
prod-load-test-before-launchshould-fixperformancereliabilityLoad test before launch, past the expected peak, with pass and fail thresholds written down.
prod-profile-on-demanddiscussperformanceobservabilityMake a CPU and memory profile of a production instance obtainable, through a continuous profiler or a private profile endpoint.
What the badges mean
blockmust be fixed before merge or release.
should-fixfix it, but it need not block.
discussraise it and decide together.
stablevetted; use for verdicts.
draftnot vetted; do not use for a verdict.
Tool gate: what a linter or CI check could enforce instead of a reviewer.
Filters
Concerns
Topic tags
Topics
Cards
prod-alerts-symptoms-golden-signals
should-fixstableobservability
Applies when: A service ships alerts or dashboards.
Rule. Page on user-visible symptoms, such as error rate and latency, not on causes.
Why. Cause-based pages (CPU high) wake people without user impact. A mean hides the slow tail, and failed-request latency mixed with successes hides both. Require an error-rate alert to hold for a few minutes, so one blip does not page. Also cover latency, traffic, errors, and saturation, track latency as a distribution, and delete alerts that never fire.
FAIL
alert: cpu > 80% -> page
dashboard: avg latency only
PASS
alert: error ratio > X, for 5m -> page
latency p50/p99 histogram, errors split from successes
Counterpoint. Cause alerts help diagnosis. Keep them as tickets or dashboard hints, not pages. The Honeycomb wide-event camp argues against pre-set metrics, while the SRE book favors simple metric-based paging, so this card does not ban events, only cause-paging.
Applies when: A change, config or dependency update is rolled out to production.
Rule. Roll out in stages behind a canary, and abort by rule, not by judgment.
Why. A 100% push exposes everyone to the bug. A canary must be large and long enough to be representative. Errors of 20% on 5% of traffic are only 1% overall, so tie the canary to your error budget.
FAIL
deploy: all hosts at once, no comparison, rollback = "ask someone"
PASS
# compare the canary to a control on a few chosen metrics
5% canary 30m -> compare error rate, p99 -> auto-rollback if worse -> 25% -> 100%
Counterpoint. "Hotfixes can't wait." Then keep the canary short, not absent. Fleet tools should also have velocity limits. A hermetic, reproducible build tied to one revision (code-build-hermetic) is the precondition for rollback to mean anything.
Tool gate: Deploy pipeline with canary stage and automatic abort.
Applies when: A new risky feature or a dependency fallback is added.
Rule. Make a risky feature switchable off without a deploy, and exercise every fallback path in test or in production on a schedule.
Why. A deploy is the slowest way to stop a feature that is hurting users. An untested fallback fails exactly when you need it: a fail-open path that never ran under load can overload the systems behind it, and a kill switch applied to a case it never met can raise its own error. Run the service with the dependency turned off, such as the cache disabled, and check that the safeguards hold.
FAIL
if redis_down:
return backup_lookup(key) # never run since it was written
new_checkout(cart) # no flag; turning it off needs a deploy
PASS
if flags.enabled("checkout_v2"): # runtime toggle, default off
new_checkout(cart)
# chaos job, weekly: block Redis for 10 minutes and assert backup_lookup serves
Counterpoint. "Flags add branches." True for release flags, so delete each one after the rollout. A kill switch is meant to stay: keep it, give it an owner, and exercise it with the fallbacks, so it still works the day you need it. For runbook steps that a person runs, such as a rollback or a failover, see inc-rollback-path-tested.
Applies when: A latency SLO, dashboard, or alert is defined for a service.
Rule. Define latency SLOs and alerts on high percentiles, never on the mean.
Why. A mean can look healthy while 1 request in 100 takes seconds, and those are the users who leave. Averaging per-instance quantiles is also wrong: quantiles do not add, so compute the percentile from a histogram, the one form that aggregates correctly across replicas.
histogram_quantile(0.99, sum by (le) (rate(request_seconds_bucket[5m]))) > 0.3
SLO: 99% of requests under 300 ms over 28 days
Counterpoint. "Percentiles are noisy at low traffic." They are, so set a minimum request count before the alert fires, or alert on the error budget burn rate. Pick bucket edges around the objective, or the percentile will be a guess between two wide buckets.
Tool gate: Lint alert rules for avg() over latency or over pre-computed quantiles.
Applies when: A new service, a new feature with its own traffic, or a large architecture change is about to launch.
Rule. Load test before launch, past the expected peak, with pass and fail thresholds written down.
Why. Overload behaviour is hard to predict from code, and launch traffic often beats the estimate. Test the whole workload on production-like infrastructure: testing one component, on smaller machines, or only up to the expected load misses the queue that fills first.
FAIL
launch checklist: "load test: n/a, staging handled 50 rps fine"
expected peak 2,000 rps; no test above 1,000
PASS
k6 stress run on the prod-sized env: ramp to 3x expected peak (6,000 rps)
thresholds: p99 < 400 ms, errors < 0.1%; result linked in the launch doc
Counterpoint. "A production-sized test environment costs too much." Size the test to the risk. A scaled-down environment is acceptable for a non-critical workload, and skipping the test is low risk for a small one. For a launch that could take down a shared dependency, the test is cheaper than the outage. Roll out in stages either way (prod-canary-staged-rollout).
Tool gate: Load-test job with thresholds (k6, Gatling) in the release pipeline.
Applies when: A service runs in production where CPU, memory, or latency problems would need a diagnosis.
Rule. Make a CPU and memory profile of a production instance obtainable, through a continuous profiler or a private profile endpoint.
Why. Test environments rarely match production load, so the hot path in a test run is often not the hot path in production. When latency spikes, a redeploy to add profiling loses the moment and changes the system you wanted to observe, so the profiler is in place before the incident and the runbook says how to reach it.
FAIL
runbook "high CPU": restart the pod
no profiler agent, no /debug/pprof, no sampling profiler in the image
PASS
continuous profiler agent at ~2% overhead, 14-day retention
or: pprof on 127.0.0.1:6060 only, reached by port-forward; steps in runbook
Counterpoint. "Profiling costs CPU and opens an endpoint." Sampling profilers stay at a few percent, and you can profile one replica at a time. Bind the endpoint to localhost or an admin port. A profile endpoint on a public listener is a security fail.
A draft card is not vetted. Read it for ideas, but do not use it for a verdict.
prod-a11y-audit-before-launch
should-fixdraftaccessibilityusability
Applies when: A new UI service or a major redesign is about to open to the public.
Rule. Before a public launch, run an accessibility audit against WCAG 2.2 AA that includes assistive technology, and link the report in the launch checklist.
Why. Quick checks and automated tools are a first pass, not an evaluation. A page can pass both and still block a screen-reader user from finishing the task. Finding that after launch means fixing it under load and in public.
FAIL
Launch checklist
[x] Load test [x] Rollback plan [x] Monitoring
(no accessibility line)
PASS
Launch checklist
[x] Accessibility audit vs WCAG 2.2 AA, with screen reader, magnifier, and voice input.
Report: docs/audit-2026-10.pdf. Open issues: 2, both minor, owner and date set.
Counterpoint. "We have no budget for an external audit." The auditor can be in-house, if they are trained and did not build the service. Cut scope instead of the audit: audit the main task flows first, and say in the report which parts are not covered.
Tool gate: none; a launch checklist item that links the audit report.
Applies when: Reads, writes, or a pipeline's traffic move from an old system to a new one, such as a new data store, a database upgrade, or a rewritten service.
Rule. Before you move traffic to a new system, run it beside the old one on live or captured production traffic, diff the results, and cut over only when every mismatch is fixed or explained.
Why. Fixtures cover the cases someone thought of, and production holds the rest. Airbnb's replay of real queries against MySQL 5.7 and 8.0 found queries whose row order was never fixed, and an application that sent the same cacheable query over and over. Neither shows up in a test suite. A diff on real traffic finds these while the old system still answers users, so a mismatch costs a log line instead of an incident.
FAIL
def get_order(order_id):
return new_store.get(order_id) # cut over in one step; never compared on real reads
PASS
def get_order(order_id):
old = old_store.get(order_id) # users still get the old answer
try:
if new_store.get(order_id) != old:
shadow_mismatch.inc()
log.info("shadow mismatch", extra={"order_id": order_id})
except Exception:
shadow_error.inc()
return old
# the cutover flag flips once shadow_mismatch is zero or every mismatch is explained
Counterpoint. "Running both is slow and doubles the load." It is slower than a flag-day switch, and Netflix says that is the reason it trusts the result. Keep the cost down: shadow a sample, call the new system off the request path, and never shadow a call with side effects such as a charge or an email. For writes, compare data snapshots after replication, as Stripe does, instead of sending each write twice. A refactor inside one process needs a characterization test, not production traffic. See chg-refactor-has-behaviour-pin.
Applies when: A service has users or dependents who rely on its availability.
Rule. Define an SLO with an indicator and window, and use its remaining error budget to gate risky releases.
Why. Without a stated target, nobody can say whether reliability work or a freeze is warranted, and every incident is argued from feelings.
FAIL
"we aim for high availability" ; no SLI, alerts at arbitrary thresholds
PASS
SLO: 99.9% of requests succeed in 300ms over 30d ; budget burn alert; freeze at 0
Counterpoint. "SLOs are bureaucracy for small teams." A one-line target is enough. Skip it for internal tools with no users.
prod-wide-events-bounded-labels
should-fixdraftobservability
Applies when: A request handler or job emits telemetry.
Rule. Emit one structured event per request carrying trace, tenant, build and flag state.
Why. Many narrow log lines cannot be joined to answer "who is affected". A metric labeled with user id explodes cost. Without a deploy marker, you cannot compare before and after. Also keep metric labels bounded and record deploys and flag changes as telemetry.
Counterpoint. This is a vendor position (one Honeycomb post), and its "metrics are the past" claim is not a review criterion. Only the structural checks above are. Wide events cost storage; sample if needed.
Tool gate: Lint for high-cardinality metric labels.