Reliability is a system's ability to perform required functions under specified conditions over time. Recovery concerns restoring service after failure; failover means switching to a backup component or site. These properties are related but need different checks. Successful failover does not prove data preservation, and having a backup does not prove service can be restored.
QA turns “the system survives failures” into observable scenarios. Each needs impact boundaries, expected degradation, experiment stop rules, and a way to confirm business outcomes after recovery. Start with a controlled test environment and an agreed failure model.
Reliability from the user's perspective
Checking a process or open port is insufficient. A service can answer health checks while losing confirmed orders. Choose a useful operation: create an order, read a balance, or obtain a document. Define success, failure, and exceeding the allowed duration.
Observe the period after failure as well. Queues may build up, retries may overload dependencies, and the interface may leave users waiting indefinitely. Restoring a process does not necessarily restore normal product operation.
RTO and RPO
RTO is the target time for restoring an acceptable service level after disruption. RPO is acceptable data loss expressed as a time interval before failure. Recovering within ten minutes differs from losing no more than one minute of records. These are independent commitments.
Before testing, agree on RTO's start and end: detection, incident declaration, and actual failure yield different intervals. For RPO, compare restored data with a known operation sequence. Backup frequency alone does not establish the recovery point achieved.
Which failures to model
Choose architecture-relevant failures: stopping an application instance, unavailable dependencies, network delays, queue failure, or insufficient disk space. A complete site outage and a brief request timeout are different experiments with different scope.
Record which components keep operating. Simultaneously disabling the database, network, and monitoring makes results hard to explain. Start with one failure and expected reaction, then move to justified combinations. Experiment complexity should help answer a specific question.
Data invariants
Define properties that must hold: an order is created at most once, stock never becomes negative, and confirmed operations remain available after recovery. For money, check consistency of ledger entries separately from the displayed balance.
Prepare a log of test operations with unique IDs and confirmation times. Compare it with final state through an authorized method. Resending with the same idempotency key and sending a new request with a different key should have distinguishable outcomes according to the contract.
Failover and failback
During failover, check routing, existing connections, discovery of the new leader, and write restrictions. Split-brain risk occurs when several nodes simultaneously believe they may modify the same data. Detect its consequences through invariants and logs rather than a green dashboard status.
Failback also needs testing. A returning node must synchronize before assuming its role. Automatic failback can cause a second interruption if the backup is still processing a backlog or the old node contains stale data.
Practical order scenario
A separate environment has two application instances and a test database. The infrastructure owner has approved stopping one instance. This example targets a limited service interruption and no duplicate orders; the team defines numerical limits from requirements beforehand.
- 1.Create a control order and confirm its availability through the user API. Start logging subsequent operations and record baseline latency.
- 2.While submitting a new order, stop the agreed instance. Expect the intended retry or a clear message about an uncertain outcome, without endless waiting.
- 3.Repeat the request using the original idempotency key. Expect one business order and a consistent response if the contract promises this.
- 4.Restore the instance, wait for readiness, and check both orders, queues, and errors across the entire interval. Expect no lost confirmed records and latency returning to acceptable levels.
This tests an application instance failure. It does not establish database restoration from backup or whole-region resilience.
Testing a backup
Restore the backup into an isolated environment with safe integrations. Check structure, migrations, data reads, permissions, and a business operation. Successful completion of the restore command alone does not establish operability.
To check RPO, create marked records before and after the selected point, then compare restored state with the operation log. Consider attachments, search indexes, and external storage separately: a restored database may reference missing objects.
Observability and stopping
Record a timeline of failure, detection, switching, and recovery. Observe user errors, latency, queue depth, replication lag, and resources. Correlate times across systems; inconsistent clocks distort event ordering.
Define stop conditions before the experiment: an invariant violation, impact beyond the test environment, lost observability, or excessive duration. Assign responsibility for stopping the disruption and restoring service. Otherwise, the experiment may leave accumulating damage instead of a useful result.
Common mistakes and conclusions
Mistakes include confusing failover with recovery, checking only availability, overlooking retry duplicates, and forgetting failback. Declaring zero RPO from one successful run without sufficient evidence and storage conditions is also unsafe.
The report should contain the failure model, test boundaries, measured intervals, checked invariants, and remaining limitations. Reliability is supported by reproducible results and a working recovery procedure the team has actually executed.