Fault handling and safe states
Assume faults will occur: detect them in time, contain their effects, move the system to a hazard-informed condition and preserve enough evidence to understand what happened.
After this module, you should be able to:
- Distinguish faults, errors and failures
- Derive responses from hazards and operating context
- Design detection, containment and recovery
- Plan fault-injection verification
A fault becomes dangerous through a chain of effects.
A fault is an abnormal condition or cause, an error is an incorrect internal state, and a failure is externally visible loss of required behaviour. The purpose of fault handling is to break the progression before unacceptable harm occurs.
A safe state is specific to the hazard and operating context.
“Switch everything off” is not a universal answer. Removing power may release a brake, stop cooling, lose containment or interrupt a physical action at its most hazardous point. Some systems need a safe sequence, a degraded mode or continued operation long enough to transfer control.
| Question | Design implication | Evidence needed |
|---|---|---|
| What hazard can this fault create? | Define the physical condition that must be prevented or limited. | Hazard analysis and fault-response requirement. |
| How quickly must control act? | Allocate detection and response time across hardware and software. | Fault-tolerant time interval and timing verification. |
| Is immediate stop safe? | Choose stop, hold, complete-current-step, controlled shutdown or degradation. | State-dependent response rationale. |
| Can the primary controller be trusted? | Add independence, diversity or an external protection mechanism where necessary. | Architecture and independence analysis. |
| Can operation resume? | Define latching, acknowledgement, self-test, service and restart conditions. | Recovery specification and test results. |
| What must be retained? | Record fault identity, time, context and configuration without blocking the response. | Diagnostic and event-log specification. |
Typical fault classes
Consider sensor absence, drift and implausibility; actuator non-response; communication corruption or silence; missed deadlines; memory corruption; task lock-up; stack or resource exhaustion; brownout and reset; configuration mismatch; excessive temperature; and loss of an external dependency.
Design the complete response, including recovery.
- Start with hazards. Identify which system conditions are unacceptable and which faults can lead to them.
- Set the response bound. Determine how long the system may remain uncontrolled before the condition becomes hazardous.
- Select diagnostics. Use range, plausibility, temporal, sequence, redundancy or active self-test checks with stated coverage.
- Allocate containment. Decide what hardware, firmware, software or mechanical feature limits the effect.
- Define state-dependent action. Specify the transition, outputs, user indication and handling of in-progress work.
- Define persistence and recovery. State whether the fault latches, which tests permit restart and when service is required.
- Challenge common causes. Examine shared power, clocks, memory, software, connectors, environments and assumptions.
Worked example: position-controlled drive
The controller commands a moving carriage while monitoring position and motor current. Position feedback becomes frozen but remains numerically within range.
A simple range check would miss this fault. A temporal plausibility diagnostic compares commanded motion, elapsed time, encoder change and current. The drive removes torque only after the mechanical design confirms that the brake can engage safely. Restart requires a successful sensor self-test and a deliberate operator action.
Demonstrate the response at the system boundary.
Faults, causes, affected functions, detection methods, latency and coverage assumptions.
State-specific containment, outputs, timing, indication, latching and recovery.
Independence, common causes, single points of failure and protection layers.
Method, injection point, system state, expected sequence and acceptance criteria.
Time-correlated sensor, command, diagnostic, state, output and alarm evidence.
Power-cycle, reset, reconnection, self-test, acknowledgement and service scenarios.
Common failure patterns
“System stops safely” does not define outputs, sequence, time or behaviour of the physical load.
A reset may repeat the fault, erase evidence or briefly re-energise an unsafe output.
A failed value remains in range and is accepted because no independent relationship is checked.
The initial fault response works, but reconnect, restart or repeated faults create a hazardous state.
Further learning
- NASA Fault Management HandbookPrinciples and practices for detecting, isolating, reporting and responding to faults.
- NASA Systems Engineering HandbookSystem design, technical risk, verification and validation across the lifecycle.
Safe behaviour must be derived, timed and tested.
Effective fault handling connects hazard, detection, containment, system response and recovery—with evidence that the complete physical system reaches and maintains an acceptable condition.