Verification, debugging and test automation
Build confidence across code, electronics and physical behaviour: verify requirements, preserve diagnostic evidence and automate tests without hiding the system under test.
After this module, you should be able to:
- Choose the right test level for each engineering claim
- Design embedded software for controllability and observability
- Debug anomalies without losing evidence
- Build reliable host, target and hardware-in-the-loop automation
Verification asks whether evidence supports a defined claim.
Testing is one verification method. Analysis, inspection, review and demonstration may be more effective for particular requirements. The method should be chosen from the claim, risk, failure modes and the level at which behaviour becomes observable.
Place each test where it gives the clearest answer.
| Level | Best suited to | Limitations to remember |
|---|---|---|
| Static verification | Standards, data flow, control flow, interfaces, complexity and defect patterns. | Cannot demonstrate dynamic timing or physical behaviour. |
| Host unit test | Algorithms, state machines, conversions, boundary values and error logic. | Compiler, word size, concurrency and hardware behaviour may differ. |
| Target component test | Drivers, memory layout, interrupts, peripherals and target-specific behaviour. | Fixtures and instrumentation can alter timing or load. |
| Software integration test | Tasks, queues, components, protocols, start-up and recovery sequences. | Simulated hardware models may omit physical faults. |
| Hardware-in-the-loop | Real firmware with controlled sensor, load, network and fault stimuli. | Model fidelity, calibration and fixture control need evidence. |
| System test | End-to-end behaviour, physical performance, user interaction and safe response. | Root cause is harder to isolate and edge cases may be expensive to reproduce. |
Use many fast tests close to the code, but retain enough target and system testing to address compiler, processor, timing, electronics and physical behaviour. Trace each safety or security control to a method that can actually challenge it.
Automation needs control of both product and test system.
- Identify the configuration. Record source revision, build options, toolchain, binary hash, board revision, calibration and fixture version.
- Control preconditions. Reset state, clear retained data where required, stabilise supplies and verify instruments.
- Stimulate through defined interfaces. Use public APIs and electrical interfaces; reserve internal hooks for justified diagnostic tests.
- Measure independently. Observe physical outputs, buses and timing rather than accepting the product’s own report alone.
- Apply objective oracles. Include tolerances, time windows, permitted intermediate states and unambiguous pass/fail rules.
- Preserve artefacts. Store logs, traces, waveforms, versions and failure context automatically.
- Detect test-system faults. Distinguish product failure, fixture failure, infrastructure failure and inconclusive execution.
Worked example: communication-loss safe response
A HIL rig must show that a controller removes drive within 250 ms after loss of valid host messages.
The acceptance point is measured electrically, not from a firmware log that shares the same clock and logic as the function under test. The rig repeats the test around timeout boundaries and under maximum background load, then retains bus capture, output waveform and product configuration with the result.
Preserve the symptom before changing the system.
Exact configuration, preconditions, stimulus, timing, symptom and occurrence rate.
Software events, bus traffic, electrical signals and physical outputs on a common timeline.
Predicted evidence for plausible causes and one controlled experiment at a time.
Trigger, underlying defect, propagation path and explanation of all material observations.
A test that fails before the fix, passes after it and covers neighbouring conditions.
The fastest trustworthy automated level that will prevent recurrence.
Common failure patterns
Single-stepping, halted peripherals or altered optimisation removes the timing-dependent fault.
The same defective software both performs the function and declares that it succeeded.
Intermittent automation failures are retried away until genuine timing defects are normalised.
Executed lines are counted without showing that requirements, boundaries and failure modes were challenged.
Further learning
- Zephyr Test Runner (Twister)An example of automated test execution across emulated and physical target platforms.
- NASA Software Engineering HandbookGuidance covering software verification, testing, reviews, configuration and assurance evidence.
- GoogleTest documentationA widely used framework for repeatable unit and component tests in C++.
Trust evidence that is controlled, independent and repeatable.
Strong embedded verification combines the right test level, known configurations, objective observations and automation that preserves rather than obscures diagnostic evidence.