Welcome back to our series on static & dynamic analysis and unit testing. In this post, we discuss dynamic analysis, which is when you actually execute the program on a real or virtual machine to test it.
In order to use dynamic analysis accurately, you must have identified the correct targets and provide sufficient inputs to create the desired output.
Prior to the 1970s, little distinction was made between debugging and testing like there is today. If your code compiled, it was considered good to go! However, in the 1980s, the aerospace industry introduced destructive or stress testing, which deliberately strains the component or system to expose faults and understand performance. Contrast this with nondestructive testing, which evaluates standard component or system behaviors in order to collect and understand performance metrics under expected conditions.
As developers began to distinguish between debugging and testing, it became a best practice for developers to debug software and then have a separate test group evaluate it. Later in the 1990s, developers decided it would be beneficial to integrate testing into the development cycle so that bugs could be prevented before the code reached the verification stage. This was the beginning of dynamic analysis as we know it.
Forms of Dynamic Analysis
Dynamic analysis includes assessments of Modules, such as an individual C-file (and can also incorporate its matching header file), and Units, or individual functions within a module. For testing purposes, both the API and the static units are counted separately.
· Unit Testing is white-box testing conducted on the smallest testable components of the software. In procedural languages like C, this is an individual function in isolation. In object-oriented programming, this may be an entire class or interface.
· Integration Testing (also called module testing or interface testing) is grey-box testing on related or grouped modules. These tests typically go through the API, purposely constraining themselves to the interface’s functionality as it relates to the module or module grouping, and attempt to stress the interface. This includes low-level “fuzzing,”pushing garbage or noise as an input to see how the program responds.
· System Testing is black-box testing of the entire software, using only externally-available stimulus such as the Communication Interface Protocol (CIP). It involves both destructive/stress and nondestructive/evaluative testing and seeks to answers questions like, “what happens if we overload communications channels?” as well as “how long does it take the program to return the answer to a complicated set of inputs?”
Unit Testing
Unit testing consists of isolating individual units from each module and the overall software system and subjecting each group to a series of tests. External calls made by the unit are stubbed with mock functionality. Any shared parameters should also be mocked out because you want to test the individual unit as independently from the rest of the module as possible. Once isolated and mocked, each group within the module is challenged for various conditions with a pass/fail criteria.
Test Case: each test case exercises a specific functional or behavioral path within a unit (function) for a given module
Test Suite: a collection of orthogonal tests cases for a given module
Note: Test cases should be completely independent of each other. If you have to combine test suites in any manner, they are not being executed properly.
Mocks and Stubs: Mocks or stubs are specially generated functions that replace the actual function calls from the unit under test because they reside outside the module in scope. This gives the test developer flexibility to add/modify the stub functionality to inject the necessary stressed for a given test scenario or behavior. This also allows for removing hardware dependencies or requirements for testing.
Test Harness: a collection of test suites, stubs, and mocks along with test validation functionality
Test Runtime: a tool-specific run time executable or library
Unit testing provides statement coverage, meaning it assures that every line of code was executed and addresses every edge, branch, and condition. Next, it ensures that the code recognizes boundary conditions and correctly responds to an out-of-bounds input. For example, if you typed in zero or six into a one-to-five input, the boundary conditions will detect the problem and enforce backup procedures. It detects security problems stemming from low-quality code, such as using unsafe string functionality. When static analysis identifies a possible security flaw, unit testing can confirm the validity of the static analysis. Unit testing assists with fault localization, isolating specific buggy code. It includes flow analysis, which checks permutations to see if there is unnecessary or repetitive code that can be merged or deleted to save space.
Finally, unit testing encompasses memory leak detection and concurrency defects. These last two are not always required for programs developed in embedded medical applications but do apply for programs designed to run on an operating system (PC or mobile platform).
Here’s a simplified example of the kind of problem unit testing can uncover:
As written, no input value which passes the first two checks will ever fail the third. Writing a test to confirm that this check works correctly isn’t possible, so testing has successfully identified a problem with the code (results which we would expect to fail the third check always pass).
Although it would be ideal to perform unit tests as often as possible, practical limitations prevent this from becoming a reality. That said, unit testing should always be performed:
(1) When the developer feel that initial unit implementation is complete (coding/debugging is complete)
(2) After static analysis, but before formal code review
(3) Every time the unit or its parent module is updated or refactored
(4) Test-Driven Development (TDD) philosophy encourages unit testing and implementation to be done in parallel. Ideally, the tests would be generated toward the design prior to implementation – i.e., you should identify and devise testing during design review – but this is difficult in practice since it tends to push implementation milestones farther away from project start in the timeline. Look for a compromise between TDD best practice and project timeline.
Unit testing is required to comply with IEC 62304 as part of required unit verification for Class B and C medical devices. Note that ‘unit verification’ here does not necessarily mean that 100% unit testing coverage is required; risk analysis will determine how much unit testing is necessary, and which areas of the program can be assured through integration testing instead. Meanwhile, the FDA’s updated premarket submission guidance does require unit testing with boundary conditions to be performed as part of SAST.
Next time, we’ll look at unit testing with Parasoft, a configurable tool that we use and recommend for secure development of medical device software. Along with our post, we’ll release a free download bundle with a testing demo and instruction set for configuring Parasoft for your medical device project needs.
