Welcome back to our final installment of the series on Static & Dynamic Analysis, including Unit Testing. This last blog post will focus on the basics of the Parasoft testing tool, as well as provide you with a configuration package and customization written by Velentium engineers.
We shopped around extensively, researching tools that we could configure to perform testing against the ISO and best-practice requirements needed for medical device software and SAMD (Software As a Medical Device). Tools we liked include:
· Understand C, a visualization tool that has built-in static analysis. A great tool, inexpensive, easy to use, but it does not analyze against Cert C – only MISRA. Moreover, it does not support all the forms of dynamic analysis or unit testing we need.
· TESSY by Razorcat is great at dynamic, unit, and integration testing, but doesn’t do static analysis.
· Coverity and other web-based services offer excellent results in most cases but require the online transfer of the code being tested – via GitHub or other means – which represents unnecessary additional exposure from a security and IP perspective.
Parasoft proved to be the best fit for us. It offers the right set of configuration options and test coverages (through purchased licenses) without being prohibitively expensive. The base package works on C and C+, and additional modules are available to cover Java, C Sharp, database testing, and so on. Standards are purchased out-of-box, including specific licenses for medical device software development. From there, it auto-generates test harnesses, suites, mocks and stubs, and an easy-to-modify template that can be replicated for different tests by adjusting pre- and post-conditions. It isn’t perfect for high-complexity code and will require manual oversight, but for the most part, the autogenerated output is reliable.
Tiered Standards Compliance: Parasoft Basics
Parasoft flags code according to the following standards-based tiers:
MISRA
· Required: shall be complied with (limited deviations, suppressions)
· Advisory: considered good practice; compliance is less formal.
· Mandatory (added in 2012): shall always be complied with (no deviations, suppressions)
CERT C
· Rules: shall be adhered to (limited variations, suppressions)
· Recommended: good practice, improve code quality. specific conditions, compliance is less formal.
Parasoft checks against MISRA and Cert C by default. If it finds a violation, it flags the violation and requires that you change the code to comply. Very few deviations are allowed. However, if you determine that a violation isn’t related to your test objectives or is stemming from a section of overall program code you don’t have control over, you can suppress the violation, annotate your reasoning, and the program will drop the flag from your code. Parasoft will then incorporate how many items you have suppressed and why into its test reporting.
MISRA violations use ‘Required,’ ‘Advisory,’ and ‘Mandatory’ flags. ‘Required’ is equivalent to Cert C’s ‘Rule’ flag, and functions as described above. ‘Advisory’ flags represent good practices that are less formal in terms of required compliance, equivalent to the ‘Recommended’ flag under Cert C. Finally, ‘Mandatory’ flags are used to highlight “no exceptions” issues that cannot be suppressed or excused because of security concerns.
Those are the basics, but for testing medical device software, we need to go quite a bit further.
Getting Results++: Parasoft Customization
To achieve the output we require, Velentium configures Parasoft for:
· MISRA 2004 & MISRA 2012:A3
· SEI CERT C
· Additional security recommendations, as advised by our cybersecurity SMEs
· Additional code optimizations, bug detection, and flow analysis, as advised by our development SMEs.
In total, Parasoft provides access to over 3,790 checks across a wide variety of standards. Of those, 1,328 were determined to be potentially applicable for us, and a final determination was made to enable 522 checks explicitly scoped to meet the compliance and optimization needs of Velentium’s clients and advisees within medical device development.
We selected these tests by comparing checks derived from applicable standards against Parasoft’s built-in severity metric and building a two-dimensional flow to determine which checks were critical.
You can get access to our complete Parasoft configuration by downloading the bundle at the end of this article.
Unit Testing with Parasoft
Putting all of this together: what does unit testing with Parasoft actually look like?
Parasoft has workspace support for multiple projects. A user can place as many projects as they want, and each will be assigned a unique runtime setup. Then, they can enable/disable test suites for each unit and choose or create individual test cases for each suite. It’s very flexible, allowing on/off customization options for each case and suite.
We like this flexibility because we do a lot of cross-platform build systems development. Think Microsoft Visual Studio, GCC variants (Cygwin, Mingw), Keil uVision, IAR, Code Composer, etc., all of which Parasoft supports out-of-the-box. In many cases we can’t port the entire test setup onto our project hardware – it’s too resource-constrained to accept the whole project at once. That’s why the ability to isolate individual modules, mock and stubs, and tests on a per-module basis across these various platforms with only specific tests enabled is so critical.
One customization we’ve written is for Keil’s uVision ITM Simulator. We found that the previously-documented process and flow to execute unit tests under the ITM Simulator do not work correctly with the latest version of Keil and Parasoft because of how Keil handles the necessary compiler settings through its package management – but we developed a workaround and have shared our insights with Parasoft.
Once you’ve finished customizing, Parasoft has a useful exportable template function for storing and sharing custom properties, which is also included in our download bundle. Finally, there is a number of well-formatted reporting options available for test outputs, and test reports are validated through the certification body TṺV.
The Download
Here’s the best part of this post. If you’d like to understand precisely how we’ve configured Parasoft, see the evaluation process behind our check choices, view a demo video of Parasoft unit testing with our setup enabled, and get a taste of how Velentium puts Parasoft to work in medical device software development, click here for a free download bundle , which includes our unique customization.
This ZIP package also includes configuration and setup instructions, plus configuration for using Keil uVision with Parasoft for unit testing on projects that utilize MDK-ARM Cortex M4 microcontrollers, which are ‘frequent fliers’ in our industry sector.
If you have any additional questions, or would like to contact us directly, please send an email to info@velentium.com!