In the intricate world of systems engineering, professionals often find themselves navigating a...
Tangram’s Under The Hood blogs are technical pieces written by our engineers to share their knowledge, ideas, and cool things going on in our technology. In this post, Eric Zwirner, Senior Engineer at Tangram Flex, provides a look at the tools that have been built into our products that are used to test automatically generated APIs. Eric has a rich background in software engineering for avionics systems and cross domain solutions and provides leadership and expertise in developing solutions for Tangram customers. If you’re just starting out learning about component based software engineering, check out his previous (less technical but very informative) blog, Component-Based Software Engineering: How to Eat the Software Elephant
Let’s Get Technical: Notes from Eric
At Tangram we’ve built a powerful, configurable and robust messaging capability, which is essential to the successful design and implementation of a component-based software engineering system. We’ve included significant test features to validate the operation of the generated messaging API and provide essential artifacts for accreditation activities as well as run-time features for continuing assurance on platform. In the following sections I’ll break down the testing tools we include in our products today and why they are important for engineers working in software integration.
Testing of Tangram Pro™ Generated Messaging APIs
One of the most important capabilities of Tangram Pro is its ability to generate the source code implementation of a messaging API from Flex message specifications. Flex is Tangram’s message specification language. The messaging API’s job is to make it easy for the application developer to establish the component interface and to use that interface to communicate with other software components without the developer having to spend considerable time and effort developing the code.
We call the code generated for sending and receiving data between software components a Component Software Interface, or CSI. Recognizing the value of a library that can support a multitude of use cases, we’ve worked to generalize the generation of interface libraries to produce a Generic CSI. The Generic CSI provides a clean application interface consisting of getters and setters with names based on the field of the provided data structures in the ontology, a variety of serialization and deserialization mechanisms such binary buffer and XML, and a transport layer capable of using variation data distribution libraries to send and receive the data.
Tangram understands the importance of a robust messaging capability in order to achieve a successful component-based software solution that is safe and secure. The following sections introduce the various testing features that are available with Tangram Pro generated messaging API code packages.
Build Options
There are several options for controlling the build of the software. It is possible to influence the build to provide built-in test code as well as make the code friendly to a debugger tool. As a general note, Tangram Pro generated messaging API code is tested against the g++ and clang++ compilers.
Tangram Pro includes g++ and clang++ compiler plug-ins so that generated messaging API code can be built with the Tangram Pro environment. It is also possible to use Tangram Pro to generate the code and build it elsewhere. Tangram message API code is C++11 compliant in order to work with a wide variety of tool chains and operating environments.
Debug Mode
This mode disables compiler code optimization and includes debugging information making the build artifact friendly for use with a debugger or any tools that may use debugging information to provide better insight to the user.
Alternate Compiler Choices
By default, Tangram messaging API code is built using the GNU g++ compiler. It is also possible to use a different compiler front-end such as clang++ or one that is more focused on testing. For instance the American Fuzzy Lop (AFL) fuzz tester provides compiler front ends that can be used to build the code with additional information that can be used to guide that fuzzer.
Sanitizers
There are multiple code sanitizers that are supported by the code. A code sanitizer allows for building the code with built-in instrumentation to catch various issues at run-time. These are as follows:
- Address Sanitizer (ASAN) — This capability builds in instrumentation for detecting conditions like memory leaks, buffer overflows and use of memory after it has been freed.
- Undefined Behavior Sanitizer (UBSAN) — This capability builds in instrumentation that looks for things such as division by zero, integer overflow and a variety of other similar issues.
When using a sanitizer, the code is run in a representative way and the built-in instrumentation will stop the program and report any issues it finds. Sanitizers are commonly used when other test code as described in the sections to follow. For example, a common use case would be to build test code using ASAN when that code is being fed by a fuzz tester. This allows for checking for memory errors as the code is executed down each path that the fuzz tester finds.
Unit Tests
The generic CSI capability uses a set of static C++ code (called genericapi-cpp) at its core representing the code necessary to handle base data types such as integers, floating-point values and strings. This code is subjected to a variety of unit tests representing over 3861 assertions being applied to test its function. The Catch2 framework is used for unit testing. Along with execution of the unit tests, it is also possible to generate a code coverage report. These reports show how much of the code is tested by the unit tests.
Static Code Analysis
Generated CSI code from Tangram Pro supports three static code analysis tools as delivered. Each tool can be run from “make” rules directly.
cppcheck
This is a static code analysis tool that performs its operations by just examining the source code. No build steps are required. With cppcheck installed on the host system, it can be run against the generated CSI code by simply invoking “make cppcheck”.
Infer
Infer is a static code analysis tool from Facebook. It performs the analysis by capturing the build process and working with intermediate data generated by the compiler. With Infer installed on the host system, it can be run against the generated CSI code by simply invoking “make infer”.
scan-build
The scan-build tool is associated with the clang compiler and its associated set of utilities. The clang compiler and tool chain is associated with Apple’s work to bring Mac OS to Intel chips. The scan-build static analyzer is similar to Infer in that it captures the build process and uses intermediate compiler generated data performing analysis based on that data. With scan-build installed on the host system, it can be run against the generated CSI code by simply invoking “make scan-build”.
Generated Test Code
In addition to the test capabilities above, Tangram Pro also generates source code for a variety of different test situations that are available to the user.
Use of Fuzzers
Several of the generated tests are intended to be used solely with data provided by a fuzz tester or have the option to run in this configuration. A fuzz tester is a program that generates essentially randomized data and executes the code under test a vast number of times mutating that data each time in an attempt to find input combinations that can crash or hang the test code. Fuzzers are an extremely important tool in security testing and have found a great number of vulnerabilities in many programs.
There are two fuzzers currently being supported. The first is the latest version of the American Fuzzy Lop (AFL) fuzzer called AFL++. The second is libfuzzer. Both fuzzer have come out of work from and are used internally by Google.
Message Population
Though not test code by itself, population code is generated that allows for taking a message object and providing appropriate data for each field. Population code is used by other test code to create test messages. Population code can generate randomized data or can accept inputs from a file or data from a fuzz tester and use that to populate test messages.
End-to-End Tests
These test result in the creation of a reader and a writer program where the writer is able to create and populate messages of a particular type and send them to the reader via a supported network transport (ActiveMQ, Kafka, RabbitMQ or ZeroMQ). The end-to-end test code uses the message population code to generate test messages.
Serialization Tests
These tests perform the following operations for a given message and serializer combination:
- Create and populate the message object. Population is from randomized data or data provided by a fuzz tester.
- Serialize the message from #1 above.
- Deserialize the byte buffer created from #2 above.
- Re-serialize the message created from #3 above.
- Compares the two resultant message objects and byte buffers for equality.
The goal of these tests is to verify the process of serializing messages to byte buffers and deserializing them from byte buffers. The process should be completely reversible and should not modify or lose any of the data in the process. As mentioned above, these tests can be fed from a fuzz tester. In that case, the test support being built with guided fuzzing data to provide the fuzz tester data about how its inputs affect the behavior of the test code.
Serialization tests can be built to test virtually any of the available Tangram serializers
Deserialization Tests
These tests are intended to accept data from a fuzz tester over a vast number of iterations to ensure that the process of deserializing message data is not able to cause the code to crash or hang regardless of the inputs. This is an important set of tests to ensure that the code cannot be remotely attacked since a message interface is a doorway from the external world. The process involves building the tests using build command switches to indicate what fuzzer is to be used. The source code is built with instrumentation that the fuzzer can use to determine how well its inputs are driving code coverage. The fuzzer then runs, passing a vast amount of somewhat random data to the test fixture and recording any set of inputs that cause unexpected program behavior.
Property-Based Tests
The messaging API itself is extensively tested by these tests. Property-based testing operates similarly to unit tests with the notable difference being that static inputs are not used. Instead the property-based testing engine is able to generate a wide range of inputs as it attempts to cause the property assertions for each test to fail. By default each property-based test case runs 100 times using a variety of inputs. As part of the property-based tests a coverage report can be generated that shows that a very high level of code coverage is achieved. The property-based tests are designed to exercise each API call in the generated message set and provide excellent code coverage. The RapidCheck property-based test framework is used for Tangram generated property-based tests.
Run-Time Assurance
The Tangram software stack includes various capabilities to allow for run-time assurance activities. The term run-time assurance is used for various activities that can be performed in a running system. These may be permanent features of the system or just activated during test or debugging activities.
Transport Event Hooks
The transport mechanism is an optional capability of Tangram Pro’s software stack that allows for moving messages across a network using a variety of common messaging protocols. This includes ActiveMQ, Kafka, RabbitMQ and ZeroMQ with this capability being easily extended to others as needed.
The transport mechanism provides the ability to register callbacks that are invoked when events of certain types occur. This includes sending and receiving messages. Using this capability, it is possible to setup listener plugins that are able to evaluate and monitor messages in-flight. Tangram has demonstrated this capability for the purpose of performing run-time evaluation of correct message sequencing on an active network.
Serializer Hooks
Serializers are the component that converts message objects to their on-the-wire format prior to the message being sent on a network. Deserializers perform the reverse operation. Tangram Pro provides a variety of serializers to address the various supported on-the-wire message formats and it is easy to add new serializers as the need arises.
Similar to transports, serializer hooks allow for a callback to be resisted with a serializer such that whenever a message is serialized or deserialized, the event callback is invoked. The callback has access to the message and is able to inspect it as needed. For the case when a serialization or deserialization operation fails, the callback will be informed and can use the available information for forensic purposes.
Reader/Writer Hooks
Reader and writer objects are distributed with Tangram Pro. These objects provide an easy way to read messages from and write messages to the network. As one might expect, these objects also provide a callback mechanism that can be used when messages are transferred over the network. The difference between these events and those supplied by the transport event hooks is that transports are generally message set agnostic and are instead developed to send data using a particular network protocol. Readers and writers have full access to the parsed message making inspection of the data within quick and easy.
Run-Time Field Constraints
Generated CSIs from Tangram Pro include the ability to have assigned run-time constraints on message fields. This capability allows for assigning valid ranges to field types such as integers and floating-point values. For string fields regular expressions can be used. Once applied, the ability to set a field value can be limited by what is defined as valid for that field. This feature also allows for performing validation operations. This applies to the entire message or any subset of it. The validation capability can be a very important step in assuring that a received message is compliant with expectations.
We are working on some really important challenges at Tangram and continue to increase the capabilities and use cases for Tangram Pro. As an engineering-driven company we value input and dialogue with one another and with engineers. If you’re interested in learning more, take a look at our docs site and discussion forum, or give Tangram Pro a try.
Tangram Flex is a product-driven software company that provides custom service and expertise in system modernization, integration, assurance, and autonomy. We believe every mission deserves access to innovation, and so we deliver software research, prototypes, services, and products that enable rapid integration with confidence.
Tangram Flex has experience from DoD, Fortune 50 companies, and innovative software startups. We are dedicated to walking alongside our customers to keep pace with changes in technology. To get in touch, drop a note at https://tangramflex.com/contact
