Cantata Logo - unit and integration testing tool for C/C++


AutoTest


Automatic test case generation

 

 

AutoTest automatically generates passing Cantata unit tests to exercise all the C source code. It can be used on new or legacy projects, making it easy to:

Screenshot of Cantata AutoTest Video Cover

Autotest Video

What is Cantata AutoTest?

Cantata AutoTest creates complete unit test scripts or adds test cases to existing Cantata scripts. It parses the C source code to determine all possible paths through the code as required by a structural code coverage metric target:

  • 100% function Entry-points
  • 100% Statements
  • 100% Decisions
  • 100% Unique Cause MC/DC

An algorithm creates test case vectors which exercise all required code paths, using the Cantata powerful white-box capabilities to set data, parameters and control function call interfaces.

 

The test vectors drive the code, and check the parameters passed between functions, values of accessible global data, order of calls and return values.

 

AutoTest generated cases, are editable in the same ways as user generated cases, and each test case has a description of what path through the code it was created to exercise, making them easy to maintain. 

Procedural and state machine testing diagram - Cantata procedural tests verify the processing logic or algorithms in functions - flow chart

What does AutoTest help me do?

CONFIGURE AUTOMATIC TEST GENERATION

 

AutoTest generation options can be easily configured via the GUI or command line:

  • Code coverage required (Rule Set)

  • How function calls are simulated or intercepted

  • Whether to access global static variables

  • Isolation unit test or cluster integration test static functions

  • What data to check (i.e. global data, function call arguments or return values);

  • Whether global data is modified during function calls to force paths;

  • Generation max limits on time, paths, arrays etc.

AutoTest Code Generation Screenshot - AutoTest Generation options can be easily configured - GUI - command line
Cantata code coverage - Measure Coverage - Function entry points - function call-returns - basic blocks - statements - decisions -loops - relational operators - boolean operators conditions - MC/DC

Selecting a code coverage Rule Set will determine the depth (i.e. number of test cases needed to achieve the target).  The style is determined by use of the AutoTest specific and standard Cantata Workspace code generation options.

 

Isolation unit tests or cluster integration tests, using black-box or white-box testing approaches, provide flexibility to fit with whatever initial testing thoroughness and test maintainability for code refactoring is required. 

 

 

AutoTest Generation Report Screenshot - identify code testability issues - AutoTest provides useful insight into any testability issues within the source code

IDENTIFY CODE TESTABILITY ISSUES

 

Where AutoTest is unable to generate tests, or the tests are incomplete, the AutoTest Generation Report provides useful insight into any testability issues within the source code. 

 

Warning messages are written to the AutoTest Generation Report, showing where and why it was not possible to generate test cases, indicating possible problems in the source code or the ability to dynamically test it. It can therefore be a valuable supplement to code review, even if AutoTests are not dynamically run.

GENERATE TESTS WITH FULL CODE COVERAGE

 

The Cantata AutoTest Generation Report identifies the expected code coverage which will be achieved by executing the generated tests. As AutoTest cases are initially generated to pass, the only failing checks will be where a Coverage Rule Set checks achieved coverage against a target.

 

Achieving full code coverage is of course only part of the testing objectives.  AutoTest generated test cases can be reviewed and where suitable mapped to requirements (see Cantata Requirements Traceability capability for more details).  In this case, AutoTest dramatically reduces the developer effort to select test case vectors to verify the requirements.

AutoTest Results Summary Screenshot - generate tests with full code coverage - build summary - results summary - coverage summary

PLUG EDGE CASE GAPS IN COVERAGE

 

Where complex code is being tested, even detailed requirements, and thorough robustness tests may leave untested execution paths in the code, because the different combinations of input vectors required for full coverage are difficult to identify. For these ‘edge cases’, AutoTest can generate test cases to plug any gaps in code coverage if the code is not redundant.

 

AutoTest can be used on just selected functions to generate a focused set of test cases achieving 100% coverage.  Cantata automatic code coverage optimisation can at a single click delete/disable any test cases which do not add to the overall code coverage achieved.

AutoTest Coveragre Results Explorer Screenshot - plug edge case gaps in coverage - generating test cases to plug any gaps in code coverage if the code is not redundant
AutoTest Test Cases Screenshot - plug edge case gaps in coverage - generating test cases to plug any gaps in code coverage if the code is not redundant
Cantata AutoTest Test Safety net - Baseline regression testing

CREATE A BASELINE REGRESSION TEST SAFETY NET

 

Legacy code that relies on system testing is very expensive and inefficient to modify, but writing a comprehensive set of unit tests from scratch is simply not commercially viable. Cantata AutoTest automatically generates such a suite of unit tests for all selected code. These tests can act as a baseline safety net against future regression errors as code changes. Unit test are more thorough and pin-point errors more precisely, so reliance on expensive and time consuming system tests can therefore be reduced.

 

Cantata AutoTest generates a set of Cantata Makefiles, these can be easily integrated into any open source or commercial CI tools to enable fully automated testing during development, at every integration.

UPGRADE FROM LEGACY TEST TOOLS

 

Even where other unit testing tools are already in use, there can be real advantages in upgrading to the modern powerful test facilities available in Cantata.  This especially true where legacy tools are not well supported or are not qualified for safety standards tool certification needs.

 

Cantata AutoTest provides two highly automated routes for upgrading unit test tools:

 

For a code base with an accepted level of quality, existing tests can simply be replaced by an AutoTest generated set of Cantata unit tests.

 

For legacy unit tests in written in C (e.g. xUnit tools), existing tests can be incorporated into Cantata test scripts, retaining investment in legacy test cases and then enhanced by additional Cantata capabilities including AutoTest.

Man jumping from old computer to new laptop - Upgrade from legacy test tool

How does AutoTest work?

AutoTest can be invoked using a Test Script Wizard, or it can generate additional test cases in an existing test script in the Test Script Manager view. For larger code-bases it may be more convenient to run AutoTest from the command line.

 

AutoTest parses the source code and uses Cantata options to generate:

  • AutoTest Generation Report, which provides both summary and detailed information on the tests that have been generated.
  • Cantata test scripts (or additional test cases to existing scripts)
  • Cantata Makefiles to build the test executable

The generated tests are then built and run, producing the standard set of Cantata results and reports.

AutoTest Process Chart - Chart showing process of AutoTest testing source code

AUTOTEST GENERATION ALGORITHM

 

Cantata AutoTest creates complete Cantata unit test scripts from source code using an algorithm that automatically generates test case vectors that exercise code paths, check data, check parameters and record call orders. The code paths used to generate the test cases are determined by selecting the metric types (Entry-point, Statement, Decision or unique cause MC/DC) in a code coverage Rule Set. 

 

Preferences are available to influence the AutoTest generation algorithm, including the required level of code coverage, the method of controlling calls, what types of data to check, how data may be modified to force execution paths, time limits, path limits, and many more. These preferences affect the style of the Cantata tests, and limit the test generation scope. 

 

Tests can be more or less thorough, and can use white-box or black-box approaches. The decision on what style of unit tests to generate with Cantata AutoTest will depend on regression plan objectives and how code changes. For example, allowing flexibility for code refactoring of static functions without breaking black-box tests.

AUTOTEST WARNING AND ERROR MESSAGES

 

Where Cantata AutoTest generation results in incomplete tests or encounters suspect code quality, warning messages are written to the AutoTest Generation Report.  These warning messages indicate possible problems in the source code or its dynamic testability, so they can be of significant value. 

 

Cantata AutoTest will report:

  • unreachable code, where it cannot generate test cases to exercise all parts of the software under test
  • potential crash scenarios such as null pointers, an out-of-bounds array index, decisions using uninitialised variables, and division by zero
  • type truncation, where type casting might result in data loss
  • uninitialised data use, such as a field within a large structure
  • non-constant static data within a function, as its value will be unknown at the start of a test case
  • non-returning functions, where a function does not return control to the caller – although Cantata does have the capability to work around this for testing purposes.
  • implicit function declarations, where a function is called before it has been declared
More information about Cantata Autotest

 

For more detailed information about Cantata Autotest please see:  

Start
Trial
Learn
more
QA-Systems