README

README

Name

Cutter

Author

  • Kouhei Sutou <kou@cozmixng.org>

  • Hiroyuki Ikezoe <poincare@ikezoe.net>

License

LGPLv3

What's this?

Cutter is an Unit Testing Framework for C.

This is a list of features of Cutter:

  • easy to write tests.

  • outputs result with useful format for debugging.

  • tests are built as shared libraries.

Dependency libraries

  • GLib >= 2.14

Get

<URL:http://sourceforge.net/project/showfiles.php?group_id=208375>

% svn co https://cutter.svn.sourceforge.net/svnroot/cutter/trunk cutter

Install

% ./configure
% make
# make install

Usage

% cutter [Options] [Directory which has test_*.so]

Note: test_*.so are searched recursively.

Options

--version

Cutter shows its own version and exits.

-s DIRECTORY, --source-directory=DIRECTORY

Cutter prepends DIRECTORY to file name when test fails. This is for tolls (like Emacs) which have function jumping to error line.

-t TEST_CASE_NAME, --test-case=TEST_CASE_NAME

Cutter runs test cases that are matched with TEST_CASE_NAME. If TEST_CASE_NAME is surrounded by "/" (e.g. /test_/), TEST_CASE_NAME is handled as regular expression.

This option can be specified n times. In the case, Cutter runs test cases that are matched with any TEST_CASE_NAME. (OR)

-n TEST_NAME, --name=TEST_NAME

Cutter runs tests that are matched with TEST_NAME. If TEST_NAME is surrounded by "/" (e.g. /test_/), TEST_NAME is handled as regular expression.

This option can be specified n times. In the case, Cutter runs test that are matched with any TEST_NAME. (OR)

-m, --multi-thread

Cutter runs a test case in a new thread.

--test-case-order=[none|name|name-desc]

It specifies test case order. Cutter doesn't sort by default. (none)

If 'name' is specified, Cutter sorts test cases by name in ascending order. If 'name-desc' is specified, Cutter sorts test cases by name in descending order.

-u[console|gtk], --ui=[console|gtk]

It specifies UI. The default is console UI.

-v[s|silent|n|normal|v|verbose], --verbose=[s|silent|n|normal|v|verbose]

It specifies verbose level.

This option is only for console UI.

-c[yes|true|no|false|auto], --color=[yes|true|no|false|auto]

If 'yes' or 'true' is specified, Cutter uses colorized output by escape sequence. If 'no' or 'false' is specified, Cutter never use colorized output. If 'auto' or the option is omitted, Cutter uses colorized output if available.

This option is only for console UI.

--xml-report=FILE

Cutter outputs a test report to FILE as XML format.

-?, --help

Cutter shows common options.

--help-all

Cutter shows all options.

How to test

Executing flow of test is the following.

  1. Write a test.

  2. Compile it and build test_*.so.

  3. Execute cutter. It loads test_*.so and runs them.

See TUTORIAL and sample/stack/.

Test result

Here is an example test result:

..........F.................................................

1) Failure: test_test_case_count
<1 == cut_test_case_get_n_tests(test_object, NULL)>
expected: <1>
 but was: <0>
test/test-cut-test-case.c:143: test_test_case_count()

Finished in 0.020857 seconds

60 test(s), 253 assertion(s), 1 failure(s), 0 error(s), 0 pending(s), 0 notification(s)

Progress

A part that contains "." and "F" of the test result shows test progress:

..........F.................................................

Each "." and "F" shows a test case (test function). "." shows a test case that is succeeded and "F" shows a test case that is failed. There are "E", "P" and "N". They shows error, pending and notification respectively. Here is a summary of test case marks:

.

A succeeded test

F

A failed test

E

A test that had an error

P

A test that is marked as pending

N

A test that had an notification

The above marks are showed after each test is finished. We can confirm the test progress from the output in testing.


Summary of test result

Cutter outputs a summary of test result after all tests are finished. The first of a summary is a list of a detail of test result of non-succeeded test. In the example, cutter outputs a detail of test result because there is a failure.

1) Failure: test_test_case_count
<1 == cut_test_case_get_n_tests(test_object, NULL)>
expected: <1>
 but was: <0>
test/test-cut-test-case.c:143: test_test_case_count()

In the example, test_test_case_count test case is failed. We expected that cut_test_case_get_n_tests(test_object, NULL) is 1 but was 0. The failed assertion is in test_test_case_count() function in test/test-cut-test-case.c at 143th line.

Elapsed time for testing is showed after a list of a detail of test result:

Finished in 0.020857 seconds

The last line is an summary of test result:

60 test(s), 253 assertion(s), 1 failure(s), 0 error(s), 0 pending(s), 0 notification(s)

Here are the means of each output:

n test(s)

n test case(s) (test function(s)) are run.

n assertion(s)

n assertion(s) are passed.

n failure(s)

n assertion(s) are failed.

n error(s)

n error(s) are occurred (cut_error() is used n times)

n pending(s)

n test case(s) are pending (cut_pending() is used n times)

n notification(s)

n notification(s) are occurred (cut_notification() is used n times)

In the example, 60 test cases are run, 253 assertions are passed and an assertion is failed. There are no error, pending, notification.


XML report

Cutter reports test result as XML format if --xml-report option is specified. A reported XML has the following structure:

<report>
  <result>
    <test-case>
      <name>TEST CASE NAME</name>
      <description>DESCRIPTION OF TEST CASE (if exists)</description>
    </test-case>
    <test>
      <name>TEST NAME</name>
      <description>DESCRIPTION OF TEST CASE (if exists)</description>
      <option><!-- ATTRIBUTE INFORMATION (if exists) -->
        <name>ATTRIBUTE NAME (e.g.: bug)</name>
        <value>ATTRIBUTE VALUE (e.g.: 1234)</value>
      </option>
      <option>
        ...
      </option>
    </test>
    <status>TEST RESULT ([success|failure|error|pending|notification])</status>
    <detail>DETAIL OF TEST RESULT (if exists)</detail>
    <backtrace><!-- BACKTRACE (if exists) -->
      <entry>
        <file>FILE NAME</file>
        <line>LINE</line>
        <info>ADDITIONAL INFORMATION</info>
      </entry>
      <entry>
        ...
      </entry>
    </backtrace>
    <elapsed>ELAPSED TIME (e.g.: 0.000010)</elapsed>
  </result>
  <result>
    ...
  </result>
  ...
</report>

Test coverage

You can see the code coverage with Cutter if your system have LTP tools . To see the coverage, add the followling line in your configure.ac and type "make coverage".

AC_CHECK_COVERAGE

References

Assertions

See cutter/cut-assertions.h or cutter-cut-assertions.html .


Attributes

You can add attributes to your test to get more useful information on failure. For example, you can add Bug ID like the following

const char *bug_invalid_input(void);
void test_invalid_input(void);

const char *
bug_invalid_input (void)
{
     return "123";
}

void
test_invalid_input (void)
{
     cut_assert_equal("OK", get_input());
}

In the above example, test_invalid_input test has an attribute that the test is for Bug #123.

You need to define a function that returns const char * and has name "#{ATTRIBUTE_NAME}_#{TEST_NAME - 'test_' PREFIX}" to add an attribute to a test. In the above example, an attribute set function, "bug_invalid_input", is defined to add "bug" attribute to "test_invalid_input" test.


Template

The following is a template of test.

#include <cutter.h>

#include "HEADER_FILE_OF_YOUR_PROGRAM"

void test_condition(void);
void test_strstr(void);

static int condition = 0;

void
setup (void)
{
    condition = 1;
}

void
teardown (void)
{
    condition = 0;
}

void
test_condition(void)
{
    cut_assert_equal_int(1, condition,
                         "The condition value should be set to 1 in setup()");
  ...
}

void
test_strstr(void)
{
    cut_assert_equal_string("sub-string",
                            strstr("string sub-string", "sub"));
    ...
}