Testing libc++
Getting Started
libc++ uses LIT to configure and run its tests.
The primary way to run the libc++ tests is by using make check-libcxx.
However since libc++ can be used in any number of possible configurations it is important to customize the way LIT builds and runs the tests. This guide provides information on how to use LIT directly to test libc++.
Please see the Lit Command Guide for more information about LIT.
Setting up the Environment
After building libc++ you must setup your environment to test libc++ using LIT.
-
Create a shortcut to the actual lit executable so that you can invoke it easily from the command line.
$ alias lit='python path/to/llvm/utils/lit/lit.py'
-
Tell LIT where to find your build configuration.
$ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
Example Usage
Once you have your environment set up and you have built libc++ you can run parts of the libc++ test suite by simply running lit on a specified test or directory. For example:
$ cd path/to/src/libcxx
$ lit -sv test/std/re # Run all of the std::regex tests
$ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
$ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
Sometimes you'll want to change the way LIT is running the tests. Custom options can be specified using the --param=<name>=<val> flag. The most common option you'll want to change is the standard dialect (ie -std=c++XX). By default the test suite will select the newest C++ dialect supported by the compiler and use that. However if you want to manually specify the option like so:
$ lit -sv test/std/containers # Run the tests with the newest -std
$ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
Occasionally you'll want to add extra compile or link flags when testing. You can do this as follows:
$ lit -sv --param=compile_flags='-Wcustom-warning'
$ lit -sv --param=link_flags='-L/custom/library/path'
Some other common examples include:
# Specify a custom compiler.
$ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
# Enable warnings in the test suite
$ lit -sv --param=enable_warnings=true test/std
# Use UBSAN when running the tests.
$ lit -sv --param=use_sanitizer=Undefined
LIT Options
:program:`lit` [options...] [filenames...]
Command Line Options
To use these options you pass them on the LIT command line as --param NAME or --param NAME=VALUE. Some options have default values specified during CMake's configuration. Passing the option on the command line will override the default.
Environment Variables
Benchmarks
Libc++ contains benchmark tests separately from the test of the test suite. The benchmarks are written using the Google Benchmark library, a copy of which is stored in the libc++ repository.
For more information about using the Google Benchmark library see the official documentation.
Building Benchmarks
The benchmark tests are not built by default. The benchmarks can be built using
the cxx-benchmarks target.
An example build would look like:
$ cd build
$ cmake [options] <path to libcxx sources>
$ make cxx-benchmarks
This will build all of the benchmarks under <libcxx-src>/benchmarks to be
built against the just-built libc++. The compiled tests are output into
build/benchmarks.
The benchmarks can also be built against the platforms native standard library
using the -DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON CMake option. This
is useful for comparing the performance of libc++ to other standard libraries.
The compiled benchmarks are named <test>.libcxx.out if they test libc++ and
<test>.native.out otherwise.
Also See:
Running Benchmarks
The benchmarks must be run manually by the user. Currently there is no way to run them as part of the build.
For example:
$ cd build/benchmarks
$ make cxx-benchmarks
$ ./algorithms.libcxx.out # Runs all the benchmarks
$ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
For more information about running benchmarks see Google Benchmark.