FATE Automated Testing Environment


Table of Contents


1 Introduction

FATE provides a regression test suite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

2 Running FATE

2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the test suite location, pass ‘--samples=<path to the samples>’ to configure or set the SAMPLES Make variable or the LIBAV_SAMPLES environment variable to a suitable value.

To use a custom wrapper to run the test, pass ‘--target-exec’ to configure or set the TARGET_EXEC Make variable.

The dataset is available through rsync, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
# make fate-rsync SAMPLES=fate-suite

3 Manual Run

FATE regression test can be run through make. Specific Makefile targets and Makefile variables are available:

3.1 FATE Makefile targets

fate-list

List all fate/regression test targets.

fate-rsync

Shortcut to download the fate test samples to the specified test suite location.

fate

Run the FATE test suite (requires the fate-suite dataset).

3.2 FATE Makefile variables

V

Verbosity level, can be set to 0, 1 or 2.

0

show just the test arguments

1

show just the command used in the test

2

show everything

SAMPLES

Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.

THREADS

Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.

THREAD_TYPE

Specify which threading strategy test, either slice or frame, by default slice+frame

CPUFLAGS

Specify a mask to be applied to autodetected CPU flags.

TARGET_EXEC

Specify or override the wrapper used to run the tests.

GEN

Set to 1 to generate the missing or mismatched references.

HWACCEL

Specify which hardware acceleration to use while running regression tests, by default none is used.

    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate

4 Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, tests/fate.sh is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to http://fate.libav.org/ or directly examined locally.

4.1 Testing Profiles

The configuration file passed to fate.sh is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
#branch=release/10                      # the branch to test
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description
build_only=     # set to "yes" for a compile-only instance that skips tests

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'

4.2 Special Instances

The TARGET_EXEC option provides a way to run FATE wrapped in valgrind, qemu-user or wine or on remote targets through ssh.

4.3 Submitting Reports

In order to send reports you need to create an ssh key and send it to root@libav.org.