Rust Nightly, Travis CI and Code Coverage

How to setup test a Rust Nightly project with a workspace in Travis CI and collect coverage information with kcov and Codecov

Written by Baptiste GelezNovember 1, 2018

I just spent the evening improving Plume's continuous integration, and it wasn't as easy as I could have imagined.

A git history, full of funny commits

That's why I decided to write a small article to summarize how to setup continuous integration for a Rust project, with Travis CI (and how to get nice coverage reports with Codecov too).

This solution works for Rust project using workspaces too.

Basic setup

Travis CI uses .travis.yml as its configuration file. The simplest config for a Rust project is:

language: rust
cache: cargo

Actually the second line is not even required, but it will make future build faster by caching artifacts.

Rust Version

Travis CI has a predefined preset for Rust projects, that install all the required tools with rustup for you. In their doc they explain how to choose the Rust version(s) to install. But what is not obvious is that you can specify any valid toolchain here, it will be installed.

For instance, Plume needs a very specific Nightly version to be installed. Before, we were just telling Travis to install Rust nightly, and it was installing the latest Nightly build. That was not a real problem, because Plume has a rust-toolchain file, and when the build was started with the wrong Rust version, the correct one was installed, so nothing ever broke. The only issue was that we were downloading Rust twice (and Rust is not a very light thing). To avoid that, you can directly specify the toolchain you want to use:

rust:
  - nightly-2018-07-17

Building and testing in parallel

Then I wanted to both build Plume and run the tests at the same time. You can achieve that in Travis CI by running multiples "jobs" in the same "stage". Here is a simplified version of Plume's config to understand how to do that:

jobs:
  include:
    - stage: test and build
      script: cargo test  --all
    - stage: test and build
      script: cargo build

So all you have to do, is to put two (or more) jobs descriptions in the jobs.include list, with the same value for stage, and they will run in parallel.

If you want to learn more about jobs and stages, I advise you to check out the official documentation.

Code Coverage with kcov and Codecov

That part was the most difficult to figure out. The solution I came with uses kcov and Codecov.

First of all, you need to install kcov. But because the distribution packages are totally outdated, you'll have to compile it from source (it doesn't take too much time fortunately). Then you'll have to run it for all the test executables in your workspace. These executables are generated by cargo when running cargo test --all, and can be found in target/debug. Finally, you'll have to upload the result to Codecov (they are providing a great Bash script to do that easily). Here is the script to achieve that:

sudo: true
dist: trusty
addons:
  apt:
    packages:
      - libcurl4-openssl-dev
      - libelf-dev
      - libdw-dev
      - cmake
      - gcc
      - binutils-dev
      - zlib1g-dev
      - libiberty-dev
jobs:
  include:
    - stage: test and build
      env:
        - RUSTFLAGS='-C link-dead-code'
      script: cargo test --all
      after_success:
       - |
           wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
           tar xzf master.tar.gz &&
           cd kcov-master &&
           mkdir build &&
           cd build &&
           cmake .. &&
           make &&
           sudo make install &&
           cd ../.. &&
           rm -rf kcov-master &&
           for crate in WORKSPACE_MEMBERS lib; do for file in target/debug/$crate-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; kcov
--exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done; done &&
           bash <(curl -s https://codecov.io/bash) &&
           echo "Uploaded code coverage"

The after_success part is where all the steps I described above lies. But there are few other thing that were added to our config. First of all, we install some dependencies required by kcov, since we are building it from source (sudo, dist and addons.apt). Then we are compiling with a special flag: link-dead-code. It tells Rust to keep dead code in our executable. This way, even functions that are not called during our tests will be in the coverage report, which will be more accurate.

The most complex line of the script is probably the for loop near the end. It actually iterates over a list of workspace members (you should replace WORSPACE_MEMBERS with the name of the crates in your workspace) and lib, tries to find a test executable that start with that name and run kcov for it. Test executable are either named after the name of the crate (for unit tests) or lib-something (for integration tests).

The last step is to add the Codecov token to your Travis CI environment so that you can upload reports. You can find your token at https://codecov.io/gh/OWNER/REPO/settings, and add it in Travis at https://travis-ci.org/OWNER/REPO/settings (the variable should be named CODECOV_TOKEN).

That's it! You should now be able to build, test and get coverage reports for any Rust project (even with workspaces). I hope this little article helped you, and if you found an error please tell me.

This article is under the CC-0 license.

Baptiste Gelez

I'm the main Plume developer.

Follow

Login or use your Fediverse account to interact with this article