Thomas Gazagnaire

I use Alcotest and dune-release for almost every OCaml package I release: the first runs the tests, and the second publishes the result on opam. Both had collected issues and pull requests over the years (the oldest issue I closed this week dates from 2017), so I spent the past week going through them. Alcotest 1.10.0 and dune-release 2.3.0 are the result. For me, the most useful changes are that Alcotest can now write JUnit reports without any change to your tests, and that tests can carry tags. A tool that reads a folder of reports can tell a flaky test from a broken one, and a tag then puts the flaky test aside until it is fixed.

dune-release 2.3.0 is on opam, and Alcotest 1.10.0 is about to be. The opam-repository CI shows failures in packages that depend on it, which I want to triage properly first. Testing it before then is very welcome, with a pin:

$ opam update
$ opam install dune-release
$ opam pin add -n https://github.com/mirage/alcotest.git#1.10.0
$ opam install alcotest

Nested tests

Alcotest suites used to have exactly two levels. suite, test and group now nest to any depth, and the existing test_case and tuples keep working next to them. A group's name is joined to its suite's with a dot, and a regular expression on that name selects what runs:

open Alcotest

let parse () = check int "parse" 42 (int_of_string "42")
let hex () = check int "hex" 255 (int_of_string "0xff")
let big () = check int "big" 1_000_000 (int_of_string "1000000")
let neg () = check int "neg" (-1) (int_of_string "-1")

let () =
  run "parser"
    [
      suite "int"
        [
          test "parse" parse;
          group "edge" [ test "big" big; test "neg" neg ];
          test "hex" hex;
        ];
    ]

$ ./test_parser.exe test '^int\.edge'
Testing `parser'.
This run has ID `M0FK86F8'.

  [SKIP]        int               0   parse.
  [OK]          int.edge          0   big.
  [OK]          int.edge          1   neg.
  [SKIP]        int               1   hex.

Full test results in `logs/M0FK86F8'.
Test Successful in 0.001s. 2 tests run.

Nesting was a long-standing request. Craig Ferguson opened the issue in 2020, and followed it with an RFC the next year: abstract test types nested to any depth, with tags and source positions. In the discussion, Etienne Millon called the two-level limit Alcotest's biggest, and Martin Jambon argued for a flat list with dotted names instead. Yawar Amin later proposed a suite function to make tests shorter to write. The RFC stalled on its abstract types, which would have broken every existing test suite. This release takes group from Craig's RFC, suite and test from Yawar's proposal and the dotted names from Martin, and adds them next to the existing types rather than in place of them. The RFC's tags and source positions are in this release too, in the sections below. Thanks to all of them.

Failures that point at your test

Every failure now starts with the file and line of the test, including uncaught exceptions and failures inside helpers. Here is a test with a bug (-0 does not survive a round trip):

let round_trip s =
  Alcotest.(check string) "round trip" s (string_of_int (int_of_string s))

let decimal () = List.iter round_trip [ "1"; "-1"; "-0" ]

let () =
  Alcotest.run "parser"
    [ ("int", [ Alcotest.test_case "decimal" `Quick decimal ]) ]

# 1.9.1
File "list.ml", line 114, character 12:
FAIL round trip

# 1.10.0
File "test_parser.ml", line 8, character 16:
FAIL round trip

(The idea and some of the code come from Daniel Bünzli's B0_testing. A test case written as a plain tuple can still lack a location, and wasm and js_of_ocaml builds do not record backtraces, so they print none: this issue tracks a fix.)

JUnit reports without touching your tests

GitLab, CircleCI and Jenkins all read test results as JUnit XML. Other ecosystems differ in how much work this takes. pytest, Deno, Bun and Node.js have an option. Go needs gotestsum, Jest needs jest-junit, and Rust needs nextest (cargo test only offers it on nightly).

In OCaml, junit_alcotest by Louis Roché has written these reports for Alcotest since 2017. Each test executable calls its run_and_report in place of Alcotest.run, then writes the report to a path of its choice. As a layer on top, it only sees what Alcotest's public interface gives it, which has no source locations. Inside Alcotest, the report can take each test's file and line from the same place as the failure messages above, and it can still be written when a test ends the process early.

Alcotest now has --junit, and an environment variable, ALCOTEST_JUNIT, that goes through dune. Existing tests and dune files need no change to export JUnit reports. Each run adds its report to _build/_tests/junit:

$ ALCOTEST_JUNIT=1 dune test --force
$ ls _build/_tests/junit
parser-20261109-101751-ZGXMDTK7.xml
store-20261109-101749-K6LO6TC3.xml

# .gitlab-ci.yml
test:
  variables:
    ALCOTEST_JUNIT: "1"
  script:
    - dune test
  artifacts:
    when: always
    reports:
      junit: _build/_tests/junit/*.xml

Each failure carries its file and line, and the report survives a test that ends the process. Here, write calls exit, as a library that gives up might, and the report still names it:

open Alcotest

let read () = check string "read" "a" "a"
let write () = exit 3
let close () = ()

let () =
  run "store"
    [
      suite "kv"
        [ test "read" read; test "write" write; test "close" close ];
    ]

<testcase name="read" classname="store.kv" time="0.000"/>
<testcase name="write" classname="store.kv" time="0.000">
<error message="[crash] The process stopped while
this test was running." type="crash">

The reports are experimental, which is why they are off by default for now. I would like to hear from people who use them before deciding whether they should be on by default and which fields they are missing.

Flaky or broken?

Keeping one report per run gives a history that other tools can read. To try it, I wrote a toy key-value store with two kinds of failure. cache.lookup loses a race about one run in six (faked with Random, since a real race does not fail on demand), and after twenty runs I committed a refactoring of compact that forgets to drop deleted keys:

-  Hashtbl.filter_map_inplace (fun _ v -> Option.map Option.some v) t;
+  Hashtbl.filter_map_inplace (fun _ v -> Some v) t;

flakemap reads the whole history from that folder (I ran the first twenty runs before the refactoring and the last ten after it):

$ export ALCOTEST_JUNIT=1
$ for i in $(seq 30); do dune test --force; done
$ flakemap _build/_tests/junit --html flakemap.html

flakemap marks cache.compact as broken since the twenty-first run, the first after the refactoring, and cache.lookup as flaky, failing in 13% of the runs. Nothing was added to the reports for that: each file is a run, and the test names inside are the rows. flakemap is very new (its repository dates from September 2026) and orders the runs by the files' modification times.

I picked flakemap because it was the first tool I found that draws a history from local JUnit files, and others probably do more, or do it better. junit2html, around since 2016, draws the same grid with --report-matrix, for instance. That is the point of a common format, even if with JUnit each tool seems to change the standard a little. Alcotest gives each failing test a file and a line, as pytest's xunit1 reports do, and GitLab shows that file next to the failure.

Putting a flaky test aside

Once a test is known to be flaky, I would rather skip it in normal runs and keep running it somewhere else. Tests can now carry tags, and a run picks the tags it wants with --tags or ALCOTEST_TAGS (a bit like Go's build tags). A test tagged `Tag "flaky" is skipped unless the run asks for it, so a nightly job can keep an eye on it:

-  suite "cache" [ test "lookup" lookup; test "compact" compact ];
+  suite "cache" [ test ~tags:(`Tag "flaky") "lookup" lookup; test "compact" compact ];

$ dune test
  ...
  [SKIP]        cache          0   lookup.
  [OK]          cache          1   compact.

Test Successful in 0.003s. 4 tests run.

$ ALCOTEST_TAGS=flaky dune test --force
  ...
  [OK]          cache          0   lookup.
  [OK]          cache          1   compact.

Test Successful in 0.001s. 5 tests run.

The JUnit report keeps the skipped test as <skipped/>, and flakemap leaves those runs out of lookup's rate rather than counting them as passes, so a test put aside does not start to look healthy. The built-in `Slow tag works the same way (-q stops those tests), and tags combine, as in Tag.(slow && v "release").

The same tests under wasm

Alcotest now works with wasm_of_ocaml, thanks to Hugo Heuzard and Quentin Hocquet. Add one mode to a dune stanza, and dune test runs the tests both ways (you need wasm_of_ocaml-compiler, Binaryen and Node.js). This is worth doing because int has 31 bits under wasm and 63 on a 64-bit native build, and that is easy to forget:

(test
 (name test_parser)
 (modes exe wasm)
 (libraries alcotest))

open Alcotest

let gib n = n * 1024 * 1024 * 1024
let five_gib () = check string "5 GiB" "5368709120" (string_of_int (gib 5))
let () = run "sizes" [ suite "gib" [ test "five" five_gib ] ]

Upgrading Alcotest

A suite that only uses test_case and tuples should compile unchanged. Alcotest now requires OCaml 4.14 and dune 3.17, and the empty alcotest-js package is gone (depend on js_of_ocaml-compiler instead). speed_level is now an alias of the new tag type, which has more cases than `Quick and `Slow. Code that restates the old type as [ `Quick | `Slow ] stops compiling, and a match that handles only those two cases is no longer exhaustive. The new names test, suite and group can also hide values of your own. The changelog has the rest, including UTF-8 in string failures and new testables for Uchar.t and Iarray.t.

dune-release on Windows

dune-release distrib and dune-release check now work on Windows, building on work that Etienne Millon started in 2021, and the test suite runs there too. publish and opam submit are only tested there with --dry-run, and tar, bzip2 and curl must be in the PATH, so reports from Windows users are very welcome.

The other change you will notice is from Josh Berdine: files marked export-ignore in .gitattributes no longer reach the release archive, as with git archive:

# .gitattributes
dune-workspace export-ignore
.github/** export-ignore

$ dune-release distrib
$ tar tjf _build/hello-0.1.0.tbz
hello-0.1.0/.gitattributes
hello-0.1.0/CHANGES.md
hello-0.1.0/LICENSE.md
hello-0.1.0/README.md
hello-0.1.0/dune-project
hello-0.1.0/hello.opam

dune-release also builds with the opam 2.6 libraries (thanks to Vincent Laporte), publish respects --dev-repo (thanks to Shon Feder), and opam submit always prints the URL of the pull request it opened. Its changelog has the rest.

Thanks to everyone above and to the other contributors and reviewers named in the changelogs, some of whom waited for years. Feedback is welcome on the Alcotest and dune-release issue trackers. Before 1.10.0 reaches opam, I am most interested in Alcotest code that stopped compiling, and in how the JUnit reports fare in your CI. The new features are meant to be backward compatible, unless you write your own engine or rely on one of its internal types. Apart from the cases in the upgrading section above, a test suite that breaks is a bug.