PoC: Allow parallel 'make test' (was: Re: PATCHv2: Allow parallel make)

["Daniel Shahaf" <d.s@xxxxxxxxxxxxxxxxxx>]

Philippe Altherr wrote on Sat, 19 Sep 2026 15:45 +00:00:
> Unfortunately, as expected, the following stayed the same:
>
> make     check  39.63s user 51.56s system 52% cpu 2:53.97 total
> make -j8 check  40.37s user 52.18s system 51% cpu 2:57.99 total
>
> For me, being able to run tests in parallel would be a big improvement.
> Doing so is probably non-trivial.

It turns out that the ztst files hardcode various path-related assumptions:

The tests all use the same cwd, and ZTST_cleanup removes *.tmp... which
could remove the still-in-use *.tmp files of other *.ztst files being run in
parallel.  (B01cd.ztst documents the use of *.tmp filenames.)

To sidestep this, I've gone ahead and made each ztst file have its own
cwd, but that runs into tests assuming that ../config.modules and
../Src/zsh exist.

I didn't want to make the per-ztst-file workdirs immediate children of
the build tree root, so I went ahead and mocked up these two paths.  See
the second TODO in the attachment.

> Just avoiding garbled output might already be non-trivial.

flock(1) with a well-known filename seems to do the trick.

It does require buffering the data using tempfiles.  (The tempfiles are created
in the build tree.  Given the context, I assume the build tree will be on a ramdisk.)

> However, a version that runs in parallel with
> garbled output but a correct summary would already be a great improvement.

Proof of concept attached.

Known issues:

- No UI for opting in to / out of parallel mode (see the first TODO)

- Using per-ztst-file workdirs *only* in parallel mode looks weird.
  The «if (( MAX_CPUS > 1 ))» in run_one_file() should become «if true», right?

- One print hasn't been guarded by the mutex (see the fourth TODO)

- Occasional failures in V08zpty.ztst.  Not investigated.

- Only tested with out-of-tree builds, not with in-tree builds.

Observations:

- Inflates runtests.zsh from 66 lines to 215 lines.

Timings:

make -C zsh test  63.19s user 14.19s system 55% cpu 2:18.63 total
make -C zsh test  67.71s user 20.41s system 161% cpu 54.494 total

If someone wants to pick this up and finish it, please do.

Cheers,

Daniel


> Usually, when I run all tests, it's just to check whether all tests still
> pass and, if not, find out which ones have problems.

diff --git Test/runtests.zsh Test/runtests.zsh
index ea862d2b3..4a27ded2d 100644
--- Test/runtests.zsh
+++ Test/runtests.zsh
@@ -2,6 +2,82 @@
 
 emulate zsh
 
+
+### Define helpers and constants
+
+# Use a suffix other than ".tmp" in order to prevent premature removal by ztst.zsh:ZTST_cleanup
+ZTST_exe=${ZTST_exe:P}
+export ZTST_root_of_build_tree=${ZTST_exe:h:h}
+export ZTST_root_of_source_tree=${ZTST_srcdir:h}
+readonly RUNTESTS_WORKDIR=${ZTST_root_of_build_tree}/Test/workdir-for-h${HOST}p$$.parent-tmp/
+mkdir -- ${RUNTESTS_WORKDIR} || {
+  print -ru2 - "runtests: error: failed to create unique workdir"
+  exit 1
+}
+
+# How many test files to run in parallel.
+#
+# Must be a positive integer.
+#
+# When >2, flock(1) must be available.
+#
+# TODO: make this configurable (and document)
+integer MAX_CPUS=1
+MAX_CPUS=$(
+  DEFAULT=4
+  if type python3 > /dev/null; then
+    # Return os.process_cpu_count(), falling back to os.cpu_count() and $DEFAULT if os.process_cpu_count() is unavailable or returns None
+    python3 -c "import os; print(  getattr(os, 'process_cpu_count', os.cpu_count)() or $DEFAULT  )"
+  else
+    print $DEFAULT
+  fi
+)
+#MAX_CPUS=1  # quick toggle for testing
+
+if (( MAX_CPUS > 1 )); then
+  # Load zargs
+  () {
+    local fpath
+    fpath+=( ${ZTST_root_of_source_tree}/Functions/Misc/zargs(:h) )
+    autoload -U +X zargs
+  }
+
+  # Make it possible for run_one_file() to cd into per-ztst-file directories,
+  # by arranging for $ZTST_testdir/../foo to exist for those values of foo that
+  # tests currently use.
+  #
+  # TODO: In all ztst files, replace "$ZTST_testdir/.." [literal dot-dot,
+  #       not an ellipsis] with "$ZTST_root_of_build_tree", then remove the
+  #       following block.
+  if true; then
+    mkdir -- "${RUNTESTS_WORKDIR}"/Src
+    ln -s "${ZTST_exe}" "${RUNTESTS_WORKDIR}/Src/zsh"
+    ln -s "${ZTST_root_of_build_tree}/config.modules" "${RUNTESTS_WORKDIR}/config.modules"
+  fi
+fi
+
+# Define with_stdout_mutex() appropriately.
+#
+# with_stdout_mutex(): A precommand that runs its argv under the stdout mutex, if necessary.
+if (( MAX_CPUS > 1 )); then
+  type flock >/dev/null || {
+    print -ru2 - "$0: error: parallel mode requires flock(1)"
+    exit 1
+  }
+  : > ${STDOUT_SEQUENCER_FILE::="${RUNTESTS_WORKDIR}/.stdout-mutex"}
+  with_stdout_mutex() {
+    # TODO: use 'zsystem flock' (and fall back to the MAX_CPUS=1 codepath when 'zsystem flock' is unavailable)
+    flock "${STDOUT_SEQUENCER_FILE}" "$@"
+  }
+else
+  with_stdout_mutex() {
+    "$@"
+  }
+fi
+
+
+### Main logic
+
 # Run all specified tests, keeping count of which succeeded.
 # The reason for this extra layer above the test script is to
 # protect from catastrophic failure of an individual test.
@@ -19,14 +95,66 @@ typeset -a ffiles res
 [[ $TERM == *color* || "$( tput colors 2> /dev/null )" == <8-> ]] &&
 typeset -x ZTST_failcolour=red
 
-for file in "${(f)ZTST_testlist}"; do
-  file=${file#./}
-  $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh $file
-  retval=$?
+# Set $REPLY to a filename component that ---
+#
+# * depends on the basename of $1;
+# * can be appended to ${RUNTESTS_WORKDIR} without ending up in another directory
+#   (i.e., does not have slashes or dot-dot components).
+make_unique_suffix_for_file() {
+  REPLY=for-${1:t}.unique
+}
+
+# Run the ztst file $1.  When done, save its exit status in a file, and emit
+# its stdout and stderr with synchronization.
+run_one_file() {
+  local file=${1#./}
+
+  make_unique_suffix_for_file "${file}"
+  local unique_suffix=$REPLY
+
+  if (( MAX_CPUS > 1 )); then
+    local workdir_for_this_ztst=${RUNTESTS_WORKDIR}/workdir-${unique_suffix}
+    mkdir -- "${workdir_for_this_ztst}"
+  fi
+
+  (
+    if (( $+workdir_for_this_ztst )); then
+      cd -- "${workdir_for_this_ztst}";
+    fi || {
+      print -ru2 - "$0: failed cd to ${(qqqq)workdir_for_this_ztst}"
+      exit 1
+    }
+    $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh $file
+    print -r - > ${RUNTESTS_WORKDIR}/ztst-status-${unique_suffix} "$?"
+  ) \
+    1>${RUNTESTS_WORKDIR}/ztst-output-${unique_suffix} \
+    2>${RUNTESTS_WORKDIR}/ztst-errput-${unique_suffix}
+
+  # Print immediately, otherwise 'make test' would have no stdout at all until everything finished.
+  #
+  # Note: Can't add "--" before "cat", as on Linux at least that would interpret "--" as a command name to execvp().
+  with_stdout_mutex \
+    cat -- \
+      ${RUNTESTS_WORKDIR}/ztst-output-${unique_suffix} \
+      ${RUNTESTS_WORKDIR}/ztst-errput-${unique_suffix} \
+      =(print)
+}
+
+# Update counters for the ztst file $1.
+#
+# Precondition: The exit status file.
+tally_one_file() {
+  local file=${1#./}
+
+  make_unique_suffix_for_file "${file}"
+  local unique_suffix=$REPLY
+
+  local retval=$(< ${RUNTESTS_WORKDIR}/ztst-status-${unique_suffix})
   if (( $retval == 2 )); then
     (( skipped++ ))
   elif (( $retval )); then
     (( failure++ ))
+    # TODO: Move to run_one_file() and use with_stdout_mutex()
     (( $retval > 128 )) && {
       [[ -n $ZTST_failcolour ]] && print -rnP "%F{$ZTST_failcolour}"
       print -r - "$file: test failed: SIG$signals[$retval - 127]."
@@ -36,8 +164,28 @@ for file in "${(f)ZTST_testlist}"; do
   else
     (( success++ ))
   fi
-  print
-done
+}
+
+if (( MAX_CPUS > 1 )); then
+  print "${0:t}: Running tests (via zargs -P${MAX_CPUS})..."
+  zargs \
+    --no-run-if-empty \
+    -n1 \
+    --max-procs "${MAX_CPUS}" \
+    "${(f)ZTST_testlist}" \
+    -- run_one_file
+
+  for file in "${(f)ZTST_testlist}"; do
+    # This updates various counters, so it can't run in zargs (which uses subshells).
+    tally_one_file "$file"
+  done
+else
+  print "${0:t}: Running tests (sequentially)..."
+  for file in "${(f)ZTST_testlist}"; do
+    run_one_file "$file"
+    tally_one_file "$file"
+  done
+fi
 
 res=(
   "$success successful test script${${success:#1}:+s}"
@@ -63,4 +211,5 @@ print -rl - '' "failing test script${${#ffiles:#1}:+s}:" ${(@)ffiles/#/  }
 print -rl - $rule
 (( $#col )) && print -rnP - %f
 
+rm -rf -- ${RUNTESTS_WORKDIR}
 return $(( failure ? 1 : 0 ))