Re: PoC: Allow parallel 'make test' (was: Re: PATCHv2: Allow parallel make)

[Philippe Altherr <philippe.altherr@xxxxxxxxx>]

Awesome, that looks very promising. The whole test suite runs up to about 5 times faster:

-   1 job  -> make check  40.83s user 53.52s system  53% cpu 2:55.26  total
-   8 jobs -> make check  44.52s user 61.41s system 133% cpu 1:19.36  total
-  16 jobs -> make check  48.62s user 67.03s system 201% cpu   57.387 total
-  32 jobs -> make check  49.53s user 68.33s system 268% cpu   43.830 total
-  64 jobs -> make check  50.45s user 70.69s system 315% cpu   38.367 total
- 128 jobs -> make check  52.15s user 73.38s system 357% cpu   35.089 total

The number of jobs is MAX_CPUS.

With 128 jobs, there are 6 tests that often or always fail:

- A08time.ztst      --x-x-x-xx  --
- C03traps.ztst     xxxxxxxxxx  --
- V06parameter.ztst xxxxxxxxxx  xx
- V08zpty.ztst      x-xxx--x--  --
- V11db_gdbm.ztst   xxxxxxxxxx  xx
- V14system.ztst    -x-x--xxxx  --

The 'x's denote failures. The last two columns are runs with only 8 jobs. No other test failed but one more test than usual (D03procsubst.ztst) was skipped and maybe also some additional test cases. I haven't looked into why the tests failed or were skipped.

I build and run tests in-tree (I run 'make' and 'make check' in the zsh root directory). To make your patch work, I had to apply the changes in the attached patch.

I didn't want to install flock, so I did as suggested and implemented a solution based on 'zsystem flock'. I have no idea whether what I did makes any sense. With high numbers of jobs, like 128, the output looks somewhat garbled. In particular there is a big block of seemingly bogus text at the very beginning:

runtests.zsh: Running tests (via zargs -P128)...

7;file://maloja/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-V11db_gdbm.ztst.unique/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-Y04regexargs.ztst.unique7;file://maloja/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-X04zlehighlight.ztst.unique7;file://7;file://7;file://7;file:///Users/paltherr/work/zsh-dev/[… more of the same …]

Philippe

diff --git a/Test/runtests.zsh b/Test/runtests.zsh
index b5c222f08..721b70ef0 100644
--- a/Test/runtests.zsh
+++ b/Test/runtests.zsh
@@ -7,6 +7,8 @@ emulate zsh
 
 # Use a suffix other than ".tmp" in order to prevent premature removal by ztst.zsh:ZTST_cleanup
 ZTST_exe=${ZTST_exe:P}
+ZTST_srcdir=$ZTST_srcdir:P
+ZTST_testdir=$ZTST_srcdir:h/Test
 export ZTST_root_of_build_tree=${ZTST_exe:h:h}
 export ZTST_root_of_source_tree=${ZTST_srcdir:h}
 readonly RUNTESTS_WORKDIR=${ZTST_root_of_build_tree}/Test/workdir-for-h${HOST}p$$.parent-tmp/
@@ -60,14 +62,11 @@ fi
 #
 # with_stdout_mutex(): A precommand that runs its argv under the stdout mutex, if necessary.
 if (( MAX_CPUS > 1 )); then
-  type flock >/dev/null || {
-    print -ru2 - "$0: error: parallel mode requires flock(1)"
-    exit 1
-  }
   : > ${STDOUT_SEQUENCER_FILE::="${RUNTESTS_WORKDIR}/.stdout-mutex"}
+  zmodload zsh/system
   with_stdout_mutex() {
     # TODO: use 'zsystem flock' (and fall back to the MAX_CPUS=1 codepath when 'zsystem flock' is unavailable)
-    flock "${STDOUT_SEQUENCER_FILE}" "$@"
+    ( zsystem flock "${STDOUT_SEQUENCER_FILE}"; "$@" )
   }
 else
   with_stdout_mutex() {
@@ -124,7 +123,7 @@ run_one_file() {
       print -ru2 - "$0: failed cd to ${(qqqq)workdir_for_this_ztst}"
       exit 1
     }
-    $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh $file
+    $ZTST_exe +Z -f $ZTST_testdir/ztst.zsh $ZTST_testdir/$file
     print -r - > ${RUNTESTS_WORKDIR}/ztst-status-${unique_suffix} "$?"
   ) \
     1>${RUNTESTS_WORKDIR}/ztst-output-${unique_suffix} \