Re: PoC: Allow parallel 'make test' (was: Re: PATCHv2: Allow parallel make)

["Daniel Shahaf" <d.s@xxxxxxxxxxxxxxxxxx>]

Philippe Altherr wrote on Thu, 24 Sep 2026 00:04 +00:00:
> I didn't want to install flock, so I did as suggested and implemented a
> solution based on 'zsystem flock'. I have no idea whether what I did makes
> any sense.

It seems to function correctly in isolation:

    % : >> ${L::=/tmp/lockfile}
    % with_stdout_mutex() { ( zsystem flock "${L}"; "$@" ) }
    % time ( with_stdout_mutex sleep 1 & with_stdout_mutex sleep 1 & wait; )
    ( with_stdout_mutex sleep 1 & with_stdout_mutex sleep 1 & wait; )  0.00s user 0.00s system 0% cpu 2.007 total
    %

Note it took 2 seconds overall.  (The converse case — two parallel
invocations on different files — takes 1 second, likewise as expected.)

> With high numbers of jobs, like 128, the output looks somewhat
> garbled. In particular there is a big block of seemingly bogus text at the
> very beginning:
>
> runtests.zsh: Running tests (via zargs -P128)...
> 7;file://maloja/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-V11db_gdbm.ztst.unique/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-Y04regexargs.ztst.unique7;file://maloja/Users/paltherr/work/zsh-dev/Test/workdir-for-hmalojap44320.parent-tmp/workdir-for-X04zlehighlight.ztst.unique7;file://7;file://7;file://7;file:///Users/paltherr/work/zsh-dev/[…
> more of the same …]

I see this too under «make test TESTNUM=B», but without the "file://7" part; I see just the hostname and directory.

After bisection + code reading + educated guesses, my theory is that that output is from:

cd [in runtests.zsh] -> bin_cd() -> cd_new_pwd() -> zleentry -> ... -> zle_main_entry() -> notify_pwd() -> write_loop(SHTTY, "\033]7;file://", 11);

That zleentry() call is conditional on «zle_load_state == 1».  My build scripts set link=static for zsh/zle in config.modules.

>

Separately: We might be able to squeeze further speedup if we can
arrange for the slowest ztst files to run in parallel to each other, and
at the start of the overall run: this would prevent a situation where
all but one ztst file have finished, but the remaining one is a slowish
one that delays the overall results.  We might be able to approximate
that by sorting the ztst files by size descending:

[[[
--- Test/Makefile.orig
+++ Test/Makefile
@@ -192,8 +192,7 @@
 	if test -z "$$ZTST_handler"; then \
 	  ZTST_handler=runtests.zsh; \
 	fi; \
-	if ZTST_testlist="`for f in $(sdir)/$(TESTNUM)*.ztst; \
-           do echo $$f; done`" \
+	if ZTST_testlist="`ls -S -- $(sdir)/$(TESTNUM)*.ztst`" \
 	 ZTST_srcdir="$(sdir)" \
 	 ZTST_exe=$(dir_top)/Src/zsh \
 	 $(dir_top)/Src/zsh +Z -f $(sdir)/$$ZTST_handler; then \
]]]

>

Excuse terseness.

Cheers,

Daniel