Zsh Mailing List Archive
Messages sorted by: Reverse Date, Date, Thread, Author

Efficient way to map a list of values to multiple processes, then accumulate their output

Hello all!  Thanks in advance for the assistance.

Ultimately, I'd like to have some quasi-implementation of xargs -P in pure zsh -- which maps a __function__ to a list of arguments, and returns the results in an array or associative array.  xargs cannot be used to run shell functions, so this is out.

I think this is a situation that Zsh is not well-suited for, but let's say I have a list of files that I want to perform some processing on, then gather the results when the processing has completed.

Simply backgrounding each process with & and then wait'ing on all jobs to complete should work, in theory.  However, the output from each invocation may be intermixed, even in the simple case of "one line of output per file" the order of completion is non-deterministic and the output may become garbled.

This seems something that would fit nicely with co-processes, but it appears each zsh instance can only have one.

Another idea is to farm the output of each to a temporary file, and have an associative array of e.g. [input]=$(mktemp) as well as [input]=job_id and then grabbing the output as each job completes.

I expect I am not the first person to try this (equivalent to the Python multiprocessing.map) but I figured it was worth asking!

Zach Riggle

Messages sorted by: Reverse Date, Date, Thread, Author