Zsh Mailing List Archive
Messages sorted by:
Re: Multi-core loops
- X-seq: zsh-users 14593
- From: Mikael Magnusson <mikachu@xxxxxxxxx>
- To: "Nadav Har'El" <nyh@xxxxxxxxxxxxxxxxxxx>
- Subject: Re: Multi-core loops
- Date: Sun, 22 Nov 2009 23:40:13 +0100
- Cc: Kazuo Teramoto <kaz.rag@xxxxxxxxx>, zsh-users@xxxxxxx
- Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=XFs6NLSs29H6730F66oWBH8CPJgV8EG9Tj16YEHOv8k=; b=HD87NCZEGnEfLOpmQT3l+BZNYSASsJ5H4Mz7XSB7noYh526VKVUJqc2fS01gEXkJi6 ZCQWubba1Ja+rxndm61Y3RzstrNxgg1nCSzhKGNjOXiF0t3EvdHLjvFEAAySUvuP3FkI 2FsDjkK1puDzMtPV0HN6R510zQv92JrtMNwfU=
- Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=xkYbrwMdq0mAAgCgZu/kc8P9dlytYECi7yATUzBvQZ5IS5RS9xJ1BCwvTwYquxjrOB LvZR1Pmh2lbADwacZV3PbFZfV5k+T5Ywd+JOa2txbJDUOk5NIfd/6392G2xFonU1QTPO dAJFnPEPhVUTjNqTz8DvwJr0m0C+xCEOg8Pd0=
- In-reply-to: <20091122221655.GA28699@xxxxxxxxxxxxxxxxxxxxxxxxxx>
- List-help: <mailto:firstname.lastname@example.org>
- List-id: Zsh Users List <zsh-users.zsh.org>
- List-post: <mailto:email@example.com>
- Mailing-list: contact zsh-users-help@xxxxxxx; run by ezmlm
- References: <20090628103129.GA15270@xxxxxxxxxxxxxxxxxxxxxxxxxx> <5992e17f0911221305j53522ed6j55d8b1308fda2214@xxxxxxxxxxxxxx> <20091122221655.GA28699@xxxxxxxxxxxxxxxxxxxxxxxxxx>
2009/11/22 Nadav Har'El <nyh@xxxxxxxxxxxxxxxxxxx>:
> On Sun, Nov 22, 2009, Kazuo Teramoto wrote about "Re: Multi-core loops":
>> On Sun, Jun 28, 2009 at 8:31 AM, Nadav Har'El <nyh@xxxxxxxxxxxxxxxxxxx> wrote:
>> > Zsh, like all shells, lets you easily do something many times in a loop.
>> > But when "dosomething" is CPU intensive, this is *not* what you'd want to
>> > do on a multi-core (multi CPU) machine, which have more-or-less become
>> > standard nowadays...
>> > Such a loop would only use one of the CPUs, and leave the other(s) unused.
>> > Instead, you'll want to keep all CPUs busy all the time, running M (=number
>> > of CPUs) processes at the same time.
>> Any update on this?
>> I'm searching for a solution. Perhaps this can't be done as a built-in
>> syntax but what about a more complex solution. I'm a noob (and with
>> the numbers of features of zsh, I'm gonna be a noob forever), and cant
>> find a small, beautiful, zsh-is-so-cool-ish solution for it and dont
>> know how to effective implement it e.g., without using python, I like
>> some zsh only solution.
> Unfortunately, no.
> I am running into this need very often - be it a small script to convert a
> bunch of media files, or a script to download a bunch of web pages, and so
> on, and always need to come up with some ugly half-working solution.
> I am still really surprised that no shell (that I know) comes with a
> convenient built-in syntax to do such loops.
> At one point I decided to go ahead an modify zsh myself. After some
> deliberation with myself, I came up with the following syntax:
> for i in 1 2 3
> parallel 2 do
> echo $i
> sleep 3
> I.e., one adds the keyword "parallel" and the number of concurrent processes
> just before the "do/done" block. This syntax makes it easy to parrelize
> all kinds of for loops (C-like, csh-like, bourne-shell-like, etc.) in
> one syntax.
> What this should have done is to run the do/done block in the background
> (like with a &), and additionally block while 2 of these are already running,
> waiting until one of them stopped (we know when one stops because the shell
> has a SIGCHLD interrupt handler).
> Unfortunately, I found understanding the zsh parser much harder than I
> had originally anticipated. I managed to add the "parallel" syntax, but
> was not able (in the several hours I investing in trying to understand)
> how to generate the correct "instructions" (of the zsh virtual machine)
> to put the do/done block in the background, for example. All the tricks
> I tried with WCB_LIST, WCB_SUBLIST, WCB_PIPE, set_list_code, WCB_END and
> other strange things I tried, stopped short of actually working.
> And even if I had managed to pull that off, I still had some more missing
> pieces, like keeping a list of process ids of these backgrounded processes,
> recognizing (in the interrupt handler) when they're gone, and waiting until
> one of them is gone.
>> How people solved this problem any home-made solution for this, any tip?
> By the way, in my original thread I mentioned a post from 10 years ago (!)
> which suggested an eleborate trick to do what I was after:
> Have you tried this?
I came across this script a while ago, it's written in bash and I
haven't tried it, but it claims to run stuff in parallel :). Maybe it
can be zsh-ified to get rid of the warning about commands probably not
working with spaces.
Messages sorted by: