Zsh Mailing List Archive
Messages sorted by: Reverse Date, Date, Thread, Author

Re: Multi-core loops

On Sun, Jun 28, 2009, Nadav Har'El wrote about "Multi-core loops":
> 	for i in ...
> 	do
> 		dosomething $i
> 	done
> But when "dosomething" is CPU intensive, this is *not* what you'd want to
> do on a multi-core (multi CPU) machine, which have more-or-less become
> standard nowadays...
> Such a loop would only use one of the CPUs, and leave the other(s) unused.
> Instead, you'll want to keep all CPUs busy all the time, running M (=number
> of CPUs) processes at the same time.

Sorry for replying to my own emails, but I just remembered another point.
The parallel loop I propose to add is not only relevant to SMP machines
and CPU-intensive tasks.

It is also important, sometimes even more important, for some types of
non-CPU intensive tasks. For example, consider a loop like this for
fetching the content of a list of URLs:

	cat urllist | while read url
		wget $url

The computer will be idle most of the time, as wget will most of the time
just wait for responses from the network. If I could easily tell the loop
to run 10 wgets at a time, this would (in most cases) boost the performance
of this loop almost 10-fold! And this is true even if you have just one CPU.

Searching the web, I found that Chapter 14 "Throttling Parallel Processes"
of Ron Peters' "Shell Script Pearls" is about a similar use case, and his
solution is a very eleborate one using a variant of the coprocess-based
solution in the thread I mentioned earlier from the zsh list 10 years ago.
So I think there is definitely a need for such a parallel loop feature.

Nadav Har'El                        |       Sunday, Jun 28 2009, 6 Tammuz 5769
nyh@xxxxxxxxxxxxxxxxxxx             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Creativity consists of coming up with
http://nadav.harel.org.il           |many ideas, not just that one great idea.

Messages sorted by: Reverse Date, Date, Thread, Author