Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm
Precedence: bulk
X-No-Archive: yes
List-Id: Zsh Workers List <zsh-workers.zsh.org>
List-Post: <mailto:zsh-workers@zsh.org>
List-Help: <mailto:zsh-workers-help@zsh.org>
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on f.primenet.com.au
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham
	autolearn_force=no version=3.4.0
From: Kamil Dudka <kdudka@redhat.com>
To: zsh-workers@zsh.org
Subject: infinite recursion when handling the "out of memory" state
Date: Mon, 25 Jan 2016 16:02:50 +0100
Message-ID: <4775934.DA9sGOASTP@kdudka.brq.redhat.com>
User-Agent: KMail/4.14.10 (Linux/4.3.3-300.fc23.x86_64; KDE/4.14.14; x86_64; ; )
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="nextPart2237854.4GAHtHgKed"
Content-Transfer-Encoding: 7Bit
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Seq: zsh-workers 37775

--nextPart2237854.4GAHtHgKed
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"

If zsh is compiled with multibyte support, handling of the "out of memory" 
state does not work well in certain cases -- instead of printing the error 
message and exiting, zsh ends up in an infinite recursion and crashes due to 
stack overflow.

The memory allocation functions in mem.c use zerr() to print the fatal error 
messages.  However, zerr() calls zwarning() and transitively mb_niceformat(), 
which allocates heap memory (and may call zerr() on failure).

I see three options how to prevent the stack overflow in such cases:

1. avoid using zerr() to print "out of memory" error messages

2. implement zerr() such that it does not trigger heap memory allocation
(I am afraid this is not really possible)

3. introduce a flag that would prevent zerr() from recurring into itself

I have attached a reproducer script.  It works reliably if the zsh executable 
is linked statically.  Otherwise it may happen that the dynamic linker is more 
hungry on memory than zsh itself, which would hide the bug.

Originally reported at: https://bugzilla.redhat.com/1300958

Kamil
--nextPart2237854.4GAHtHgKed
Content-Disposition: attachment; filename="oom.zsh"
Content-Transfer-Encoding: 7Bit
Content-Type: application/x-shellscript; name="oom.zsh"

#!/usr/bin/zsh

# attempt to start a zsh instance with 'ulimit -v' set to $1
probe() (
    printf "ulimit -v %d ... " $1
    ulimit -v $1 || return $?
    zsh -fc true
)

# exponentially decrease the 'ulimit -v' value until zsh fails to start
mem_limit=$((2**30))
while test 1 -lt $mem_limit && probe $mem_limit; do
    printf "OK\n"
    mem_limit=$(($mem_limit / 2))
done

# increment the 'ulimit -v' value in a loop and check if a signal is raised
probe_limit=$(($mem_limit * 4))
while test $mem_limit -lt $probe_limit; do
    printf "\r"
    mem_limit=$(($mem_limit + 1))
    probe $mem_limit
    ec=$?
    if test 128 -lt $ec; then
        printf "raised signal %d\n" $(($ec - 128))
    fi
done

--nextPart2237854.4GAHtHgKed--