Re: Substitution ${...///} slows down when certain UTF character occurs

[Peter Stephenson <p.stephenson@xxxxxxxxxxx>]

On Wed, 30 Sep 2015 14:19:37 -0700
Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
> On Sep 30,  3:04pm, Peter Stephenson wrote:
> } In some complex git completion --- reproducible only in one set up ---
> } the heap allocation is apparently going out of bounds.  Yet I've checked
> } and nothing seems able to happen that can affect this.  If I add
> } debugging, the same pointer is reported as returned from zhalloc() that
> } is later claimed to be out of bounds.  Yet with more debugging there's
> } no popheap() or switch_heap() happening, as by eye there can't be.  I'm
> } a bit stuck; how can a piece of memory returned by zhalloc() suddenly go
> } invalid?
> } 
> } valgrind agrees with gdb but doesn't give any more help.
> 
> Did you --enable-zsh-heap-debug and --enable-zsh-valgrind ?  I always
> forget at least the latter of those ...

I don't have the full valgrind library framework, just the executable,
on the system where this is happening.  (This is 64-bit and I haven't
seen it with a 32-bit system I have more control over in the same repo.)
But I tried heap debug and the last bit of output doesn't look obviously
suspicious --- I don't see a sign it's allocating masses of memory, for
example. Indeed, the last thing it did was a push, not a pop.

> } Maybe we're just running out of memory after all?
> 
> That seems really unlikely, unless the explicit zalloc version shows a
> similar issue.

Yes, zhalloc() gets memory from zalloc() which explicitly tests, and
there's no obvious way you could get a pointer of the sort I've got.

> But I suppose if the heap is never popped it might be
> possible to run out of memory on a deep recursive glob.  Perhaps a
> call to freeheap() is needed somewhere prior to the popheap()?  Or
> maybe there *is* a freeheap() happening where it isn't any longer safe
> to do so -- did you check for that?

Yes, I looked for popping, freeing and swapping of heaps and there
doesn't appear to be any in the immediate vicint.

> Even if out of memory, I wouldn't expect an existing pointer to go bad.
> You'd just fail to get a new one.  And heaps are mmap()d if possible so
> in that case you'd have to be out of address space, not just out of
> memory?

I tried not using mmap and the symptom is the same.

I get the feeling there's something here that simply switching back to
zalloc() is likely to obscure rather than fix.

pws