Re: PATCH: parse from even deeper in hell

[Peter Stephenson <p.stephenson@xxxxxxxxxxx>]

> > Mikael Magnusson <mikachu@xxxxxxxxx> wrote:
> I actually thought the patch in 34587 had fixed it, turns out I just
> lost my --enable-zsh-debug flag. This patch doesn't fix it either
> though, and I get a different set of errors every time I open a
> terminal now, so that's fun. Which is to say it's all the same
> wordsplit error, but it's printed for a different set of lines. Some
> of them are the same though.

Obviously, if you want to go down that route, you'll have to look at see
what's up with those lines.  It seemed to work on the line you originall
gave.

> I noticed another thing from these errors too, they're printed also
> when I exit zsh. There's not much point in lexing the history at
> write-out time, is there?

I can't remember what that is, but I've noticed it before and I have a
vague feeling I looked and decided I wasn't going to look any more.

> I tried playing around with the code a bit. The thing that looks
> suspicious to me is
>     if (*ptr != Meta && imeta(*ptr)) {
> Shouldn't they check ptr[0] and ptr[1] or so? I tried this,
>     if ((ptr == lineptr || ptr[-1] != Meta) && imeta(ptr[0])) {
> (in both places) but it didn't improve matters.

No, the problem here isn't a standard one with Tok -> Meta + NonTok.
It's that there's something that looks like Tok but isn't.  So we
need to turn it into Meta + NonTok.  That's the second part of the
test.  However imeta(*ptr) triggers if *ptr is Meta, which isn't what we
want because in that case it means we've hit a correct Meta + NonTok
sequence.

One thing I didn't think I needed to do, but may have got wrong,
is skip the byte after the Meta, i.e.

if (*ptr == Meta)
    ptr++;

(which acts together with the existing increment).

You could try that.

> I tried the following instead of the for+if:
>     unmetafy(lineptr, NULL);
>     lineptr = metafy(lineptr, -1, META_USEHEAP);
> and it gets rid of the errors, but it also causes insert-last-word to
> do nothing. So maybe my whole theory is wrong.

I'd be worried about doing that on every single line -- we already know
HIST_LEX_WORDS can be very slow, which is the only reason it's an option
(it's logically the right thing to do, modulo lex failures).

However, it should be possible to combine that with the "remeta" check,
i.e. see if the line needs it first, but not use the value of remeta,
just whether it's non-zero, and then it becomes easy and not too
intrusive --- and also safe about false positives.

I don't see why this would cause failures later on, particularly ones
apparently unrelated to the meta state of the string.

pws