Re: mb_metacharlenconv vs. tokens

[Andrey Borzenkov <arvidjaar@xxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 26 September 2006 22:10, Peter Stephenson wrote:
> Andrey Borzenkov wrote:
> > Then this is very basic bug, because as simple as running function from
> > V01 test results in:
> >
> > Breakpoint 2, mb_metacharlenconv (
> >
> > s=0xb7c41951 "\215\210M\211\205\215\210f\211\231\212\210zmodload
> > -d\211\231\2 16:
> > \204\207:\207 \205m\210 \207\214\211\216", wcp=0xbfd4ef88)
> >
> > which corresponds to the line in zmodunload:
> >
> > if [[ -z ${(M)${(f)"$(zmodload -d)"}:#*:* $m( *|)} ]]
> >
> > > It should have been
> > > untokenized at some point in the sequence leading to the call,
> >
> > #2  0x080bd2a4 in paramsubst (l=0xbfd4f7ac, n=0xbfd4f7a0, str=0xbfd4f388,
> >     qt=0, ssub=4) at /home/bor/src/zsh/Src/subst.c:1499
>
> The problem is probably here (or around here... there's been some
> recursive jiggery pokery). 

This is one seems pretty much top level.

> We should untokenize a nested 
> substitution before trying to do anything with it, and only tokenize it
> later if the effect of GLOB_SUBST is present.  This is roughly where I
> saw the problem before.

I am not sure if this is really possible (or feasible). While paramsubst case 
is trivially solved (worked around actually) by

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.63
diff -u -p -r1.63 subst.c
- --- Src/subst.c 23 Sep 2006 20:25:06 -0000      1.63
+++ Src/subst.c 27 Sep 2006 16:23:41 -0000
@@ -1496,12 +1496,13 @@ paramsubst(LinkList l, LinkNode n, char
      * these later on, too.
      */
     c = *s;
- -    if (itype_end(s, IIDENT, 1) == s && *s != '#' && c != Pound &&
+    if (*s != '#' && c != Pound &&
        c != '-' && c != '!' && c != '$' && c != String && c != Qstring &&
        c != '?' && c != Quest &&
        c != '*' && c != Star && c != '@' && c != '{' &&
        c != Inbrace && c != '=' && c != Equals && c != Hat &&
- -       c != '^' && c != '~' && c != Tilde && c != '+') {
+       c != '^' && c != '~' && c != Tilde && c != '+' &&
+       (itok(c) || itype_end(s, IIDENT, 1) == s)) {
        s[-1] = '$';
        *str = s;
        return n;


fetchvalue() case is not; and in fetchvalue() we explicitly look for tokenized 
string.

I am afraid that mb_metastrlenconv is rather overloaded. As it looks like 
traversing string character by character is valid operation for input as 
well, so we cannot exclude tokens there.

Let's put it differently - what we intend is to avoid passing bogus character 
to mbrtowc(). If we *know* the context is tokenized we could just as well 
pass a flag to itype_end() and mb_metacharlenconv() so they will check for 
tokens and skip them. Does it actually make sense?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFGqdnR6LMutpd94wRAvpwAKCfhCFzK6FdvNnvSGHkDx8rx2PjnQCcDaZp
ThXNuFDZQcgDGLknXoAY5jE=
=fLqH
-----END PGP SIGNATURE-----