Re: [BUG] In reference to patch 39815, about (z) flag and $( parse error

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On Thu, Oct 12, 2017 at 8:50 AM, Peter Stephenson
<p.stephenson@xxxxxxxxxxx> wrote:
> On Thu, 12 Oct 2017 08:26:35 -0700
> Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
>> It'd most likely need to be in bufferwords(), which somehow needs to
>> know that the STRING token returned from the lexer is [or not] a
>> comment
>
> The traditional fix for this is a hacky state variable.

After beating my head against this for a while and observing
confusingly different (and sometimes confusingly NOT different)
behavior with various attempts to track whether we were scanning
comment or code, I finally went back to the original commit c0d01a6f,
and realized that it rewrote skipcomm() which is called from down in
the guts of gettok() via gettokstr().  After I got there it became
ridiculously easy.  (As they say, a $1000 job -- $1 for the missing
screw, $999 for knowing where to put it.)

diff --git a/Src/lex.c b/Src/lex.c
index 8493d47..e0190af 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -2128,8 +2128,12 @@ skipcomm(void)
      * function at the history layer --- this is consistent with the
      * intention of maintaining the history and input layers across
      * the recursive parsing.
+     *
+     * Also turn off LEXFLAGS_NEWLINE because this is already skipping
+     * across the entire construct, and parse_event() needs embedded
+     * newlines to be "real" when looking for the OUTPAR token.
      */
-    lexflags &= ~LEXFLAGS_ZLE;
+    lexflags &= ~(LEXFLAGS_ZLE|LEXFLAGS_NEWLINE);
     dbparens = 0;      /* restored by zcontext_restore_partial() */

     if (!parse_event(OUTPAR) || tok != OUTPAR) {