Re: invalid characters and multi-byte [x-y] ranges

[Ismail Donmez <ismail@xxxxxxxx>]

Hi,

Peter Stephenson <p.stephenson <at> samsung.com> writes:

> 
> On Thu, 3 Sep 2015 11:09:44 +0100
> Stephane Chazelas <stephane.chazelas <at> gmail.com> wrote:
> > A discussed approach there was to internally represent bytes not
> > forming part of a valid character as code points in the range
> > D800-DFFF (specifically DC80 DCFF for bytes 0x80 to 0xff)
> 
> That's easy if wchar_t is actually Unicode.
> 
> I'm not sure how to do it otherwise.  We could treat it identically to
> the Unicode conversion of 0xdC00 + STOUCH(ch) to wchar_t, e.g. iconv
> UCS-4 to WCHAR_T, but is that guranteed to work?  This needs to be a
> robust fallback and it's not clear relying on iconv is the right thing
> to do.
> 
> The safe option would be only to use this if #ifdef __STDC_ISO_10646__.
> 
> On the other hand, it's probably not going to be worse than the previous
> code...

This seems to break glob tests:

./D02glob.ztst: starting.
*** /tmp/zsh.ztst.out.6468      Fri Sep  4 13:52:19 2015
--- /tmp/zsh.ztst.tout.6468     Fri Sep  4 13:52:19 2015
***************
*** 91,107 ****
  0:  [[ [ = [[] ]]
  0:  [[ ] = []] ]]
  0:  [[ [] = [^]]] ]]
! 0:  [[ fooxx = (#i)FOOXX ]]
  1:  [[ fooxx = (#l)FOOXX ]]
! 0:  [[ FOOXX = (#l)fooxx ]]
  1:  [[ fooxx = (#i)FOO(#I)X(#i)X ]]
! 0:  [[ fooXx = (#i)FOO(#I)X(#i)X ]]
! 0:  [[ fooxx = ((#i)FOOX)x ]]
  1:  [[ fooxx = ((#i)FOOX)X ]]
  1:  [[ BAR = (bar|(#i)foo) ]]
! 0:  [[ FOO = (bar|(#i)foo) ]]
! 0:  [[ Modules = (#i)*m* ]]
! 0:  [[ fooGRUD = (#i)(bar|(#I)foo|(#i)rod)grud ]]
  1:  [[ FOOGRUD = (#i)(bar|(#I)foo|(#i)rod)grud ]]
  0:  [[ readme = (#i)readme~README|readme ]]
  0:  [[ readme = (#i)readme~README|readme~README ]]
--- 91,114 ----
  0:  [[ [ = [[] ]]
  0:  [[ ] = []] ]]
  0:  [[ [] = [^]]] ]]
! 1:  [[ fooxx = (#i)FOOXX ]]
! Test failed:  [[ fooxx = (#i)FOOXX ]]
  1:  [[ fooxx = (#l)FOOXX ]]
! 1:  [[ FOOXX = (#l)fooxx ]]
! Test failed:  [[ FOOXX = (#l)fooxx ]]
  1:  [[ fooxx = (#i)FOO(#I)X(#i)X ]]
! 1:  [[ fooXx = (#i)FOO(#I)X(#i)X ]]
! Test failed:  [[ fooXx = (#i)FOO(#I)X(#i)X ]]
! 1:  [[ fooxx = ((#i)FOOX)x ]]
! Test failed:  [[ fooxx = ((#i)FOOX)x ]]
  1:  [[ fooxx = ((#i)FOOX)X ]]
  1:  [[ BAR = (bar|(#i)foo) ]]
! 1:  [[ FOO = (bar|(#i)foo) ]]
! Test failed:  [[ FOO = (bar|(#i)foo) ]]
! 1:  [[ Modules = (#i)*m* ]]
! Test failed:  [[ Modules = (#i)*m* ]]
! 1:  [[ fooGRUD = (#i)(bar|(#I)foo|(#i)rod)grud ]]
! Test failed:  [[ fooGRUD = (#i)(bar|(#I)foo|(#i)rod)grud ]]
  1:  [[ FOOGRUD = (#i)(bar|(#I)foo|(#i)rod)grud ]]
  0:  [[ readme = (#i)readme~README|readme ]]
  0:  [[ readme = (#i)readme~README|readme~README ]]
***************
*** 110,122 ****
  0:  [[ 633 = <1->33 ]]
  0:  [[ 633 = <->33 ]]
  0:  [[ 
12345678901234567890123456789012345678901234567890123456789012345678901234
567890foo = <42->foo ]]
! 0:  [[ READ.ME = (#ia1)readme ]]
  1:  [[ READ..ME = (#ia1)readme ]]
! 0:  [[ README = (#ia1)readm ]]
! 0:  [[ READM = (#ia1)readme ]]
! 0:  [[ README = (#ia1)eadme ]]
! 0:  [[ EADME = (#ia1)readme ]]
! 0:  [[ READEM = (#ia1)readme ]]
  1:  [[ ADME = (#ia1)readme ]]
  1:  [[ README = (#ia1)read ]]
  0:  [[ bob = (#a1)[b][b] ]]
--- 117,135 ----
  0:  [[ 633 = <1->33 ]]
  0:  [[ 633 = <->33 ]]
  0:  [[ 
12345678901234567890123456789012345678901234567890123456789012345678901234
567890foo = <42->foo ]]
! 1:  [[ READ.ME = (#ia1)readme ]]
! Test failed:  [[ READ.ME = (#ia1)readme ]]
  1:  [[ READ..ME = (#ia1)readme ]]
! 1:  [[ README = (#ia1)readm ]]
! Test failed:  [[ README = (#ia1)readm ]]
! 1:  [[ READM = (#ia1)readme ]]
! Test failed:  [[ READM = (#ia1)readme ]]
! 1:  [[ README = (#ia1)eadme ]]
! Test failed:  [[ README = (#ia1)eadme ]]
! 1:  [[ EADME = (#ia1)readme ]]
! Test failed:  [[ EADME = (#ia1)readme ]]
! 1:  [[ READEM = (#ia1)readme ]]
! Test failed:  [[ READEM = (#ia1)readme ]]
  1:  [[ ADME = (#ia1)readme ]]
  1:  [[ README = (#ia1)read ]]
  0:  [[ bob = (#a1)[b][b] ]]
***************
*** 138,144 ****
  0:  [[ aaaXaaabY = (#a1)(a##b)##Y ]]
  0:  [[ aaaXbaabY = (#a1)(a##b)##Y ]]
  1:  [[ read.me = (#ia1)README~READ.ME ]]
! 0:  [[ read.me = (#ia1)README~READ_ME ]]
  1:  [[ read.me = (#ia1)README~(#a1)READ_ME ]]
  0:  [[ test = *((#s)|/)test((#e)|/)* ]]
  0:  [[ test/path = *((#s)|/)test((#e)|/)* ]]
--- 151,158 ----
  0:  [[ aaaXaaabY = (#a1)(a##b)##Y ]]
  0:  [[ aaaXbaabY = (#a1)(a##b)##Y ]]
  1:  [[ read.me = (#ia1)README~READ.ME ]]
! 1:  [[ read.me = (#ia1)README~READ_ME ]]
! Test failed:  [[ read.me = (#ia1)README~READ_ME ]]
  1:  [[ read.me = (#ia1)README~(#a1)READ_ME ]]
  0:  [[ test = *((#s)|/)test((#e)|/)* ]]
  0:  [[ test/path = *((#s)|/)test((#e)|/)* ]]
***************
*** 177,180 ****
  0:  [[ test.bash = *.?(#c1,2)sh ]]
  0:  [[ test.bash = *.?(#c1,)sh ]]
  0:  [[ test.zsh = *.?(#c1,)sh ]]
! 0 tests failed.
--- 191,194 ----
  0:  [[ test.bash = *.?(#c1,2)sh ]]
  0:  [[ test.bash = *.?(#c1,)sh ]]
  0:  [[ test.zsh = *.?(#c1,)sh ]]
! 14 tests failed.
Test ./D02glob.ztst failed: output differs from expected as shown above 
for:
  globtest globtests
Was testing: zsh globbing