Re: mb_metacharlenconv vs. tokens

[Peter Stephenson <pws@xxxxxxx>]

Andrey Borzenkov wrote:
> I am afraid that mb_metastrlenconv is rather overloaded. As it looks like 
> traversing string character by character is valid operation for input as 
> well, so we cannot exclude tokens there.
> 
> Let's put it differently - what we intend is to avoid passing bogus character
> to mbrtowc(). If we *know* the context is tokenized we could just as well 
> pass a flag to itype_end() and mb_metacharlenconv() so they will check for 
> tokens and skip them. Does it actually make sense?

Yes, that makes perfect sense.  It's then a case of deciding on the
context, but that's a lot less heavyweight than untokenizing.

-- 
Peter Stephenson <pws@xxxxxxx>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php