Re: zsh-3.1.5-pws-5: mixing "old" and "new" completions?

[Sven Wischnowsky <wischnow@xxxxxxxxxxxxxxxxxxxxxxx>]

Bart Schaefer wrote:

> Could somebody who's managed to keep track of all this stuff (Sven?  Peter?)
> please post a complete summary for _both_ zle -c and zle -C of
> 
>   * the new commands (compadd, complist, compcall, etc.)
>   * the new variables (what's in them and when they're available)
>   * the parameters passed to a user's shell functions in each case
> 
> Unfortunately, I haven't had time to study Sven's posts enough to do it.
> 
> A whole lot of stuff has gone by recently that sounds good in the abstract
> but that feels, in Sven's words, "rather hackish."  If it were all written
> down in one place it might become obvious how it could all be made to fit
> together better.

Well...

New style completion stuff:

  zle -c name comp func

This defines a widget named `name' that calls the shell function
`func' to produce matches. All the other things completion does, like
displaying a list, deciding what to insert in the line, deciding if
menu-completion should be used and so on is currently only controlled
by the C-code. To allow the user to say how he would like this widget
to behave, (s)he has to give the name of one of the builtin
completion-widgets as the `comp' argument, e.g.:

  zle -c my-comp expand-or-complete my-comp-func

Defines the widget `my-comp', makes it call the shell function
`my-comp-func' and makes the completion code behave as if
`expand-or-complete' were used.

Now, inside the shell function the completion code sets up some
variables and offers a few extra builtins and condition codes (the
stuff below is copied and updated from message 4850):


The parameters are:

  PREFIX   the prefix of the current word from the command line
  IPREFIX  an ignored prefix (see below)
  SUFFIX   the suffix of the current word
  COMMAND  this is something like the `direct context'; for completion 
           of arguments it contains the command name, but for
	   subscripts or when completion the right hand side of a
	   parameter assignment this holds the name of the parameter;
	   also, when completing a redirection, this holds the
	   redirection operator itself (the command is still in argv)
  CONTEXT  the context we are currently completing in, this can be
           any of:

             `command'   we are completing in command position
             `argument'  we are completing in argument position
	     `redirect'  ... after a redirection operator
	     `math'      ... in a math environment
	     `subscript' ... in a subscript
	     `value'     ... in the value of an variable assignment
	     `condition' ... inside `[[...]]'

  argv     if completing an argument this holds the argument strings
           (not including the command name); when completing an
	   redirection it contains the command line strings (including 
	   the command name); when completing on the right hand side
	   of an array assignment this holds the strings in the
	   parentheses
  CURRENT  this is the index into `argv' pointing to the current word
  NMATCHES this is the number of matches produced (and accepted) so far

These variables are settable, the example file contains two convenient 
aliases to save and restore them. The completion code uses a wrapper
function to reset them to their old values at the end of each function
executed from the completion widget.


The conditions work on the contents of these variables, probably
changing them, they are modeled directly after the `compctl -x'
condition code:

  -prefix str
     is true if PREFIX starts with str

  -iprefix str
    like `-prefix', but makes the matched portion be
    ignored; this is done by appending the matched
    portion to IPREFIX and removing it from PREFIX

  -position beg [end]
    is true if we are currently in position `beg' (or between `beg'
    and `end' if that is given); if `end' is given COMMAND, CONTEXT,
    CURRENT, and argv are changed so that they contain only the range
    of words between these positions
    (as always when one of these condition takes an integer argument
    an expression may be given, like `-position a+1')

  -word index str
    is true if the `index'th argument is `str'

  -mword index pat
    like `-word' but with pattern matching

  -current index str
  -mcurrent index pat
    like `-word' and `-mword' but the `index' is take relative to the
    current position

  -string str
  -string index str
    true if the current word contains `str'; anything up to the
    `index'th occurrence of `str' is ignored (or anything up to the
    last occurrence if no `index' is given, again this sets IPREFIX
    and PREFIX)

  -class str
  -class index str
    like `-string' but the `str' is taken as a character class

  -words min [max]
    true if we have at least `min' arguments (and less than or equal
    to `max' arguments if that is given)

  -between s1 s2
    true if we are between an argument equal to `s1' and before an
    argument equal to `s2' (if there is such an argument)

  -mbetween p1 p2
    like `-between' but with pattern matching

  -after str
    true if we are after an argument equal to `str'

  -mafter pat
    like `-after' but with pattern matching

  -nmatches num
    true if we have generated `num' matches

Of course, tests can also be done by using the normal condition stuff
and the above variables (probably changing them), but these condition
codes may be faster and/or easier to use.


The builtins are:

complist ...

  This gets almost the same arguments as compctl, except the more
  complicated ones (like xor'ed completions, `-x', `-l', ...).
  When using this builtin, the completion code uses the current values
  of the variables above, so that the combination of this builtin and
  the condition codes makes life really simple...

compadd [options] matches...

  This allows access to the structure used internally used for storing 
  matches. When using this builtin the variables are not used and the
  completion code does not check if the strings added match what is
  one the command line, so you should know what you are doing.

  Supported options are:

    -q, -Q, -U, -P str, -S str, -J name, -V name
      these are used like the options with the same names in compctl and 
      complist

    -f 
      if this option is given the strings are treated like filenames
      (showing files types with LISTTYPES)

    -a
      this says to put the matches in the alternative set (those that
      are build for fignore ignored)

    -n
      this keeps the strings from appearing in the list of matches

    -i str
      this gives an ignored previous to use when inserting the match;
      such a string is not listed and inserted before a prefix given
      with the `-p' option

    -p str
      this gives another prefix that is not listed; but it is inserted 
      and used for testing file types

    -s str
      like `-p' but giving a suffix

  When I announced all this I said that this is least stable part of
  the new completion stuff because I hadn't had the time to check most 
  of this. Now, the example file contains a function that heavily uses 
  this builtin and seems to work fine.
  It is intended as a utility to give full control to the things stored 
  for each match. I would like to add some more functionality to make
  it easier to use for things people can be expected to want to have.
  For example:

  - a flag that says that the matches are filenames and that makes the 
    code automatically check fignore; this could also automatically
    try to list only path name components and the like (but I think we 
    an extra option to support this would be better, since the results 
    may probably be unexpected or unwanted)

  - a flag that says that normal matching rules should be applied
    (together with support for the `-M' option known from compctl and
    complist); this is important to have since currently all words
    given to compadd are taken even if they don't match what's on the
    line, so you have to do all the matching yourself (which can get
    complicated if you use `-M'); also, this isn't that hard to add
    since most of the code is already there

  - a way to specify when a added -S-suffix should be removed
    (i.e. after which characters are typed should the suffix be
    removed)

  Of course, other things may also turn out to be interesting, we'll
  see.

compcall [ -[TD] ]

  This is the youngets of the builtins. It allows to call the
  `compctl'-stuff from a new style completion widget. It takes no
  arguments, only two options. Normally, `compctl -T' and `compctl -D'
  are not used when calling `compctl's, since I expect this to be used
  mainly for command-`compctl's. However, if you want them to be used
  you can give the respective option to `compcall' and you'll get them 
  used. The completion code called is made to believe that the line
  looks like the stuff described in the parameters above (PREFIX,...).
  For this we needed some conversion and copying-around which may not
  be fully correct yet (so here is some code that at least needs some
  verifying, more likely some changes).
  If this turns out to be interesting enough to stay alive (and I
  almost expect it to be), there are many things we might to want add
  or change, e.g.:

  - make it take arguments which are to be taken as the words the
    `compctl'-code should think are on the command line
  - let it accept options with strings for the stuff which is
    currently taken from the parameters
  - for now, the status-return is always zero, it may be interesting
    or even important to let it return some useful things; this can be 
    information about continuing (`compctl -t<c>'), some kind of
    signal if a compctl has been processed, and so on.
  
compctl -K ' foo'

  This is as hackish as can be. This special syntax (note the space
  before the function name `foo') currently allows us to call `foo'
  from `compctl'-code in the same way as new style completion widgets
  are called, i.e. with the parameters set up and allowed to call
  `complist', `compadd', and `compcall'. I didn't bother to search a
  free option character for it (although, if I remember correctly,
  there still was one, but not many more than one), because I'm not
  too convinced, that we should support this, although I admit that it 
  may be interesting to have in some cases. But we may as well keep
  it, especially since this (currently) adds only two lines of code.


The examples in `new-completion-examples':

  This file contains all the code you need to try all this new
  completion stuff, including the definition of a completion widget
  and the call to `bindkey' to bind it to TAB.

  It has two functions to define completion handlers, `defcomp' and
  `defpatcomp', where the first defines a handler for commands or one
  of the special contexts (like `subscript') and the second one
  defines handlers that should be used when the command name matches a 
  given pattern.

  The master function of the code is `main-complete'. It first calls
  `compcall' to get access to the old `compctl's, then calls the
  equivalent of `-T' and finally has a look at CONTEXT to find out
  which handler it should call. If you are completing a command or
  argument, `do-complete' is called to handle that, finding matching
  patterns and normal handlers or the equivalent of `-D'. This also
  emulates the `use last path-name component' and the `=' special
  handling done by the `compctl'-completion code.

  To really call the handler, `call-complete' is invoked. This looks
  up the definition, which may be a array name or a function name. If
  it is a function, it is called (and will use `complist', `coompadd', 
  and friends). If it is an array, `complist' is called with the
  contents of the array as arguments. This allows one to keep simple
  definitions simple, as in:

    defcomp __command --command--
    __command=( -c )

  For the functions there are some helpers:

    - `compsave' and `compreset' can be used to save and restore the
      states of the special parameters; this is important to have when 
      using the conditions and if you want to make more than one test
      in the same function
    - `compsub' calls `do-complete' again with the current settings of 
      the special parameters; this is the replacement for `compctl -l'
    - `compalso' can be used to add the matches for another command or 
      special context (intended for e.g. `subscript includes math')
    - `pfiles' does partial-path completion. It gets no arguments or
      one of `-f', `-/', or `-g' known from `compctl'. If you use `-g' 
      the patterns to search should be given after the `-g', one per
      argument (*not* all in one string). As its first two arguments,
      it accepts `-W <paths>' where `<paths>' looks like the things
      you can give to `compctl -W'. With all this, `pfiles' is
      intended as a replacement for `comp{ctl,list} -[f/g]', giving
      you multicomp-like partial path completion everywhere.
    - `files' just calls `pfiles' with the arguments it got and, if
      that doesn't produce any matches, calls `pfiles' again with no
      arguments, producing all filenames. This is a replacement for
      `compctl -[f/g] ... + -f'.
      Yes, this could do with some work...

  One thing that is not yet correct is the continuation-stuff. The
  main functions use the return values of the handler functions for
  this: zero means that other handlers should be tried, non-zero means 
  that no more matches should be produced. I still have to check all
  the example handler functions if they behave correctly with respect
  to this.


Mixed comments (most of them are from the old message):

- I haven't used an associative array for the variables since I would
  then like to put [LR]BUFFER and so on into it and I didn't want to
  change to many parts of the shell for this first version.
  Also: using separate variables makes handling them so easy...
- When using `complist' the completion code still does normal completion 
  after `~', `=', and `$' (unless that is in IPREFIX, of course).
- Later we may add a way to access the matches produced and to control 
  what is done with the collected data (inserting, listing, ...)
- With `complist' you can make functions be called with `-K' and `-y'
  put they can't use `read' to get at the comamnd line data. This is
  because the new style completion uses a different way to store them
  (the above parameters).
- When using conditions that change the parameter values the behavior
  may be irritating since following tests use the changed values. To
  avoid this you have to put the test into a separate function so that
  the previous state is automatically restored by the completion code
  or you have to restore the state by hand. Since I expect that many
  users will want to put all their tests for a command in one function
  (making things better readable) restoring by hand and forgetting to
  do so in the right places will become inconvenient. This is the
  reason for the `compsave' and `compreset' helper aliases, I *really* 
  would like to find a way to take this burden from the user.
- The condition codes behave a bit different from their `compctl -x'
  counterparts since they can only access the arguments (not the
  command name).
- The attentive reader of the patch will notice some hunks in zsh.h
  and in mem.c. This implement a way to re-use older heaps. In fact
  there are functions that allow one to temporarily switch to another
  heap, go back to an older heap, and to temporarily switch to another 
  heap. The patch for zsh.h implements some macros like HEAPALLOC that 
  make it easy to use this mechanism.
- The value of `COMMAND' may be a bit irritating, since it depends on
  the `CONTEXT', we might want to rename it, change it, add another
  one, or something like that.

Ok, I hope this helps for now.

Bye
 Sven

P.S.: So long a message, and not even a patch, tststs ;-)

--
Sven Wischnowsky                         wischnow@xxxxxxxxxxxxxxxxxxxxxxx