So it seems that "builtin log" takes precedence over commands named "log" in $path, but only for non-interactive shells.
Is this intentional? It seems unexpected to me, and it was quite hard to figure out what the root cause was as the error message is not useful (./myscript:log:5: too many arguments).
I know I can work around this with "command log", I was just surprised that it works this way.
$ cat what
#!/usr/bin/env zsh
which -a log
$ which -a log
/usr/bin/log
$ zsh ./what
log: shell built-in command
/usr/bin/log
$ zsh -i ./what
/usr/bin/log