Handle multi-line commands

[nrvale0]

Nice package. Having a problem using it when there are single or double-quotes in either command or the comments (using commentecho: true). For instance, the following line:

vault write -format=yaml pki_root/root/generate/internal \                                                                                                                                                                           
      common_name='somefakeorg.example Root CA' \                                                                                                                                                                                                                                                                                                                                                                                      
      ttl=8760h                                                                                                                                                                                                                                                                                                                                                                               

cause the following backtrace:

Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                                                                       
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/bin/doitlive", line 11, in <module>                                                                                                                                                                                                                                                                                                                                             
    sys.exit(cli())                                                                                                                                                                                                                                                                                                                                                                                                                      
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/click/core.py", line 722, in __call__                                                                                                                                                                                                                                                                                                               
    return self.main(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                                                                                    
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/click/core.py", line 697, in main                                                                                                                                                                                                                                                                                                                   
    rv = self.invoke(ctx)                                                                                                                                                                                                                                                                                                                                                                                                                
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/click/core.py", line 1066, in invoke                                                                                                                                                                                                                                                                                                                
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                                                                                                                                                                              
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/click/core.py", line 895, in invoke                                                                                                                                                                                                                                                                                                                 
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                                                                                                                                                                       
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/click/core.py", line 535, in invoke                                                                                                                                                                                                                                                                                                                 
    return callback(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                                                                                     
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/doitlive/cli.py", line 322, in play                                                                                                                                                                                                                                                                                                                 
    commentecho=commentecho)                                                                                                                                                                                                                                                                                                                                                                                                             
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/site-packages/doitlive/cli.py", line 129, in run                                                                                                                                                                                                                                                                                                                  
    command_as_list = shlex.split(ensure_utf8(command))                                                                                                                                                                                                                                                                                                                                                                                  
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/shlex.py", line 305, in split                                                                                                                                                                                                                                                                                                                                     
    return list(lex)                                                                                                                                                                                                                                                                                                                                                                                                                     
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/shlex.py", line 295, in __next__                                                                                                                                                                                                                                                                                                                                  
    token = self.get_token()                                                                                                                                                                                                                                                                                                                                                                                                             
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/shlex.py", line 105, in get_token                                                                                                                                                                                                                                                                                                                                 
    raw = self.read_token()                                                                                                                                                                                                                                                                                                                                                                                                              
  File "/home/redacted/.anyenv/envs/pyenv/versions/3.6.4/lib/python3.6/shlex.py", line 206, in read_token                                                                                                                                                                                                                                                                                                                                
    raise ValueError("No escaped character")                                                                                                                                                                                                                                                                                                                                                                                             
ValueError: No escaped character                                                                                                                                                                                                                                                                                                                                                                                                         

At least in the case of comments you can avoid the backtrace by escaping the single or double quotes but then the escaping also shows up in the commentecho output.

Ideas?

[sloria]

Thanks for reporting this. I don't have time to look into this right now, but I would certainly review and merge a PR for this.

[nrvale0]

This is not because of the nested quoting but because of the multiline escaped commands passed to Python shlex.

[sloria]

Ah, good catch @nrvale0 . I've renamed the issue.

[nrvale0]

The root issue is that the cli.py/run is reading the session one line at a time and then feeds each line to shlex.split(). In the situation where the command is multiline shlex is rightfully complaining that there's nothing following the line escape character. Some adjustments are necessary to the parsing logic in run()...seems like it should not be particularly difficult to add support for multiline.

[sparrow242]

Handle lines broken by '\' as one single line (on play and record) would fix the issue.
But that would lead to the behaviour that the written file will contain all parts of the ''-broken lines in a single line and if you replay a script with such broken lines they will be concated to one single line.

If that behaviour acceptable, I create a PR for that.

[sloria]

@sparrow242 Can you clarify your suggestion? Do you mean that if you have

vault write -format=yaml pki_root/root/generate/internal \                                                                                                                                                                           
      common_name='somefakeorg.example Root CA' \                                                                                                                                                                                                                                                                                                                                                                                      
      ttl=8760h                                                                                                                                                                                                                                                                                                                                                                               

in your session file, that it would play back as

vault write -format=yaml pki_root/root/generate/internal common_name='somefakeorg.example Root CA' ttl=8760h                                                                                                                                                                    ```

?

[sparrow242]

@sloria
Exactly.
If this behavior is okay, I would have a corresponding PR.

[nrvale0]

Appreciate the follow-up.

I don't have the original document anymore but I can mock something that works similarly and try that out... It'll be a minute. Stand by...

[sloria]

@sparrow242 I appreciate the suggestion and the PR, but I'm lukewarm on the behavior of implicitly concatenating the lines in the playback. Presumably, if you're adding a multiline command to your session file, it's because it's too long to look good on a single line, so you'd want the playback to respect the line breaks