Khard - Supporting a command using user input and escape sequences

[tkapias]

Introduction

I use khard to search and edit my vcard contacts.

The main issue was that it was designed for the console, so I wrote a config for hl and a function to call it with khard.

For a simple listing, there is no issue:

• khard | hl --khard
  

But a more useful command in khard is show to get details for one contact, it filters the list with options or by asking the user to provide an index number manually:

• khard show
  

• 2
  

Issues

When khard show is piped to hl the user prompt Enter Index (0 for None, q to quit): is not visible before user input.

• khard show | hl --khard
  

After the user input, there is an issue with the escaping sequence and the prompt appears on the first line of the output, with the name of the contact.

• 2
  

Question

This kind of command (khard show) is probably not designed to be piped so I don't want to open an issue with khard at first.
Is there a way to solved that on hl side?

Reproduce

• khard can be installed with most package managers or pipx.
• My configuration for khard:

~/.config/khard/khard.conf

[addressbooks]
[[principal]]
path = ~/.contacts/fake/

[general]
debug = no
default_action = list
editor = vim, -i, NONE
merge_editor = vimdiff

[contact table]
display = first_name
group_by_addressbook = no
reverse = no
show_nicknames = yes
show_uids = yes
sort = last_name
localize_dates = yes
preferred_phone_number_type = pref, cell, home
preferred_email_address_type = pref, work, home

[vcard]
private_objects = Telegram, Matrix, XMPP, Tox
preferred_version = 4.0
search_in_source_files = yes
skip_unparsable = yes

• 2 fake vcard contacts:

~/.contacts/fake/Jane_Doe.vcf

BEGIN:VCARD
UID:02446dd0-6eb1-4d54-a9d5-387af78d0a6d
VERSION:4.0
PRODID:-//BBros.us llc//bvCard.com//EN
N:Doe;Jane;;;
FN:Jane Doe
EMAIL:[email protected]
ORG:Google
TEL:123-123-123
URL;type=pref:http://google.com
ADR:;;;;;;France
END:VCARD

~/.contacts/fake/John_Doe.vcf

BEGIN:VCARD
UID:bf675293-0b61-47ac-ad4a-4966bd7aacff
VERSION:3.0
PRODID:-//BBros.us llc//bvCard.com//EN
N:Doe;John;;;
FN:John Doe
EMAIL;TYPE=home:[email protected]
NOTE:Fake ID
ORG:Microsoft
TEL;TYPE=cell:+33600000000
URL;type=pref:http://microsoft.com
ADR:;;;;;;France
END:VCARD

• My hl conf for khard:

khard            :
     -e
     -3w '^[ ]{4}?[a-zA-Z][a-zA-Z -]{1,12}:'
     -3g '^Address book: (.*)$'
     -3y '^Index.*$'
     -1y '^[0-9]+'
     -2c 'Nickname:'
     -1w '[ ]{4}([a-zA-Z]+:)'
     -3r ': (\+?\(?[0-9]{1,3}\)?[0-9 -]{8,13}[0-9])'
     -3b '([^ ]+)@[^ ]+\.[^ ]+'
     -2b '[^ ]+(@[^ ]+\.[^ ]+)'

[mbornet-hl]

Hello Tomasz,

thank you for your report.
I don't think I'll have time to investigate your problem, I'm really very busy at the moment with writting tasks (articles and a book).
The problem you describe reminds me some buffering problems. I don't think it's on hl's side, it may be related to the way khard writes to pipes. When a program writes to stdout and to stderr, if both are connected to a TTY, data are flushed immediately, but if you redirect the outputs to files, stdout is buffered (by default) and stderr is not. The problem you describe makes me think that the messages you normally see without hl (i.e. without pipe) may be buffered when you use a pipe. If it's the cause of the problem, you should have a look at the "stdbuf" command, and for example :
https://unix.stackexchange.com/questions/25372/how-to-turn-off-stdout-buffering-in-a-pipe
or try to unbuffer Python's output with the "-u" option :
https://stackoverflow.com/questions/881696/unbuffered-stdout-in-python-as-in-python-u-from-within-the-program
Tell me if that solves your problem, or if it needs to be further investigated.
Kind regards,
Martial

[mbornet-hl]

By the way, you made a nice description of the problem. Congratulations !

[tkapias]

Thank you @mbornet-hl

I just posted a solution in a new issue at Khard.

Your links helped me to eliminate the most obvious possibility (forcing the flush). it did not work, but adding some more newlines in the code worked.

hl could still be involved in the issue because piping to cat does not break the output at the user input. It would be interesting to find other python commands with an inline user input prompt to see if it would be useful to look into it more later.

[mbornet-hl]

Thank you for your quick answer.
I'll try to make some tests with simple C programs, since I think the problem is related to the way data is written into the pipe, i.e. with or without newline character, and with or without flush flag.
By the way, did you use the "-u" option of hl when you did your tests ?
-u : do not bufferize output on stdout
I'm not sure I fully tested it, but maybe it helps.

[mbornet-hl]

OK, no need to write C programs to test, I'm able to reproduce the problem with the following shell commands :
$ (echo -n abcde; sleep 10) | cat
and :
$ (echo -n abcde; sleep 10) | hl -u -3g '[a-z]'

So I confirm that there's a problem in the way hl reads its input.

[tkapias]

About -u, I tested with PYTHONUNBUFFERED=1 env variable which does the same thing. But there was no difference. And stdbuf too.

Nice test with the echo -n.
The -n does not end with a newline just like in an input prompt like this one: (read -p "Enter fullname: " fullname; echo "Hello $fullname")

I checked it with every pager I have (bat, less, more, glow) and they all wait for an end of line. So hl is behaving more like a pager than cat on this issue.

[tkapias]

I typed something wrong and made a mistake in the previous message. This command works fine:

(read -p "Enter fullname: " fullname; echo "Hello $fullname") | hl -e -3y '^[a-zA-Z0-9]'

The Prompt is not colorized (the rest is), but it's displaying at the right time.

The echo -n was the best test.

[lucc]

I just posted a bit of an analysis at the corresponding issue for khard](lucc/khard#333 (comment)). I also think the problem is routed in input/output buffering but I am not sure if there is even a solution:

I did not read your source and could not find a hint in the readme but I assume that hl works in linewise mode. So I assume it waits for a complete line on stdin until it starts to match and output that line.

But I do not know if there is any way around it. You could of course check all the regexes in the active config and for some unfinished input you can know that it will not be part of a match but for some other partial lines you never know if they will match at some later point. See for example

(echo -n foo; read x; if [ -z "$x" ]; then echo bar ; else echo baz; fi) | hl -eg '(f)ooba(r)'