Interpretation/handling of abstract commands feels unnecessarily strict

[ristomatti]

I believe the current handling of abstract commands is too strict and has some limitations that I've found a bit unexpected. This is about two separate issues, but I still decided to create only a single issue. I can split these into separate issues if that's preferred.

1. Unused abstract command makes the config invalid

In some situations this behavior is useful, but it can also create a jarring user experience. I think a warning would be more appropriate. This feels also inconsistent as an unused alias doesn't make the config invalid, even though they can sometimes be used for the same purpose.

There's two situations when I find this especially jarring. First example: when I start remapping a new app, I'd like to first write all the abstract commands I'm expecting to remap. Coming up with intuitive remappings some times takes some time so I might not do them all at once. I then have to comment out the commands and uncomment the commands, while I'd prefer them to be there when I need them.

The other type of situation I find this jarring is when some remap I've done turns out interfering with some other functionality while I'm working. I then usually just comment it out to be fixed later, but I then end up with an invalid config and have to go back and also comment out the abstract command.

2. Abstract commands cannot be combined with other output mappings

A couple of examples which illustrate how this can feel inconsistent or confusing.

Example 1:

✅ Base config

[class = "app1"]
AltLeft{F} >> Control{F}

[class = "app2"]
AltLeft{F} >> !AltLeft Control{F}

❌ Naive attempt to use abstract command, invalid

[class = "app1"]
AltLeft{F} >> find

[class = "app2"]
AltLeft{F} >> !AltLeft find

[default]
find >> Control{F}

✅ The above would be valid using an alias

find = Control{F}

[class = "app1"]
AltLeft{F} >> find

[class = "app2"]
AltLeft{F} >> !AltLeft find

Example 2:

✅ Debugging a config

log = $(echo "$1" >> ~/tmp/keymapper.log)

[class = "app1"]
AltLeft{T} >> log["app1 new tab"] Control{T}
AltLeft{F} >> find

[class = "app2"]
AltLeft{F} >> find

[default]
find >> Control{F}

❌ Further debugging, invalid config

log = $(echo "$1" >> ~/tmp/keymapper.log)

[class = "app1"]
AltLeft{T} >> log["app1 new tab"] Control{T}
AltLeft{F} >> log["app1 find"] find

[class = "app2"]
AltLeft{F} >> find

[default]
find >> Control{F}

✅ Same with an alias, valid

log = $(echo "$1" >> ~/tmp/keymapper.log)
find = Control{F}

[class = "app1"]
AltLeft{T} >> log["app1 new tab"] Control{T}
AltLeft{F} >> log["app1 find"] find

[class = "app2"]
AltLeft{F} >> find

[houmain]

Thanks for the suggestions.

1. I think it is save to allow abstract commands without mappings. It will behave just as if all mappings were in non-active contexts. I will change that.

2. Is not easy to change. While aliases are simply substituted when the configuration is loaded, abstract commands are mappings, which are resolved at runtime. I likely could allow some prefix to abstract command mappings, which would make your examples work, but I do not know when I am going to attempt that.

[ristomatti]

Number 2 is now crossed over. It sounds like it would have potential for breaking things. 🙂

[houmain]

I am also not entirely sure yet about 1. Not allowing unmapped abstract commands helps to spot typos like the following:

CapsLock >> BackSpace

The user likely wants to map the Backspace key instead of creating an unmapped command BackSpace.

[ristomatti]

Ah but I meant the other way around. To have abstract actions defined but not used in any mappings. 🙂

[ristomatti]

For instance, my use case is that I might have a set of abstract actions defined like below. Notice I have join_lines commented out as otherwise the config would not be valid. I removed the binding using it as I changed it to another action. I've still went through the trouble of adding the abstract action, so I want to keep it there for the time I come up with another binding.

[class = JetBrainsIDE]
  close_tab                 >> Alt{W}
  close_tool_window         >> Shift{Escape}

  find                      >> Control{F}
  find_in_files             >> (Control Shift){F}
  replace                   >> Control{R}
  replace_in_files          >> (Control Shift){R}

  show_actions              >> _Alt{A}
  show_context_actions      >> _Alt{Enter}
  show_hover_info           >> Control{F1}
  show_quick_documentation  >> Control{Q}

  toggle_terminal           >> ^ Alt{T}
  toggle_problems           >> ^ Alt{8}

  goto_file                 >> Alt{P}
  goto_symbol               >> Alt{S}

  goto_declaration          >> Control{B}
  goto_type_declaration     >> (Control Shift){B}
  goto_implementation       >> Control{Alt{B}}

  jump_to_text_start        >> Control{Home}
  jump_to_text_end          >> Control{End}
  jump_to_top               >> Control{PageUp}
  jump_to_bottom            >> Control{PageDown}
  jump_to_line              >> Control{G}
  jump_to_character         >> Alt{Comma}
  jump_to_word              >> Control{Comma}
  jump_to_matching_brace    >> Control{Shift{M}}
  jump_to_last_edit_loc     >> Alt{Shift{E}}

  next_error                >> !Escape _Alt{Escape}
  prev_error                >> !Escape _Alt{Shift{Escape}}

  next_function             >> _Alt{ArrowDown}
  prev_function             >> _Alt{ArrowUp}

  rollback_change           >> (Control Alt){Z}
  next_change               >> (Control Alt){ArrowDown}
  prev_change               >> (Control Alt){ArrowUp}
  next_difference           >> (Control Alt){ArrowDown}
  prev_difference           >> (Control Alt){ArrowUp}
  compare_next_file         >> (Alt Shift){N}
  compare_prev_file         >> (Alt Shift){J}

  extend_selection          >> _Alt{W}
  shrink_selection          >> _Alt{Shift{W}}

  cut_to_line_end           >> !AltGr (Shift Control Alt){Delete}
  delete_to_line_end        >> !AltGr (Control Alt){Delete}
  # join_lines                >> _Control{J}
  toggle_comment            >> !Escape _Control{7}
  toggle_block_comment      >> !Escape _Control{Shift{7}}
  accept_completion         >> ^ Tab

[houmain]

The other way round it would also have drawbacks. Allowing join_lines >> _Control{J} when join_lines was not defined would also make it impossible to detect typing errors like Caplock >> Backspace. There currently are no other warnings, only errors, which are reported as notifications, which have a very limited length.

One could use something like the following to temporarily disable a command:

OFF = Virtual99

OFF Meta{J} >> join_lines
join_lines >> _Control{J}

[ristomatti]

I see the issue now and agree it'd just make the situation worse. One way around this would be to enforce this recommendation from the documentation:

keymapper/README.md

Line 148 in 6b84248

To simplify mapping of one input expression to different output expressions, it can be mapped to an abstract command first. The command name can be chosen arbitrarily but must not be a key name. The configuration is case sensitive and all key names start with a capital letter, so it is advisable to begin command names with a lowercase letter:

This could be behind a "strict mode" startup flag or introduced as a breaking change. What do you think? Personally, I doubt the convenience would be worth the trouble for this alone. However, enforcing lower case initial letter for abstract actions sounds like an improvement in itself. It would for example:

• reduce the potential for confusion around the difference between macros and abstract actions
• simplify implementing editor syntax highlighting rules

[ristomatti]

One could use something like the following to temporarily disable a command:

OFF = Virtual99

OFF Meta{J} >> join_lines
join_lines >> _Control{J}

Neat workaround. I personally would find that a bit cluttering as often the case has been that I want to use the keybinding for some other action. But using the same idea, I will definitely start doing this:

None = Virtual99

None >> join_lines
None >> toggle_comment

It can be a useful indicator of "TODO" items also. 👍

[ristomatti]

@houmain Ok to close this?

[houmain]

I am thinking about adding directives some time.
Something like:

@skip-devices /.*/
@grab-devices /Some Keyboard/
@include "filename.conf"

Then I can also add a directive for allowing not mapped identifiers or enforcing some naming conventions.

[ristomatti]

Great idea!

[houmain]

@ristomatti did you see the new directives allow-unmapped-commands and enforce-lowercase-commands in 4.6.0?
They should be a solution for:

1. Unused abstract command makes the config invalid

[ristomatti]

Yup I've noticed! I've just installed yesterday evening. My plan was to test the new features and report back. Instead, I ended up doing a long overdue TODO item of extracting my Keymapper config and related scripts from my crowded dotfiles repo into a separate one. It ended up taking fair bit of time as I first had to learn how to use git-filter-repo to preserve the commit history. 😁

I can test the directives after sending this, perhaps also the other new features as well!

The release notes also mentioned these changes which potentially sound like something I just bumped into a few days prior to the release. Could you give a couple of more examples on what this means in practice:

Changes

• Keep key held when pressed immediately after !Any. e.g. !Any Shift{A} while Shift is already held.

I'd also be interested in examples of issues these aim to fixed:

Fixed

• Allow to undo ! in input. e.g. !Shift A Shift{B}
• Preserving order of hold back output.

I'm asking since any one of these sound like potential fixes to some unexpected things I ran across.

[ristomatti]

Happy to report both @allow-unmapped-commands and @enforce-lowercase-commands seem to work as documented 👍.

The only problem (for me) is that the fact that @allow-unmapped-commands does not remove the gripe I described here #148 (comment). What would be needed is to allow abstract commands to be unused. This was in fact what I was expecting @allow-unmapped-commands to do. 🙂

[ristomatti]

Perhaps @allow-undefined-commands would better describe what @allow-unmapped-commands does currently? Or have I managed to mix up the terminology altogether? 😄

Just to be sure I got it right, here's an example config to reflect my current understanding:

# Aliases
CursorDown = J
CursorLeft = H
CursorRight = L

# Context block
[class = /chrome/i]
  # Mapping to command "next_tab"
  AltRight{CursorRight} >> next_tab

  # Mapping to an undefined command
  AltRight{CursorUp} >> move_tab_left

  # Command
  next_tab >> Control{PageDown}
  
  # Unmapped command
  previous_tab >> Control{PageUp}

[ristomatti]

To further expand the above to cover more of the terms:

# Directive
@enforce-lowercase-commands

# Command that launches an app / executes an external command
new_terminal >> $(xterm)

# Forwarded (modifier) key
Meta >> Meta

# Mapping to type a string literal
Meta{F1} >> 'Hello world'

# Logical alias
CursorDirection = CursorUp | CursorDown | CursorLeft | CursorRight

# Parameterized alias / macro
Hold = $0{250ms}
DoubleTap = $0 !200ms $0

# Partially matched input expression
? A S D F >> F D S A

# Mapping that releases the pressed key before typing
Shift{Space} >> !Shift Backspace 

[ristomatti]

String literals as input is pretty cool. I guess it doesn't add anything that couldn't have been done already but it makes it a lot easier to use (and read) mappings consisting of a sequence of keys. For example:

AltRight 'w1' >> Meta{1}
AltRight 'w2' >> Meta{2}

I tried something like this as well and it worked:

Repeat3x = $0 $0 $0

? 'asdf' >> Repeat3x[Backspace] 'fdsa'

I also tried this:

Alt{'123'} >> '456'

That gave me an error, but I didn't even expect it work. 🙂

[houmain]

The only problem (for me) is that the fact that @allow-unmapped-commands does not remove the gripe

Sorry, now in 4.7.0 it should.

Perhaps @allow-undefined-commands would better describe what @allow-unmapped-commands does currently? Or have I managed to mix up the terminology altogether? 😄

I would say that A >> copy "defines" a command and copy >> B "maps" it.
But it does not really matter. There is only a @allow-unmapped-commands which now allows both to be incomplete.

Could you give a couple of more examples on what this means in practice:

Shift >> Shift
F1 >> "A"
[stage]
"AAA" >> X

This did not work correctly, since pressing Shift{F1} Shift{F1} Shift{F1} emitted an additional Shift between each Shift{A}, which made the "AAA" no longer match. So quite a special case.

I'd also be interested in examples of issues these aim to fixed:

Allow to undo ! in input. e.g. !Shift A Shift{B}:
A >> "aB" did output "ab" since it did not press the Shift, which was already prevented for the "a".

Preserving order of hold back output.
"aBc" >> X when A and Shift{B} are typed (and hold back) and the Space is pressed, which cancels the mapping, then "AB " was output, since the Shift was incorrectly forwarded first.

Partially matched input expression
? A S D F >> F D S A

I would rather call these no-partial-match input expressions (internally they are called no-might-match expression).

[houmain]

In 4.7.0 there is now a built-in repeat macro, so this would also work:

? 'asdf' >> repeat[Backspace, 3] 'fdsa'

And also a length and sub macro, so one can define a substitute macro like this:

substitute = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] "$1"
substitute["asdf", "fdsa"]

[ristomatti]

Just read the release notes 🤯. That sounds pretty sweet. I'll do a round of testing after I've installed my new ergo gear that just got delivered. 😁

[ristomatti]

First observation. After upgrading, my config failed validation due to an error I haven't encountered before: ERROR: Recursive macro instantiation in line 696. The offending line was this:

[device = Mouse]

Mouse was defined as:

Mouse = /Mouse|Logitech USB Receiver|G502/i

The issue went away by renaming the macro to be something that doesn't exist in the value. Is this intentional or a bug? The macro evaluation seems a bit "too eager" now? 🙂

[ristomatti]

I would say that A >> copy "defines" a command and copy >> B "maps" it.

Perhaps you're thinking about it based on how it's handled in the implementation rather than what it seems like to a user? 😬

But it does not really matter. There is only a @allow-unmapped-commands which now allows> both to be incomplete.

It does indeed! 👍 The new directives now solve the grievanses this issue was originally about, so I'm closing this.

I'll probably still post some quick feedback on the newly added features here, just to keep the discussion in one place.

[ristomatti]

# when last character of string is typed, undo using backspace and output new string
substitute = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] "$1"
substitute["Cat", "Dog"]

This is pure gold. The substitution happens so fast it feels as if I've just typed the substituted string 😂.

Does the evaluation of ? EXPR type mappings have a lot of overhead? I'm already thinking of replacing my ~20 Espanso text replacements. Perhaps these could also be used for fixing recurring typos or adding umlauts to most common Finnish words 🤔...

I'm asking because I've tried using them as part of some tricky to get right alternative modifier mappings. I don't remember the specific expressions I tried but it ended up creating a huge lag in all input. Most recent time was during an attempt to use Space as Meta when held, while retaining key repetition by quickly tapping Space twice. I almost got it working in a way to not make it come in the way of normal typing. Eventually I gave up on Space Space key repeat idea and settled with Shift{Space} >> Space. It was only one part of solution though... In case you're curious here's the complete solution I came up with after a lot of trial and error 😅

SpaceMeta       = Virtual8
Modifier        = ControlLeft | ControlRight | MetaLeft | MetaRight | AltLeft | AltRight | ShiftLeft | ShiftRight

[default]
  Shift                  >> Shift

[modifier = "SpaceMeta"]
  Modifier               >> (Meta Modifier)
  Any                    >> Meta{Any}

[modifier = "!Modifier"]
  Space{!150ms}          >> Space ^
  Space{150ms}           >> SpaceMeta ^ SpaceMeta

[default]
  Shift{Space}           >> Space
  Space                  >> Space

[ristomatti]

Just out of curiosity, I tried to simulate Espanso's string substitution with command output. It's not as elegant as few lines of YAML, but it can be done! 😁

Time = Virtual50

subst_virtual = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] $1

[default]
  ContextActive >> $(while true ; do (keymapperctl --instance time --wait-toggled "Time" --type $(date +"%H:%M:%S") &) ; sleep 1 ; done)

  subst_virtual[":time", Time]

[houmain]

The macro evaluation seems a bit "too eager" now? 🙂

Thanks for the feedback! Yes, this is a bug.

Does the evaluation of ? EXPR type mappings have a lot of overhead?

I think no more than the other mapping.

I don't remember the specific expressions I tried but it ended up creating a huge lag in all input.

I do not know of any performance problems. I would be glad for a configuration to reproduce.

Why not "simply" (I have not tested it)?

subst = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] $1
Time = $(keymapperctl --type $(date +"%H:%M:%S"))
subst[":time", Time]

[ristomatti]

Why not "simply" (I have not tested it)?

I blame the late hours (testing around 3-4am...). I didn't test yet myself but surely it will work. My first idea yesterday was using something like keymapperctl --timeout 1000 --wait-pressed Time --type $(date +"%H:%M:%S") --restart, but I figured already while typing that the time will be evaluated only once and stay constant. Besides that, it even ended up typing the value after the timeout whether the key got toggled or not. This was likely the moment my brain decided to overlook the most obvious solution... 🫠

I do not know of any performance problems. I would be glad for a configuration to reproduce

I'll check if some version of it ended up on to a commit before deletion.

[ristomatti]

To completeness sake, here's an Espanso config snippet:

matches:
  - trigger: ":tel"
    replace: "+1234567890"

  - trigger: ":gm"
    replace: "[email protected]"

  - trigger: ":dts"
    replace: "{{datetime}}"
    vars:
      - name: datetime
        type: date
        params:
          format: "%Y-%m-%d-%H-%M"

  - trigger: ":ds"
    replace: "{{date}}"
    vars:
      - name: date
        type: date
        params:
          format: "%Y-%m-%d"

  - trigger: ":now"
    replace: "{{time}}"
    vars:
      - name: time
        type: date
        params:
          format: "%H:%M"

The same ported over to Keymapper format:

type = $(keymapperctl --type $0)
format_date = $(date +"$0")

replace = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] $1

[default]
  replace[":tel"  type["+1234567890"]]
  replace[":gm",  type["[email protected]"]]

  replace[":now", type[format_date["%H:%M:%S"]]]
  replace[":dts"  type[format_date["%Y-%m-%d-%H-%M"]]]
  replace[":ds",  type[format_date["%Y-%m-%d"]]]

[ristomatti]

@houmain While cobbling that together, I ran across this rather unexpected issue:

subst_str = ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] "$1"
subst =     ? "$0" >> repeat[Backspace, sub[length["$0"], 1]] $1

Time = $(keymapperctl --type $(date +"%H:%M:%S"))

subst_str = [":gm", "[email protected]"] # ✅
subst[":time", Time] # ✅

subst[":tel", "+1234567890"] # ❌
# ERROR: Key name expected at '+' at "Backspace Backspace Backspace  +1234567890" in file '@)duU

Is this a bug or expected behavior?

I'd have expected substr_str = [":time", Time] to be invalid (like it is), but what happens to the quoted number in subst[":tel", "+1234567890"]?

[ristomatti]

Would it simplify parsing macros with parameters if there was an explicit syntax to do that? Something like a lambda expression:

subst = [$input, $output] -> ? "$input" >> repeat[Backspace, sub[length["$input"], 1]] $output

To my programmer brain this would feel easier to read, but not sure if that would be the case for non-programmers. That line becomes quite long though, it would be a nice have to be able to use e.g. shell command style string splitting. 😉

subst = [$input, $output] -> ? "$input" >> \
  repeat[Backspace, sub[length["$input"], 1]] $output

[ristomatti]

I do not know of any performance problems. I would be glad for a configuration to reproduce.

I've fiddled with the configs enough for today, but I found some earlier revisions that still had some of the potentially problematic no-might-match expressions. It might have been this one: https://github.com/ristomatti/keymapper-config/blob/29a4d248db1c5f93166426274730b078cc75ae9a/keymapper.conf#L310. IIRC all the related mappings are within the section marked "Stage 0" at the time.

BTW - in case you still want to link to my config in README.md, I've moved it from GIST into it's own repo (together with some related scripts etc.): https://github.com/ristomatti/keymapper-config. It also includes the gnarly commit history since I found Keymapper (extracted from my private dotfiles repo using git-filter-repo). 🙈

[ristomatti]

That line becomes quite long though, it would be a nice have to be able to use e.g. shell command style string splitting. 😉

Damn, I just casually threw in the idea 5 hours ago and you just went on an implemented it. 💪!

[houmain]

The macro evaluation seems a bit "too eager" now

This should be fixed in 4.7.1.

The same ported over to Keymapper format:

Great, that does not look too bad to me!

While cobbling that together, I ran across this rather unexpected issue:

This is the intended behavior. The quotes are not part of the parameter value. They are only for passing strings containing trailing spaces, a comma, ...

macro = $0

# these are the same:
macro["Abcd"]
macro[Abcd]

# these are not:
macro["A, B"]
macro[A, B]

So the macro decides, if the value should be in quotes or not:

macro = $(echo "$0")
macro = $(echo $0)

Would it simplify parsing macros with parameters if there was an explicit syntax to do that?

I don't think, it would simplify the code. Maybe the usage. But I thould bash also did not need more sophisticated parameter passing...

I found some earlier revisions that still had some of the potentially problematic no-might-match expressions

I threw these lines in my config, but did not notice a problem so far:

VSpace          = Virtual10
SpaceModifier = Control | Alt | Shift | Meta | Escape
SpaceModifier{Space}   >> SpaceModifier{Space}
? SpaceModifier Space  >> SpaceModifier Space
? Space Space          >> VSpace

BTW - in case you still want to link to my config in README.md, I've moved it from GIST into it's own repo

Link updated!

it would be a nice have to be able to use e.g. shell command style string splitting

This should also work now in 4.7.1.

[houmain]

Crazy, I think the last ten github accounts which starred keymapper are bots or something. Several joined github on the same day and most have the same incomplete README...

[ristomatti]

Thanks for the clarification on macro behavior, it makes a lot more sense now.

But I thould bash also did not need more sophisticated parameter passing...

Valid point!

[ristomatti]

Weird! Maybe some backlinking scheme? I've also got some obscure followers on my profile. Most are people I know but there's some that I have no clue why they'd follow me.

[ristomatti]

BTW, v4.7.1 seemed to break a lot of stuff. I'm getting errors at sections that have been untouched for ages. Also the just added text expansion config reports an error.

I don't know if these are all the errors but:

https://github.com/ristomatti/keymapper-config/blob/678cbb9db9f8c50a434842f979e7355c660733b2/include/text_expansion.conf#L7

ERROR: Invalid key 'repeat' at "repeat[Backspace, sub[length[":time"], 1]] $(keymapperctl --type $(date +"%H:%M:%S"))" in file '`<V
sh: 1: Syntax error: EOF in backquote substitution

https://github.com/ristomatti/keymapper-config/blob/678cbb9db9f8c50a434842f979e7355c660733b2/keymapper.conf#L761

ERROR: Invalid key 'compare_next_file' in line 761

[ristomatti]

I had backquotes within two context["..."] calls but removing them didn't seem to fix the issue.

[ristomatti]

I've got only couple of minor uncommitted changes to what's on my repo, so the config from there should error as well.

git clone [email protected]:ristomatti/keymapper-config.git

[houmain]

v4.7.1 seemed to break a lot of stuff

Oops, I removed the release.
I changed quite some code but the tests all succeeded.
I will try it with your configuration.

[ristomatti]

Not sure if relevant but note that regex matchers might contain backquotes as well.

[ristomatti]

I've downgraded back to 4.7.0, take your time. 🙂

[ristomatti]

Here's a snippet to paste in place of the @include in include/text_expansion.conf:

@include "include/.private.conf"

->

__tel = "+123123"
__email1 = "[email protected]"
__email2 = "[email protected]"
__email3 = "[email protected]" 

[ristomatti]

Crap, I forgot to warn about the parts that modify input settings and write stuff to ~/.local/state/keymapper etc. 😬

Commenting this out from keymapper.conf should avoid all that:

# Startup sequence
[default]
  ContextActive          >> on_startup
  on_startup             >> createStateDir applyInputDeviceConfig

[houmain]

Please give the new 4.7.1 a try!
I --checked your configuration and did not report any errors.

There is one breaking change though:
Aliases are no longer automatically substituted in terminal commands. One needs to use the new "string interpolation" feature there. The quotes are removed like for the macro arguments:

logColor = $(pastel --force-color paint "$1" "[$0]" >> LOG_FILE)
becomes:
logColor = $(pastel --force-color paint "$1" "[$0]" >> "$LOG_FILE")

[ristomatti]

Aliases are no longer automatically substituted in terminal commands. One needs to use the new "string interpolation" feature there. The quotes are removed like for the macro arguments:

I consider this an improvement. It was rather unexpected behavior to begin with. 👍

Breaking lines with \ seems to work based on a couple of test lines. State indicators, colored logging and newly added "text expansion" seemed to work as well. 🤞

Great work as always!