Special characters not highlighted in strings

[Av1d1ty]

Zig Version

0.12.1

Zig Language Server Version

0.13.0, 0.12.0

Client / Code Editor / Extensions

nvim 0.10, lspconfig

Steps to Reproduce and Observed Behavior

Open file with a special character like \n or \t in a string literal. See that all characters have the same color.

Expected Behavior

Special characters highlighted differently

Image above shows what it looks like when ZLS is turned off.

Relevant log output

No response

[Techatrix]

Above Terminal is NeoVim 0.10.0, Bottom Terminal is Neovim 0.9.5

With ZLS

Without ZLS

In case this is hard to see, the escape characters are highlighted except with ZLS on NeoVim 0.10.0. Everything else was as expected.
I tried out a few of the default colorschemes in NeoVim 0.10.0 but that didn't help.
I did not observe any changes when enabling the zig.vim extension.

So it seems like something is going on with NeoVim 0.10.0... 🤔

The `init.lua` being used

local vim = vim
local Plug = vim.fn['plug#']

vim.call('plug#begin')
Plug('neovim/nvim-lspconfig') -- https://github.com/neovim/nvim-lspconfig
-- Plug('ziglang/zig.vim') -- https://github.com/ziglang/zig.vim
vim.call('plug#end')

-- I disabled ZLS by removing everything below here

local lspconfig = require('lspconfig')
lspconfig.zls.setup {
  cmd = { '/home/techatrix/repos/zls/zig-out/bin/zls' },
  settings = {
    zls = {
      zig_exe_path = '/nix/store/s3nq31mhm8gxkq691p5w6q61ficw1hvr-zig-0.13.0/bin/zig',
    }
  }
}

[Av1d1ty]

Also special characters are displayed correctly in Lua files with lua-ls active.

[WillLillis]

It seems like neovim's semantic token highlighting are overriding tree-sitter's highlight's for the escaped characters. Adding the following to my config disabled semantic highlighting for zig string tokens, and appears to fix the issue:

vim.api.nvim_set_hl(0, '@lsp.type.string.zig', {})

I'm not sure what a more permanent/ "out of the box" fix would be, as it seems that semantic token highlights have a higher priority than tree-sitter's by default.

[Vexu]

The same thing happens in vscode when semantic_tokens is set to full instead of partial, maybe other editors with grammar based highlighting should also use partial?

[Techatrix]

Perhaps it is more reasonable to assume that the editor already provides syntax based highlighting for Zig code? Then we could make partial the default which also fixes this issue.

[Av1d1ty]

Can confirm that setting semantic tokens to "partial" does fix the issue, but at the same time highlighting changes in places where I prefer it would not

Semantic tokens set to "full"

Semantic tokens set to "partial"

As you can see, function parameters are no longer highlighted distinctly in function body.

[Av1d1ty]

Also, on a side note, Mason tells me that default value for semantic tokens in "partial", while in fact it is set to "full"

[Av1d1ty]

Side note 2: Mason shows config names in camelCase, while in fact they need to be set in snake_case in my config files. Not sure if it is an actual issue. Maybe I lack understanding about how Mason and configs work for language servers.

[Techatrix]

Mason's package.yaml for ZLS uses vscode-zig's package.json to provide information about config options. This should not be done because vscode-zig options don't exactly match what ZLS uses. I will look into resolving that.

[Techatrix]

Can confirm that setting semantic tokens to "partial" does fix the issue, but at the same time highlighting changes in places where I prefer it would not

This should be fixed by 2e5a2a5.

I used folke/tokyonight.nvim and set semantic_tokens to partial. Here is how it changed:

Before

After

[Av1d1ty]

Awesome, thanks!