Fix go to definition and hover for files containing multibyte characters #2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a Ruby file contains multibyte characters, the go to definition and hover features do not work correctly. This is because the index building and document referencing logic does not properly handle multibyte characters when calculating offsets. This commit fixes the issue by: •Updating index building to use character offsets instead of byte offsets •Modifying document referencing to properly handle multibyte characters when mapping between positions and offsets •Adding test cases to verify go to definition work with multibyte characters
|
I have signed the CLA! |
|
Thank for you the contribution. From a quick read, it seems the PR is doing two main things:
Since UTF-8 is almost always used, is it possible that the first can be addressed without the second? |
andyw8
added
server
|
@andyw8 |
When a Ruby file contains multibyte characters (like Japanese, Chinese, emoji, etc), the go to definition and hover features do not work correctly. Because the document referencing logic does not properly handle multibyte characters when calculating offsets. This commit fixes the issue by: *Modifying document referencing to properly handle multibyte characters when mapping between positions and offsets *Adding test cases to verify go to definition work with multibyte characters
|
I opened a new PullRequest. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Closes #1251
When a Ruby file contains multibyte characters (like Japanese, Chinese, emoji, etc), the go to definition and hover features do not work correctly. The definition location or hover documentation will be incorrect.
ref:
This is because the current implementation assumes single-byte characters when calculating offsets during index building and document referencing. We need to properly handle multibyte characters to ensure these features work reliably for all users.
Implementation
RubyLsp::DocumentandRubyLsp::Requests::Requestto calculate locations considering multibyte characters. This change utilizes the API implemented in Prism by Add code unit APIs to location ruby/prism#2406.Prism::Locationto RubyIndexer::Locationby Refactor global usage ofPrism::Locationto minimize memory usage #1917, a new field has been added to RubyIndexer::Location.Automated Tests
I added a simple expectation test for the definition.
Manual Tests