Control-click in ANTLR Preview gives NoSuchElementException

[bseib]

To reproduce this bug, quit IntelliJ and start up fresh. The bug is triggered only once, so the fresh startup is needed to see it the first time.

Using this grammar:

grammar Expr;
prog:	expr EOF ;
expr:	expr ('*'|'/') expr
    |	expr ('+'|'-') expr
    |	INT
    |	'(' expr ')'
    ;
NEWLINE : [ \r\n]+ -> skip;
INT     : [0-9]+ ;

Open the ANTLR Preview (i.e. right click prog in the source, and choose "Test rule prog")

Enter the following input:

1 + 3 * 3 - 5

Hold the CTRL key, and hover over the + character, and you should see the grammar rule tool tip appear saying #1 Type '+', Line 1:2:

While still holding the CTRL key in this spot, click the mouse button. An exception occurs with this stacktrace:

java.util.NoSuchElementException: token index 85 out of range 0..75
	at org.antlr.runtime.BufferedTokenStream.get(BufferedTokenStream.java:154)
	at org.antlr.intellij.plugin.preview.InputPanel.setCursorToGrammarElement(InputPanel.java:562)
	at org.antlr.intellij.plugin.preview.PreviewEditorMouseListener.mouseClicked(PreviewEditorMouseListener.java:46)
	at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.runMouseClickedCommand(EditorImpl.java:3932)
	at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.mouseReleased(EditorImpl.java:3850)
	at java.desktop/java.awt.Component.processMouseEvent(Component.java:6656)
	at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3385)
	at java.desktop/java.awt.Component.processEvent(Component.java:6421)
	at java.desktop/java.awt.Container.processEvent(Container.java:2266)
	at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5026)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
	at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4948)
	at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4575)
	at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4516)
	at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2310)
	at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2804)
	at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:790)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
	at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:731)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:763)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:761)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:760)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:667)
	at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.kt:615)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:570)
	at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:68)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:349)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:348)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:787)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:348)
	at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:343)
	at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:995)
	at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:113)
	at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:995)
	at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$4(IdeEventQueue.kt:343)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:831)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:385)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

I'm using the latest IntelliJ:

IntelliJ IDEA 2023.1.3 (Ultimate Edition)
Build #IU-231.9161.38, built on June 20, 2023
Licensed to Broc Seib
Subscription is active until January 11, 2024.
Runtime version: 17.0.7+10-b829.16 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 13.4
GC: G1 Young Generation, G1 Old Generation
Memory: 2000M
Cores: 10
Metal Rendering is ON
Registry:
    debugger.new.tool.window.layout=true
    ide.experimental.ui=true
    eslint.additional.file.extensions=svelte
    ide.images.show.chessboard=true

Non-Bundled Plugins:
    org.antlr.intellij.plugin (1.20)
    org.rust.lang (0.4.197.5401-231)
    com.intellij.plugins.macoskeymap (231.7515.9)
    org.jetbrains.plugins.go (231.9161.14)
    com.intellij.nativeDebug (231.8770.15)
    String Manipulation (9.9.0)
    com.dherre3.wasm-text-parser (1.0.1)
    org.jetbrains.kotlin (231-1.9.0-release-358-IJ8770.65)
    com.github.bjansen.intellij.pebble (0.9.1)

Kotlin: 231-1.9.0-release-358-IJ8770.65

[bjansen]

@parrt I need help on this one. The code is basically this:

Token tokenUnderCursor = ParsingUtils.getTokenUnderCursor(previewState, offset);
Integer atnState = parser.inputTokenToStateMap.get(tokenUnderCursor);
Interval region = previewState.g.getStateToGrammarRegion(atnState);
CommonToken token = (CommonToken) previewState.g.tokenStream.get(region.a);

For some reason, atnState refers to a state that is not in the grammar, but apparently in a internal (generated?) grammar that looks like this:

expr
    :   ( {} INT<tokenIndex=41> 
        | '('<tokenIndex=45> expr<tokenIndex=47> ')'<tokenIndex=49> 
        )
        (
          {precpred(_ctx, 4)}?<p=4> ('*'<tokenIndex=20>|'/'<tokenIndex=22>) expr<tokenIndex=25,p=5>
                  | {precpred(_ctx, 3)}?<p=3> ('+'<tokenIndex=32>|'-'<tokenIndex=34>) expr<tokenIndex=37,p=4>
        )*
    ;

Then region refers to a location in this grammar instead of the grammar being tested, that's what causes the exception.

I have no idea how to fix this.

[parrt]

wow. that's a tricky one and I don't have a huge amount of time to dig into this. I wonder if the grammar and text window are somehow out of sync or grammar is delayed getting updated. I think I saw that at some point.

Ah. yeah, most likely it has something to do with conversion of left recursive grammars. Anytime there is a left recursive grammar we convert it to a new version of the rules which is where the atnState will be referring to. going backwards to the location in the original grammar would be tricky but on guessing I have a mechanism to do that mapping if only to produce error messages from the command line. I guess we will need a check in the plugin looking for left recursive rules and, if so, look in the original grammar for the position not inthe generated grammar. I would start by poking around and seeing how I generate error messages from the command line for left recursive rules.