Feature mouse commands #448
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add skeleton NewCommandMouseView
- Finish new mouse command implementation - Add `MouseCommandView` UI - Add actions for updating mouse commands
Bump `AXEssibility` to version `0.0.9`
The mouse command models and views have been updated to support specifying the click location within the element. This enhancement adds flexibility for users to define mouse interactions more precisely by choosing from predefined locations (e.g., top, center, bottom) or specifying custom coordinates. This change addresses scenarios where the default click position might not trigger the intended behavior in the UI element. MouseCommand.Kind now includes an associated ClickLocation enum with various standard positions and a custom coordinate option. Updates to the views ensure that users can select the desired click location via menus and define custom x, y coordinates when needed. Furthermore, the MouseCommandRunner is adapted to translate the ClickLocation to the actual point on the screen and pass it to the mouse event posting functionality. This helps to simulate user interactions in a more targeted and accurate manner.
… CGRect directly In our accessibility element resolution strategy, we have refactored multiple resolvers such as AXCollectionResolver, AXGroupResolver, AXListResolver, AXOutlineResolver, AXScrollAreaResolver, and introduced AXTextEntryAreaResolver to directly return the frame (CGRect) of the focused element instead of returning an AnyAccessibilityElement. Previously, we were obtaining an AnyAccessibilityElement and then extracting its frame which was an unnecessary intermediate step. This change simplifies the API and directly provides the essential data needed for further processing. Additionally, the MouseCommandRunner has been updated to handle the new return type and to consider the newly added case .textEntryArea while resolving the frame. The added error case in MouseCommandRunnerError ensures that an appropriate error is thrown when a frame cannot be resolved, enhancing error handling.
Improved support for accessibility elements and mouse actions by introducing a variety of new resolvers and refining existing ones. This commit provides better handling of different content areas exposed via accessibility APIs and more accurately targets mouse events based on context. - Introduced `AXEditorResolver`, `AXHTMLResolver`, and `AXTableResolver` to handle specific types of UI elements, improving the targeting of mouse actions. - Updated `AXGroupResolver` to default to the parent element if no match is found within the group. - Created `CustomMouseRoutine` and associated classes to handle custom mouse routines for specific applications identified by bundle identifiers. An example being Xcode, in which a specialized routine calculates mouse target locations within the text entry area. - Refined `MouseCommandRunner` to integrate new resolver logic, including support for the new `AXHTMLResolver`. - Ensured the mouse is moved back to its original location after a command is executed to maintain user cursor context. - Added logic to `UserSpace` for obtaining selected text from web content via clipboard manipulation, accommodating cases where direct access to selected text is not possible through accessibility APIs. These enhancements provide a framework to extend and customize mouse interactions within various applications, ensuring a smoother user experience when automating tasks that rely on precise element targeting and mouse events.
Optimized the element resolution process in AXScrollAreaResolver by extracting and refactoring the logic of checking for focused or selected elements into a separate, recursive function. This enhances maintainability and readability of the code by reducing complexity within the resolveFocusedElement function and centralizing the decision-making process for finding the correct element.
…Routine To improve the maintainability and extendibility of the MouseCommand model, we've refactored the way click locations are accessed and modified. The MouseCommand.UIElement enum now includes a computed property for directly getting and setting the ClickLocation without duplicating switch-case logic. Additionally, the XcodeMouseRoutine has been updated to utilize this new property, streamlining the logic for frame calculations and making it easier to add new click location behaviors in the future. This change promotes cleaner code practices and prepares for potential feature expansion.
|
Test: succeeded ✅ |
|
Test: succeeded ✅ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MouseCommandRunnerBumpAXEssibilityto version0.0.9