WIP: Add tapping of accessibility element using AXLabel #758
Conversation
|
Hi @jhildensperger! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
| x += axframe["x"] | ||
| y += axframe["y"] | ||
| break | ||
| except: |
There was a problem hiding this comment.
What could make it raise?
There was a problem hiding this comment.
I think the enum will raise a ValueError if it doesn't exist for the role:
HIDElementType(item["role_description"])
I'm happy to change that if there is a preferred way of handling this scenario.
| axElement = HIDElementType(item["role_description"]) | ||
| axLabel = item["AXLabel"] | ||
|
|
||
| elementFound = axElement == element and ( |
There was a problem hiding this comment.
For all the element types except for back, does it make sense to allow commands without a label?
The current behavior is that:
$ idb ui axtap button
Succeeds and returns the first button it encounters, right?
There was a problem hiding this comment.
Yes, it just uses the first element of the type.
It's debatable where or not that is useful because I don't know if we can guarantee that the order of accessibilityInfo.json is the same every run, though it appeared that way when testing.
| @@ -121,6 +121,13 @@ class HIDButtonType(Enum): | |||
| SIRI = 5 | |||
|
|
|||
|
|
|||
| class HIDElementType(Enum): | |||
There was a problem hiding this comment.
Could you show an example of each of these from an actual output?
| for n in range(count): | ||
| await self.send_events(tap_to_events(x, y, duration)) | ||
| else: | ||
| print("AXElement with AXLabel: ", label, " type: ", element, " not found.") |
There was a problem hiding this comment.
We could log it, however let's not send it to stdout and preserve the current tap behavior.
There was a problem hiding this comment.
👍 Can change to logging instead of print.
There was a problem hiding this comment.
I would not preserve the current tap behavior because the coordinates provided are suppose to be relative to the accessibility element's frame and default to (1,1). It is probably better to just do nothing than send tap to unintended coordinates.
|
@fgasperij Thanks for the feedback. I would like to also have some test coverage, but I can't see to get all of the tests to run. Could to please share the command to run the tests? |
Motivation
Tapping (x,y) is great, but we might not always know the coordinates of an element on all devices.
In addition to tapping a coordinate, we can leverage the accessibility elements of the screen to find the coordinates of the element we'd like to tap. This is very similar to how developers would write their UITests in their normal workflow.
Elements supported for interaction
Examples
Tap the first back button on the screen:
idb ui axtap backTap a button:
idb ui axtap button "My Button"Tap some text:
idb ui axtap text "some text with an attached gesture recognizer"Tap a text field:
idb ui axtap textfield TextFieldTest Plan
TBD