Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Self auditing benchmark orchestration #43

Open
hidmic opened this issue Apr 12, 2023 · 0 comments
Open

Self auditing benchmark orchestration #43

hidmic opened this issue Apr 12, 2023 · 0 comments
Labels
enhancement New feature or request needs-work Ready to be worked on

Comments

@hidmic
Copy link
Collaborator

hidmic commented Apr 12, 2023

Feature description

Connected to #35, but preexisting. Using embedded argument keywords for hook stubs and overrides in Robot Framework enables a great deal of flexibility when reusing code, but it is very brittle. As defined, hook overrides are one typo away from being ignored during execution (no Before my system benchmark case iteration hook will be found if defined as Before my sstem benchmark case iteration), which at best will result in cryptic failures and at worst will fly by silently.

This is unacceptable from a usability POV. LAMBKIN must detect and flag these cases accordingly.

Implementation considerations

There are two different conditions that need to be flagged: extending non-existent functionality and failing to override appropriately. The former can be handled by removing (no op) stubs for extension. That's easy. The later is trickier. Removing stubs would force lots of boilerplate on users, and some overrides may simply be unused (e.g. when provided by a library) which on its own does not imply user error. One possible restriction to problem scope is to only audit keywords defined in a user-defined set of source files (or the top-level suite source file by default). A Robot Framework listener could instrument keywords behind scenes so as to ensure that all are invoked at least once. Tags may be used to explicitly disable this feature in selected keywords.

@hidmic hidmic added enhancement New feature or request needs-work Ready to be worked on labels Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request needs-work Ready to be worked on
Projects
None yet
Development

No branches or pull requests

1 participant