Test Python course code examples

[honzajavorek]

As the Python course grows, it would be nice to have a way to test the code examples.

In this issue, I'd like to cover testing of regular examples which are part of the lesson. There are also exercises and I expect them to be a different beast. If we do tackle them both, then fine, but if exercises are hard to tackle, I'd focus mainly on the rest.

Also I think that regular examples should be tested under all PRs, while real-world exercises should be tested only nightly, as a separate thing (similar to Lychee).

I'm keen to implement this, but at the same time I don't want to just randomly bomb something to your infra, so I'll appreciate any kind of guidance. I suppose you were thinking about testing code examples before, or maybe it's already implemented, just not yet for Python.

Challenges

• Can we use some existing tools or do we have to build this?
• How to be explicit about which examples to test? E.g. we should be able to skip incomplete ones or real world exercises which would get blocked if executed from GitHub Actions.
• How not to add onboarding burden? People contributing to the docs shouldn't be required to install Python dependencies if they don't want to run the tests. The tests should be ran under the PRs, probably as a separate check.

I didn't do any research yet, just filing this issue for the record.

[honzajavorek]

Interesting finds:

• https://github.com/modal-labs/pytest-markdown-docs
• https://github.com/scientific-python/pytest-doctestplus
• https://mastodon.mit.edu/@eichin/113320111531591464
• https://sybil.readthedocs.io/en/latest/