Running tests using workerd

[SokratisVidros]

The new capabilities that edge runtimes bring to the table are very exciting. Modern-web software is already leveraging the new available APIs but tooling for testing is not standardized yet.

Our mission at Clerk is to provide authentication across all modern runtimes. In the JS world, this translates to an interoperable NPM package that works across all JS runtimes such as Node, Deno, and Cloudflare workers.

So far, testing the code was done manually or via alternate semi-automated solutions such as Miniflare. The purpose of this discussion is to focus on how to use workerd to run test suites faster and ensure the quality of the shipped packages.

Existing JS testing tools such as Mocha, or Jest do not work in a CF worker environment out of the box. This discussion started from #158.

[kentonv]

Straw man design: What if you could define a "test handler", like:

export default {
  async fetch(req, env, ctx) {
    // ...
  }

  async test(env) {
    await doTest1(env);
    await doTest2(env);
    return true;
  }
}

Then there was a new workerd command, workerd test, which is similar to serve, but runs the test handlers for all specified services:

workerd test config.capnp myServiceName

This runs the test handler for the specified service and exits with status zero (success) if and only if the test handler returns true. If the test throws an exception or returns anything other than true, this is considered a failure, and the process exits with a non-zero status.

Of course, you will probably want to use some sort of testing framework within this, but this gives you the mechanism to invoke such tests from the command line.

Note that you don't necessarily have to embed your test handler within your service code. You could define a separate test service in your config file, which talks to the service under test via a binding. Or maybe if you are unit testing a library, then you only have a test service, which bundles the library and makes calls to it.

Thoughts?

[mrbbot]

Nice! I really like the idea of being able to run tests directly from workerd, and definitely think running tests for Workers inside Workers is the way to go. Immediate thoughts:

1. If I wanted to call the fetch handler in the same service as the test, how would I create an ExecutionContext? Would I have to define a service binding to the current service? Would I have to define my own class for this? How do I wait for waitUntiled Promises to settle and assert their results or wait until I can assert their side effects?
2. Would we add a corresponding addEventListener("test") for service workers? Or would users be required to use a module worker for test services?
3. Potentially, may want to use export const test = .../export function test(env) {} instead as this would be easier to tree shake if tests were included in the same service. Though there's potential for name collisions here with existing exports, and as you point out, you can always define a separate service.
4. I'm not 100% sure about requiring return true to pass tests. Most JavaScript testing frameworks (which I accept this isn't) require explicit throws/assertion failures to fail tests.

Will have a think about this some more too, and come back with more comments. 👍

[kentonv]

• I think if you want to test a service, invoking its top-level fetch handler, then you would want to do it through a service binding. In this case, the test code is probably hosted in a separate service entirely, and that separate test service is configured with a service binding to the service under test.
• I don't think there'd be any direct way to observe waitUntil() tasks. However, a waitUntil() task is presumably doing some sort of background I/O, i.e. making subrequests. I would suggest arranging for those to go to a mock back-end which verifies the correct requests are made. Note that the back-end could actually loop back to the test worker. So the test worker's fetch handler ends up being used to mock subrequests. (I'd probably use a separate named entrypoint, though, to make things clearer.)
• I don't think we need Service Worker syntax support. The test can use modules syntax, while making calls to a separate service that uses service workers syntax. That said, I'm not necessarily against addEventListener("test") if there is a clear use case.
• Tests could be exported under a different entrypoint name, but I think it still has to be like export let test = { async test() { ... } }. If we export a function, things get confusing, because classes are actually syntax sugar for functions, and so at present the runtime assumes any exported function is actually a DO class. But anyway, if tests are in a separate service then tree-shaking doesn't seem that important.
• I could be convinced that the test passes as long as it doesn't throw. I put the return true in there just to be a little bit more explicit but don't feel too strongly about it. That said, I don't think this creates any incompatibility with existing test frameworks as you are going to need a shim layer either way that invokes the desired framework, and that shim can still return true at the end.

[kentonv]

I've implemented this in: #341

[SokratisVidros]

Thanks a lot @kentonv. This is great!

[kentonv]

Finally managed to land it! (There were some... struggles integrating this into our internal build pipeline.)

[SokratisVidros]

This is a great start. So far, we've tried a similar strategy. Other libraries like Jose are doing it too.

The most significant challenges we are facing have to do with using established JS testing frameworks. For example, Mocha and Jest require async IO, timers, and random generation outside the worker handler if you need to unit test Fetch API calls or async flows. We managed to enable these features in Miniflare but ideally, tests should run more easily on the worker daemon directly.

This raises the question of whether this is a workerd issue or an issue of the JS testing frameworks. However, since the tooling has been around for years, finding a middle ground sounds ideal.

[kentonv]

Hmm I'm not sure I understand what you're getting at. In my proposal, the test() handler would be allowed to do I/O. So as long as the testing framework doesn't actually run the tests until you tell it to, you can tell it to do so from inside the test() handler, and everything should work, right?

[SokratisVidros]

I see. So fetch and test handlers run with different bindings and contexts. It should work.