fix(workflow): Fix multiple context leaks in `reuseV8Context` executor #1519

mjameswh · 2024-09-12T20:52:48Z

What was changed

Fix multiple context leaks and bugs in the reuseV8Context executor:
- …by reassigning a new object to an existing shared global variable: globalThis.console = { ...globalThis.console, wfId: workflowInfo().workflowId };
- …by modifying one of Node's built in global objects: globalThis.Number.a = workflowInfo().workflowId;
- …by deleting a previously set global variable: globalThis.a = 1; await sleep(1) ; delete globalThis.a ; await sleep(1) ; globalThis.a = (globalThis.a || 0) + 1; /* globalThis.a is 2 rather than 1 */

antlai-temporal · 2024-09-17T08:31:51Z

packages/common/src/type-helpers.ts

+    map.set = frozenMapSet;
+    map.delete = frozenMapDelete;
+    map.clear = frozenMapClear;


Do you need to freeze the map itself too so that they don't override .set .delete .clear again?

It is. After replacing these methods, we continue with the normal deepfreeze logic below, which will freeze the set/delete/clear methods, then the map itself.

packages/common/src/type-helpers.ts

antlai-temporal · 2024-09-17T08:41:38Z

I'm a bit confused about what is the ultimate goal and why this is needed. If we want to eliminate all side channels this is not going to work, e.g., you could have timing side-channels, and the v8 VM is not designed to eliminate side-channels. If the goal is that other untrusted code is running in the same reused v8 context and attacking other users by modifying the context, I'm not sure we are protecting all the primordial classes (a String, Number,...) either, and in any case, that's a very hard problem...

Is this just to help prevent some mistakes, nothing to do with security?

mjameswh · 2024-09-17T15:02:14Z

Is this just to help prevent some mistakes, nothing to do with security?

No, that's really not a security thing (that's simply impossible AFAIK with Node's vm). And we don't have a goal of preventing all possible escapes out of the VM.

The reuseV8Context is a performance optimization (roughly 66% reduction on RAM usage for cached workflows, and up to 50% on CPU usage). The trick is that instead of creating a new vm context for each workflow (which is slow and allocates a large number of objects), we use a single vm context, swapping global variables in and out before and after executing each activation.

However, there's been some reports of users doing uncommon but still legitimate and reasonable things in their workflow code, that would result in context from one workflow execution leaking into another workflow execution. Depending on use cases, that may result in non-deterministic behaviors, memory leaks, or some other oddities.

For example, one user recently reported that they were mocking functions in the console object inside their workflow code to redirect messages to the workflow log. They were doing that because their workflow was calling some outside library, not meant specifically for Temporal Workflows, which would print things its own logging through console.log. Agreed, that's not the best way of doing this, but still something I would have expected to be supported by our sandbox. Their code looked something like this:

async function myWorkflow() {
  const workflowId = workflowInfo().workflowId;
  globalThis.console = {
    ...console,
    log: (message) => {
      log(`${workflowId}: message`)
    }
  }
  // ...

The problem here is that previously, when leaving a workflow's execution context, the Reusable VM's would not touch any global variable that existed right after start. In this case, that means that it would leave the console object in place, even though it was no longer the original, pristine console object! That would result in them seeing log messages with an incorrect workflowId.

Once you know it, it's quite easy to fix their code to avoid this issue (they did), but it's also an opportunity for us to make our engine more reliable.

mjameswh · 2024-09-17T15:13:46Z

There's also one test, "Shared global state is frozen", that existed before this PR that was actually not testing what we thought it was testing. That is, the test was asserting attempting to add a custom property to some global object from Node's default vm context would be rejected.

However, it was doing so by adding something to the setTimetout function, but it turns out that's insufficient as the globalThis.setTimeout is overridden by the SDK itself, resulting in it being present in the globals object exposed outside of the vm, where global variables that would not been modified inside the vm would not appear in the globals object (seems to be some kind of internal optimization done by Node/V8).

This PR fixes that bug; the test is now asserting the proper behavior against the Array global function instead.

antlai-temporal · 2024-09-17T23:18:13Z

packages/common/src/type-helpers.ts

      } catch (err) {
+        if (typeof value !== 'object' || value.constructor.name !== 'Uint8Array') {


You are checking that if (value && typeof value === 'object') { already
do you need to check again because deepFreeze could change it?
Also, a comment why Uint8Array is special may help

antlai-temporal · 2024-09-17T23:37:57Z

packages/test/src/test-isolation.ts

+    t.deepEqual(res1[0], res1[1]);
+    t.deepEqual(res1[0], res1[2]);
+    t.deepEqual(res2[0], res2[1]);
+    t.deepEqual(res2[0], res2[2]);


Do we want to check res1[0] !== res2[0] for completeness

antlai-temporal · 2024-09-18T00:52:37Z

packages/worker/src/workflow/reusable-vm.ts

+    );
+    for (const [k, v] of this.pristine.entries()) {
+      if (k !== 'globalThis') {
+        deepFreeze(v.value);


Do we need to deepFreeze the key k too?

antlai-temporal · 2024-09-18T01:49:17Z

packages/worker/src/workflow/reusable-vm.ts

+    (globals as any).__pristine = this.pristine;
+    // The V8 context is really composed of two distinct objects: the `globals` object defined above
+    // on the outside, and another internal object to which we only have access from the inside, which
+    // defines the built-in global properties. We need


fix comment

antlai-temporal · 2024-09-18T02:32:31Z

packages/worker/src/workflow/reusable-vm.ts

+            for (const [pname, pdesc] of bag.entries()) {
+              Object.defineProperty(context, pname, pdesc);
+            }


I'm a bit confused aboutbag initialization, in particular if the first time we call
return vm.runInContext(TEMPORAL.api.${fn}``
the bag would be empty, and whether that matters.
I see the bag is filled inside finally{... after `bag.clear()` but not sure about the first call...

bergundy · 2024-09-19T13:26:59Z

packages/common/src/type-helpers.ts

      } catch (err) {
+        if (typeof value !== 'object' || value.constructor.name !== 'Uint8Array') {
+          console.log(`Failed to freeze ${path}.${name}`, err);


Please remove this before merging.

bergundy · 2024-09-19T13:28:59Z

packages/common/src/type-helpers.ts

-export function deepFreeze<T>(object: T): T {
+// This implementation is based on https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze.
+// Note that there are limits to this approach, as traversing using getOwnPropertyXxx doesn't allow
+// reaching variables defined using internal scopes. This implementation specifically look for and deal


Suggested change

// reaching variables defined using internal scopes. This implementation specifically look for and deal

// reaching variables defined using internal scopes. This implementation specifically looks for and deals

Why is dealing with these types important? Have you seen any shared maps and sets that are exposed to users?

bergundy · 2024-09-19T13:32:12Z

packages/test/src/test-isolation.ts

+  return [middle, final];
+}
+
+test('Deep Freeze reaches Map values', async (t) => {


Do we actually freeze the map? We're not supposed to freeze user's modules.

bergundy · 2024-09-19T13:33:34Z

packages/test/src/test-isolation.ts

-    (setTimeout as any).a = 1;
+    (Array as any).a = 1;


Why did you change this?

bergundy · 2024-09-19T13:35:48Z

packages/test/src/test-isolation.ts

+      env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }),
+      env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }),


This may be racy in CI and may not actually provide the test coverage you want. There's nothing guaranteeing that the workflows will run concurrently.

bergundy · 2024-09-19T13:36:50Z

packages/worker/src/workflow/reusable-vm.ts

-   */
-  readonly contextKeysToPreserve: Set<string>;
+  private _context?: vm.Context;
+  private pristine?: Map<string, PropertyDescriptor>;


bergundy · 2024-09-19T13:42:19Z

packages/worker/src/workflow/reusable-vm.ts

    const workflowModule: WorkflowModule = new Proxy(
      {},
      {
        get(_: any, fn: string) {
          return (...args: any[]) => {
-            Object.assign(context, bag);
+            for (const [pname, pdesc] of bag.entries()) {


nit: would have liked to prevent constructing the entries array for every time we call into the VM.

bergundy · 2024-09-19T13:45:32Z

packages/worker/src/workflow/reusable-vm.ts

+              // Looks like Node/V8 is not properly syncing deletion of keys on the outter context
+              // object to the inner globalThis object. Hence, we delete them from inside the context.
+              context.__TEMPORAL_ARGS__ = keysToCleanup;
+              vm.runInContext(


This will add a bit of overhead for every time we enter the context, I would try to find an alternative way to do this.

mjameswh requested a review from a team as a code owner September 12, 2024 20:52

mjameswh force-pushed the fix-reusev8context-reassign-bug branch from 44f477c to 04810fb Compare September 16, 2024 21:22

mjameswh changed the title ~~[WIP] fix(workflow): Fix multiple context leaks in reuseV8Context executor~~ fix(workflow): Fix multiple context leaks in reuseV8Context executor Sep 16, 2024

antlai-temporal reviewed Sep 17, 2024

View reviewed changes

packages/common/src/type-helpers.ts Outdated Show resolved Hide resolved

antlai-temporal reviewed Sep 17, 2024

View reviewed changes

antlai-temporal reviewed Sep 18, 2024

View reviewed changes

antlai-temporal approved these changes Sep 18, 2024

View reviewed changes

antlai-temporal reviewed Sep 18, 2024

View reviewed changes

bergundy reviewed Sep 19, 2024

View reviewed changes

mjameswh added 4 commits November 22, 2024 13:41

wip - fix-reusev8context-reassign-bug

2c00da1

wip

fc22b2e

Fix remaining issues

2a36b38

wip

0bd389b

mjameswh force-pushed the fix-reusev8context-reassign-bug branch from 04810fb to 0bd389b Compare November 22, 2024 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(workflow): Fix multiple context leaks in `reuseV8Context` executor #1519

fix(workflow): Fix multiple context leaks in `reuseV8Context` executor #1519

mjameswh commented Sep 12, 2024

antlai-temporal Sep 17, 2024

mjameswh Sep 18, 2024

antlai-temporal commented Sep 17, 2024

mjameswh commented Sep 17, 2024

mjameswh commented Sep 17, 2024

antlai-temporal Sep 17, 2024

antlai-temporal Sep 17, 2024

antlai-temporal Sep 18, 2024

antlai-temporal Sep 18, 2024

antlai-temporal Sep 18, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

bergundy Sep 19, 2024

		} catch (err) {
		if (typeof value !== 'object' \|\| value.constructor.name !== 'Uint8Array') {

	// reaching variables defined using internal scopes. This implementation specifically look for and deal
	// reaching variables defined using internal scopes. This implementation specifically looks for and deals

		env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }),
		env.client.workflow.execute(sharedGlobalReassignment, { taskQueue, workflowId: randomUUID() }),

fix(workflow): Fix multiple context leaks in reuseV8Context executor #1519

Are you sure you want to change the base?

fix(workflow): Fix multiple context leaks in reuseV8Context executor #1519

Conversation

mjameswh commented Sep 12, 2024

What was changed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antlai-temporal commented Sep 17, 2024

mjameswh commented Sep 17, 2024

mjameswh commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fix(workflow): Fix multiple context leaks in `reuseV8Context` executor #1519

fix(workflow): Fix multiple context leaks in `reuseV8Context` executor #1519