[Bug]: Using array of reporter types fails typescript validation: "Type 'string[]' is not assignable to type 'ReporterDescription'"

[asos-tomp]

Version

1.49.0

Steps to reproduce

Create a config with multiple reporters thus:

reporter: [["list"], ["html"]]

Expected behavior

With a javascript config, everything runs and multiple reports produced.

Actual behavior

[WebServer]     Argument of type '{ testDir: string; use: { baseURL: string; }; webServer: { reuseExistingServer: boolean; command: string; url: string; }; fullyParallel: boolean; forbidOnly: boolean; workers: number; reporter: string[][]; projects: { ...; }[]; }' is not assignable to parameter of type 'PlaywrightTestConfig<unknown, unknown>'.
[WebServer]       Types of property 'reporter' are incompatible.
[WebServer]         Type 'string[][]' is not assignable to type 'LiteralUnion<"html" | "line" | "github" | "list" | "dot" | "json" | "junit" | "null", string> | ReporterDescription[]'.
[WebServer]           Type 'string[][]' is not assignable to type 'ReporterDescription[]'.
[WebServer]             Type 'string[]' is not assignable to type 'ReporterDescription'.
[WebServer]               Type 'string[]' is not assignable to type 'readonly [string, any]'.
[WebServer]                 Target requires 2 element(s) but source may have fewer.

Additional context

No response

Environment

System:
    OS: macOS 14.7.1
    Memory: 3.91 GB / 32.00 GB
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm
    pnpm: 9.12.2 - /opt/homebrew/bin/pnpm
  Languages:
    Bash: 3.2.57 - /bin/bash

[dgozman]

@asos-tomp Could you please share your tsconfig.json? Also, I am surprised that your logs are prefixed with [WebServer] - are you running tsc inside your web server?

[asos-tomp]

@asos-tomp Could you please share your tsconfig.json? Also, I am surprised that your logs are prefixed with [WebServer] - are you running tsc inside your web server?

This is running in a github actions workflow, using an npm script thus:

playwright test --config ./config.ts

...with that config thus:

import { devices, defineConfig } from "@playwright/test";

export default defineConfig({
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  workers: process.env.CI ? 1 : undefined,
  reporter: process.env.CI ? [["blob"], ["github"]] : [["list"], ["html"]],
  projects: [
    {
      name: "chromium",
      use: { ...devices["Desktop Chrome"] },
    },
  ],
  testDir: "./examples/next/src/app/fixtures",
  use: {
    baseURL: "http://localhost:3000/fixtures/"
  },
  webServer: {
    command: "npm run build && npm run start",
    url: "http://localhost:3000",
    reuseExistingServer: !process.env.CI,
  }
});

[dgozman]

@asos-tomp Could you still share your tsconfig.json please?

your logs are prefixed with [WebServer] - are you running tsc inside your web server?

It seems like the npm run build part of your webServer command performs the type checks. This is not something we see often - usually the web server is already built before you start testing, and the command is just npm run start.

[asos-tomp]

@asos-tomp Could you still share your tsconfig.json please?

{
  "compilerOptions": {
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "baseUrl": ".",
    "target": "es2018",
    "allowJs": true,
    "skipLibCheck": true,
    "strict": false,
    "noEmit": true,
    "incremental": true,
    "module": "esnext",
    "esModuleInterop": true,
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "jsx": "preserve",
    "plugins": [
      {
        "name": "next"
      }
    ]
  },
  "include": [
    "next-env.d.ts",
    ".next/types/**/*.ts",
    "**/*.ts",
    "**/*.tsx",
    "playwright.config.ts"
  ],
  "exclude": [
    "node_modules"
  ]
}

your logs are prefixed with [WebServer] - are you running tsc inside your web server?

This is a vanilla Next.js example application, with bare bones configuration, containing the playwright config in the root. tsc may be running as part of the next setup, but by design its mostly opaque. There is a tsconfig.json as scaffolded by Next.

It seems like the npm run build part of your webServer command performs the type checks. This is not something we see often - usually the web server is already built before you start testing, and the command is just npm run start.

OK thanks.

I guess the fact the IDE is showing this error, irrespective of the contents of the webServer.command, is an indication that the types are incompatible with a documented use case? i.e. Does the principle apply that the reporters field of the config should allow string[][] as a type, as per the documentation, either way?

If I remove npm run build from the config, VSCode still shows:

 No overload matches this call.
  The last overload gave the following error.
    Argument of type '{ testDir: string; use: { baseURL: string; }; webServer: { reuseExistingServer: boolean; command: string; url: string; }; fullyParallel: boolean; forbidOnly: boolean; workers: number; reporter: string[][]; projects: { ...; }[]; }' is not assignable to parameter of type 'PlaywrightTestConfig<unknown, unknown>'.
      Types of property 'reporter' are incompatible.
        Type 'string[][]' is not assignable to type 'LiteralUnion<"github" | "list" | "html" | "dot" | "line" | "json" | "junit" | "null", string> | ReporterDescription[]'.
          Type 'string[][]' is not assignable to type 'ReporterDescription[]'.
            Type 'string[]' is not assignable to type 'ReporterDescription'.
              Type 'string[]' is not assignable to type 'readonly [string, any]'.
                Target requires 2 element(s) but source may have fewer.ts(2769)

[dgozman]

@asos-tomp Unfortunately, I still cannot reproduce the problem, even with your tsconfig.json. Given that many users have the same reporter field as you do, there's probably something different in your setup. Unless you share a self-contained reproducible, we won't be able to help you.

[asos-tomp]

I've reproduced on stackblitz here.

The type failure only shows when a config is imported from a workspace and spread to the playwright.config.ts - I'm relatively new to TypeScript so apologies if this is nothing PlayWright specific.

[dgozman]

@asos-tomp Thank you, this is a nice repro. Indeed, TypeScript would complain because you widen the type by spreading through an object with an inferred type. You can narrow the type back like this in the config.ts file:

import { ReporterDescription } from '@playwright/test';

const config = {
  reporter: [['list'], ['html']] as ReporterDescription[],
};

export default config;

Alternatively, when config is supposed to be a playwright config itself:

import { PlaywrightTestConfig } from '@playwright/test';

const config: PlaywrightTestConfig = {
  reporter: [['list'], ['html']],
};

export default config;

Let me know whether this helps.

[asos-tomp]

I have tried the first suggestion here: https://stackblitz.com/edit/github-6jrsge-txnqvw which has resolve the problem on the stackblitz, but sadly the same doesn't seem to be working in my original monorepo - but I know where to keep trying! UPDATE: I had the type on only one side of a ternary, it needed to apply to the ternary expression as a whole.

It does confuse me since string[][] is also valid for reporter, and the type I am spreading is exactly that, and no wider.

The latter suggestion also makes sense, but sadly the design includes the config being a base that is partially valid as a PlaywrightTestConfig, so doesn't look like I can type it like that.