#50 — Tech debt: parallelize the E2E CI job across Playwright projects #50

New issue

Closed

opened 2026-08-18 13:13:16 +02:00 by lena · 2 comments

lena commented

2026-08-18 13:13:16 +02:00

(Migrated from git.butzei.de)

Tech debt: parallelize the E2E CI job across Playwright projects

Reported by: QA Agent (found during #46 code review)
Priority: Could — no correctness impact, but a growing tax on every push/PR's feedback loop
Date: 2026-07-11

Problem

#46 added two Playwright projects (Desktop HD, Mobile Chrome (Pixel 8a)) alongside the
existing chromium/firefox. The .gitea/workflows/ci.yml e2e job still runs all four
projects through a single npx playwright test invocation on one runner, with
workers: isCI ? 1 : undefined (serialized — see playwright.config.ts's comment: heavy
multi-context specs like invitation.spec.ts/transfer-ownership.spec.ts caused intermittent
timeouts under concurrent load against the shared backend). Verified via npx playwright test --list: the suite went from 48 serialized test-runs (2 projects × 24 specs) to 83 (4 projects,
weighted by scope) — a ~73% growth in the e2e job's wall-clock time, with the heaviest specs
now running 3× instead of 2× (chromium, firefox, Desktop HD all execute the full desktop suite).

Proposal

Split the single e2e job into a CI matrix over
project: [chromium, firefox, 'Desktop HD', 'Mobile Chrome (Pixel 8a)'], each with its own
postgres/redis service containers and npx playwright test --project="${{ matrix.project }}".
This keeps workers: 1 within each project (preserving the documented anti-contention
property) while letting the 4 projects run concurrently across runners, turning the wall-clock
cost from the sum of all test-runs into roughly the cost of the single slowest project instead
of their total.

Acceptance criteria (PO-finalized 2026-07-12)

e2e CI job runs each Playwright project (chromium, firefox, Desktop HD,
Mobile Chrome (Pixel 8a)) in its own matrix leg with its own postgres/redis service
containers
Total E2E wall-clock time in CI drops materially versus the pre-matrix single-job baseline
— this is the story's actual purpose; confirm on the first real CI run post-merge
One leg's failure does not cancel the other legs (fail-fast: false), so a single flaky
project doesn't hide results from the other three
workers: 1 (or equivalent per-leg serialization) is preserved within each project — no
reintroduction of the concurrent-context contention #46's config comment documents
Failure artifacts (playwright-report, e2e-screenshots) still upload per leg, with a
unique artifact name per leg (project names contain spaces/parens, not valid as-is in an
artifact name) so a failure is traceable to the right project
docker job's needs: [backend, frontend] is unaffected (it does not depend on e2e
today and must not start depending on it as a side effect of this change)

Blockers

None — purely a CI workflow change, independent of any feature work.

# Tech debt: parallelize the E2E CI job across Playwright projects **Reported by:** QA Agent (found during `#46` code review) **Priority:** Could — no correctness impact, but a growing tax on every push/PR's feedback loop **Date:** 2026-07-11 ## Problem `#46` added two Playwright projects (`Desktop HD`, `Mobile Chrome (Pixel 8a)`) alongside the existing `chromium`/`firefox`. The `.gitea/workflows/ci.yml` `e2e` job still runs all four projects through a single `npx playwright test` invocation on one runner, with `workers: isCI ? 1 : undefined` (serialized — see `playwright.config.ts`'s comment: heavy multi-context specs like `invitation.spec.ts`/`transfer-ownership.spec.ts` caused intermittent timeouts under concurrent load against the shared backend). Verified via `npx playwright test --list`: the suite went from 48 serialized test-runs (2 projects × 24 specs) to 83 (4 projects, weighted by scope) — a ~73% growth in the `e2e` job's wall-clock time, with the heaviest specs now running 3× instead of 2× (chromium, firefox, Desktop HD all execute the full desktop suite). ## Proposal Split the single `e2e` job into a CI matrix over `project: [chromium, firefox, 'Desktop HD', 'Mobile Chrome (Pixel 8a)']`, each with its own `postgres`/`redis` service containers and `npx playwright test --project="${{ matrix.project }}"`. This keeps `workers: 1` *within* each project (preserving the documented anti-contention property) while letting the 4 projects run concurrently across runners, turning the wall-clock cost from the sum of all test-runs into roughly the cost of the single slowest project instead of their total. ## Acceptance criteria (PO-finalized 2026-07-12) - [ ] `e2e` CI job runs each Playwright project (`chromium`, `firefox`, `Desktop HD`, `Mobile Chrome (Pixel 8a)`) in its own matrix leg with its own `postgres`/`redis` service containers - [ ] Total E2E wall-clock time in CI drops materially versus the pre-matrix single-job baseline — this is the story's actual purpose; confirm on the first real CI run post-merge - [ ] One leg's failure does not cancel the other legs (`fail-fast: false`), so a single flaky project doesn't hide results from the other three - [ ] `workers: 1` (or equivalent per-leg serialization) is preserved within each project — no reintroduction of the concurrent-context contention `#46`'s config comment documents - [ ] Failure artifacts (`playwright-report`, `e2e-screenshots`) still upload per leg, with a unique artifact name per leg (project names contain spaces/parens, not valid as-is in an artifact name) so a failure is traceable to the right project - [ ] `docker` job's `needs: [backend, frontend]` is unaffected (it does not depend on `e2e` today and must not start depending on it as a side effect of this change) ## Blockers None — purely a CI workflow change, independent of any feature work.

lena commented

2026-08-18 13:13:16 +02:00

(Migrated from git.butzei.de)

design (50_e2e_ci_matrix_parallelization_design.md)

Architect design — `#50` E2E CI matrix parallelization

Design

Replace the single e2e job in .gitea/workflows/ci.yml with a strategy.matrix over the four
Playwright project names, fail-fast: false. Gitea Actions (GitHub Actions-compatible) spins up
one independent job instance per matrix leg, each getting its own services: containers — so
postgres/redis isolation is free, not something we have to hand-roll.

Each leg runs npx playwright test --project="${{ matrix.project }}" instead of the bare
npx playwright test. playwright.config.ts needs no change: workers: isCI ? 1 : undefined
already serializes within a single project's run, and a matrix leg only ever runs one project, so
the anti-contention property from #46 is preserved automatically — the contention that comment
guards against was cross-spec-file contention within one project's shared backend, not
cross-project (each leg gets its own backend + Postgres + Redis via its own webServer/services
block, so there's no shared state between legs to contend over).

Artifact naming

Project names (Desktop HD, Mobile Chrome (Pixel 8a)) contain spaces and parentheses, which are
not safe verbatim in actions/upload-artifact's name: (and would collide across legs if left
unparameterized — v3 upload-artifact does not append a leg-unique suffix on its own the way v4
does). Introduce matrix.project_slug alongside matrix.project in the matrix definition
(explicit include list, not a computed slug, since Gitea Actions' expression language has no
string-replace function to derive one from matrix.project inline) and use it in both artifact
name: fields: playwright-report-${{ matrix.project_slug }} /
e2e-screenshots-${{ matrix.project_slug }}.

Matrix definition

strategy:
  fail-fast: false
  matrix:
    include:
      - project: chromium
        project_slug: chromium
      - project: firefox
        project_slug: firefox
      - project: 'Desktop HD'
        project_slug: desktop-hd
      - project: 'Mobile Chrome (Pixel 8a)'
        project_slug: mobile-chrome-pixel-8a

What doesn't change

docker job's needs: [backend, frontend] — it never depended on e2e, so nothing to do here;
called out in the acceptance criteria only because matrix-izing e2e is exactly the kind of
change where a stray needs: [e2e] could get copy-pasted in by habit.
playwright.config.ts — no edits. The per-project workers: 1 behavior this story preserves is
already a property of the config, not the workflow.
Browser install step (npx playwright install chromium firefox --with-deps) installs both
desktop browsers regardless of which project a leg runs — cheap enough (~seconds vs. minutes for
the test run itself) that splitting it per-project isn't worth the added matrix complexity.

Known risks (unconfirmable in the sandbox, watch the first real CI run)

Fixed webServer ports across concurrent legs. playwright.config.ts's webServer binds
fixed host ports (backend :5000, frontend :5173) with reuseExistingServer: !isCI (false in
CI). Pre-#50 only one e2e job instance ever ran at a time, so this was never a conflict. Post-#50,
up to 4 matrix legs bind those same fixed ports concurrently. This relies on each matrix leg
getting a fully isolated job runner (its own container/network namespace) — the standard
Actions/act_runner execution model, and already implicitly relied on today by backend and
frontend running as separate concurrent jobs. Not re-architected to use dynamic per-leg ports
here: that's real added complexity to defend against a risk this repo's runner model should
already rule out by default, and the fix (if the assumption turns out wrong) is well understood
(parameterize the port from matrix.project_slug) — cheaper to apply once actually observed than
to build speculatively. If a leg ever fails CI with "port already in use," that's the signal.
4x concurrent backend cold-start. Each leg's webServer independently runs dotnet run --project CqsTodo.WebApi — a Debug-mode cold build plus a full EF migration run against a brand
new Postgres container (the 120s timeout in playwright.config.ts exists because this was
already observed to take close to 60s serially). Post-#50 this happens 4x concurrently instead of
once. This is the accepted cost side of the trade this story makes: more total compute in exchange
for the wall-clock win the first acceptance criterion measures — not a defect, but not previously
written down, so noting it here for whoever reviews the actual CI resource usage after this lands.

Security pre-review

No new secrets, no new network exposure, no change to what code runs — this is a CI topology
change only (same test code, same service images, same connection strings, now split across
parallel jobs instead of one sequential job). REGISTRY_TOKEN usage in the docker job is
untouched. Approved without further action.

**design** (`50_e2e_ci_matrix_parallelization_design.md`) # Architect design — `#50` E2E CI matrix parallelization ## Design Replace the single `e2e` job in `.gitea/workflows/ci.yml` with a `strategy.matrix` over the four Playwright project names, `fail-fast: false`. Gitea Actions (GitHub Actions-compatible) spins up one independent job instance per matrix leg, each getting its own `services:` containers — so `postgres`/`redis` isolation is free, not something we have to hand-roll. Each leg runs `npx playwright test --project="${{ matrix.project }}"` instead of the bare `npx playwright test`. `playwright.config.ts` needs no change: `workers: isCI ? 1 : undefined` already serializes within a single project's run, and a matrix leg only ever runs one project, so the anti-contention property from `#46` is preserved automatically — the contention that comment guards against was *cross-spec-file* contention within one project's shared backend, not cross-project (each leg gets its own backend + Postgres + Redis via its own `webServer`/services block, so there's no shared state between legs to contend over). ### Artifact naming Project names (`Desktop HD`, `Mobile Chrome (Pixel 8a)`) contain spaces and parentheses, which are not safe verbatim in `actions/upload-artifact`'s `name:` (and would collide across legs if left unparameterized — v3 upload-artifact does not append a leg-unique suffix on its own the way v4 does). Introduce `matrix.project_slug` alongside `matrix.project` in the matrix definition (explicit include list, not a computed slug, since Gitea Actions' expression language has no string-replace function to derive one from `matrix.project` inline) and use it in both artifact `name:` fields: `playwright-report-${{ matrix.project_slug }}` / `e2e-screenshots-${{ matrix.project_slug }}`. ### Matrix definition ```yaml strategy: fail-fast: false matrix: include: - project: chromium project_slug: chromium - project: firefox project_slug: firefox - project: 'Desktop HD' project_slug: desktop-hd - project: 'Mobile Chrome (Pixel 8a)' project_slug: mobile-chrome-pixel-8a ``` ### What doesn't change - `docker` job's `needs: [backend, frontend]` — it never depended on `e2e`, so nothing to do here; called out in the acceptance criteria only because matrix-izing `e2e` is exactly the kind of change where a stray `needs: [e2e]` could get copy-pasted in by habit. - `playwright.config.ts` — no edits. The per-project `workers: 1` behavior this story preserves is already a property of the config, not the workflow. - Browser install step (`npx playwright install chromium firefox --with-deps`) installs both desktop browsers regardless of which project a leg runs — cheap enough (~seconds vs. minutes for the test run itself) that splitting it per-project isn't worth the added matrix complexity. ### Known risks (unconfirmable in the sandbox, watch the first real CI run) - **Fixed webServer ports across concurrent legs.** `playwright.config.ts`'s `webServer` binds fixed host ports (backend `:5000`, frontend `:5173`) with `reuseExistingServer: !isCI` (false in CI). Pre-`#50` only one `e2e` job instance ever ran at a time, so this was never a conflict. Post-`#50`, up to 4 matrix legs bind those same fixed ports concurrently. This relies on each matrix leg getting a fully isolated job runner (its own container/network namespace) — the standard Actions/act_runner execution model, and already implicitly relied on today by `backend` and `frontend` running as separate concurrent jobs. Not re-architected to use dynamic per-leg ports here: that's real added complexity to defend against a risk this repo's runner model should already rule out by default, and the fix (if the assumption turns out wrong) is well understood (parameterize the port from `matrix.project_slug`) — cheaper to apply once actually observed than to build speculatively. If a leg ever fails CI with "port already in use," that's the signal. - **4x concurrent backend cold-start.** Each leg's `webServer` independently runs `dotnet run --project CqsTodo.WebApi` — a Debug-mode cold build plus a full EF migration run against a brand new Postgres container (the 120s timeout in `playwright.config.ts` exists because this was already observed to take close to 60s serially). Post-`#50` this happens 4x concurrently instead of once. This is the accepted cost side of the trade this story makes: more total compute in exchange for the wall-clock win the first acceptance criterion measures — not a defect, but not previously written down, so noting it here for whoever reviews the actual CI resource usage after this lands. ## Security pre-review No new secrets, no new network exposure, no change to what code runs — this is a CI topology change only (same test code, same service images, same connection strings, now split across parallel jobs instead of one sequential job). `REGISTRY_TOKEN` usage in the `docker` job is untouched. Approved without further action.

lena commented

2026-08-18 13:13:16 +02:00

(Migrated from git.butzei.de)

qa (50_e2e_ci_matrix_parallelization_qa.md)

QA notes — `#50` E2E CI matrix parallelization

What was verified locally

The sandbox has no Docker/Postgres/Redis (documented limitation — see
ai/roles/memory/06_qa_agent_memory.md), so the actual matrix job execution can only be confirmed
on the real Gitea CI runner. What's verifiable locally:

.gitea/workflows/ci.yml parses as valid YAML (checked via js-yaml) and the jobs mapping
still has the expected four top-level jobs (backend, frontend, e2e, docker).
Each of the four --project values used in the new matrix (chromium, firefox,
'Desktop HD', 'Mobile Chrome (Pixel 8a)') was run against npx playwright test --project=... --list
and correctly filtered to that project's own spec subset (25 / 25 / 27 / 6 tests respectively) —
confirms the quoting and project names in the workflow's matrix.include exactly match
playwright.config.ts's projects[].name values (a mismatch here would silently run 0 tests for
a leg instead of erroring).
docker job's needs: [backend, frontend] is unchanged (diff confirms only the e2e job block
was touched) — acceptance criterion "docker job unaffected" holds by inspection.
Artifact names (playwright-report-${{ matrix.project_slug }},
e2e-screenshots-${{ matrix.project_slug }}) are now unique per leg and contain only
alphanumerics/hyphens, avoiding the space/parenthesis characters in the raw project names that
upload-artifact@v3 would otherwise receive.

Confirmed on real CI (run 272, commit `19d7dad`, 2026-07-12)

All 7 jobs green: Frontend 1m47s, Backend 3m12s, Docker 9s, and all four E2E legs —
Mobile Chrome (Pixel 8a) 3m42s, Desktop HD 4m24s, chromium 4m42s, firefox 4m54s.

No port collision. All 4 legs ran (2 pairs, overlapping) without any "port already in use"
failure — the fixed-port risk flagged in the design doc's "Known risks" section did not
materialize. Per-leg runner isolation holds as assumed.
workers: 1 contention-free, as expected (unchanged config, already proven pre-#50).
Wall-clock did improve, but not to the "cost of the single slowest project" the story's
Proposal section hoped for — the runner pool only executed 2 E2E legs concurrently, not 4
(observed via polling: Desktop HD + Mobile Chrome ran first, chromium + firefox started
only once those finished). E2E wall-clock was therefore roughly two back-to-back waves
(~4m24s + ~4m54s ≈ 9m18s) rather than one (~4m54s, the slowest single leg). Still a real
improvement over the pre-#50 serial baseline (all 4 projects' full test-runs summed in one job,
no concurrency at all — Problem section estimated a ~73% wall-clock growth from adding the 3rd
and 4th projects), just smaller than the design doc's best case, which implicitly assumed
unlimited concurrent runners. Not a defect in this story's approach — worth a follow-up note in
case the runner pool's concurrency limit is itself worth raising as separate infra work, out of
this story's scope.

Security final review

No new attack surface: same test code, same images, same secrets (none added), only the job
topology changed. Approved.

**qa** (`50_e2e_ci_matrix_parallelization_qa.md`) # QA notes — `#50` E2E CI matrix parallelization ## What was verified locally The sandbox has no Docker/Postgres/Redis (documented limitation — see `ai/roles/memory/06_qa_agent_memory.md`), so the actual matrix job execution can only be confirmed on the real Gitea CI runner. What's verifiable locally: - `.gitea/workflows/ci.yml` parses as valid YAML (checked via `js-yaml`) and the `jobs` mapping still has the expected four top-level jobs (`backend`, `frontend`, `e2e`, `docker`). - Each of the four `--project` values used in the new matrix (`chromium`, `firefox`, `'Desktop HD'`, `'Mobile Chrome (Pixel 8a)'`) was run against `npx playwright test --project=... --list` and correctly filtered to that project's own spec subset (25 / 25 / 27 / 6 tests respectively) — confirms the quoting and project names in the workflow's `matrix.include` exactly match `playwright.config.ts`'s `projects[].name` values (a mismatch here would silently run 0 tests for a leg instead of erroring). - `docker` job's `needs: [backend, frontend]` is unchanged (diff confirms only the `e2e` job block was touched) — acceptance criterion "docker job unaffected" holds by inspection. - Artifact names (`playwright-report-${{ matrix.project_slug }}`, `e2e-screenshots-${{ matrix.project_slug }}`) are now unique per leg and contain only alphanumerics/hyphens, avoiding the space/parenthesis characters in the raw project names that `upload-artifact@v3` would otherwise receive. ## Confirmed on real CI (run 272, commit 19d7dad, 2026-07-12) All 7 jobs green: `Frontend` 1m47s, `Backend` 3m12s, `Docker` 9s, and all four E2E legs — `Mobile Chrome (Pixel 8a)` 3m42s, `Desktop HD` 4m24s, `chromium` 4m42s, `firefox` 4m54s. - **No port collision.** All 4 legs ran (2 pairs, overlapping) without any "port already in use" failure — the fixed-port risk flagged in the design doc's "Known risks" section did not materialize. Per-leg runner isolation holds as assumed. - **`workers: 1` contention-free**, as expected (unchanged config, already proven pre-`#50`). - **Wall-clock did improve, but not to the "cost of the single slowest project" the story's Proposal section hoped for** — the runner pool only executed 2 E2E legs concurrently, not 4 (observed via polling: `Desktop HD` + `Mobile Chrome` ran first, `chromium` + `firefox` started only once those finished). E2E wall-clock was therefore roughly two back-to-back waves (~4m24s + ~4m54s ≈ 9m18s) rather than one (~4m54s, the slowest single leg). Still a real improvement over the pre-`#50` serial baseline (all 4 projects' full test-runs summed in one job, no concurrency at all — Problem section estimated a ~73% wall-clock growth from adding the 3rd and 4th projects), just smaller than the design doc's best case, which implicitly assumed unlimited concurrent runners. Not a defect in this story's approach — worth a follow-up note in case the runner pool's concurrency limit is itself worth raising as separate infra work, out of this story's scope. ## Security final review No new attack surface: same test code, same images, same secrets (none added), only the job topology changed. Approved.