256 lines
8.9 KiB
Markdown
256 lines
8.9 KiB
Markdown
## App E2E Smoke Suite (CI)
|
||
|
||
Implement a small set of high-signal, low-flake Playwright tests to run in CI.
|
||
|
||
These tests are intended to catch regressions in the “core shell” of the app (navigation, dialogs, prompt UX, file viewer, terminal), without relying on model output.
|
||
|
||
---
|
||
|
||
### Summary
|
||
|
||
Add 6 smoke tests to `packages/app/e2e/`:
|
||
|
||
- Settings dialog: open, switch tabs, close
|
||
- Prompt slash command: `/open` opens the file picker dialog
|
||
- Prompt @mention: `@<file>` inserts a file pill token
|
||
- Model picker: open model selection and choose a model
|
||
- File viewer: open a known file and assert contents render
|
||
- Terminal: open terminal, verify Ghostty mounts, create a second terminal
|
||
|
||
---
|
||
|
||
### Progress
|
||
|
||
- [x] 1. Settings dialog open / switch / close (`packages/app/e2e/settings.spec.ts`)
|
||
- [x] 2. Prompt slash command path: `/open` opens file picker (`packages/app/e2e/prompt-slash-open.spec.ts`)
|
||
- [x] 3. Prompt @mention inserts a file pill token (`packages/app/e2e/prompt-mention.spec.ts`)
|
||
- [x] 4. Model selection UI works end-to-end (`packages/app/e2e/model-picker.spec.ts`)
|
||
- [x] 5. File viewer renders real file content (`packages/app/e2e/file-viewer.spec.ts`)
|
||
- [x] 8. Terminal init + create new terminal (`packages/app/e2e/terminal-init.spec.ts`)
|
||
|
||
---
|
||
|
||
### Goals
|
||
|
||
- Tests run reliably in CI using the existing local runner (`packages/app/script/e2e-local.ts`).
|
||
- Cover “wiring” regressions across UI + backend APIs:
|
||
- dialogs + command routing
|
||
- prompt contenteditable parsing
|
||
- file search + file read + code viewer render
|
||
- terminal open + pty creation + Ghostty mount
|
||
- Avoid assertions that depend on LLM output.
|
||
- Keep runtime low (these should be “smoke”, not full workflows).
|
||
|
||
---
|
||
|
||
### Non-goals
|
||
|
||
- Verifying complex model behavior, streaming correctness, or tool call semantics.
|
||
- Testing provider auth flows (CI has no secrets).
|
||
- Testing share, MCP, or LSP download flows (disabled in the e2e runner).
|
||
|
||
---
|
||
|
||
### Current State
|
||
|
||
Existing tests in `packages/app/e2e/` already cover:
|
||
|
||
- Home renders + server picker opens
|
||
- Directory route redirects to `/session`
|
||
- Sidebar collapse/expand
|
||
- Command palette opens/closes
|
||
- Basic session open + prompt input + (optional) prompt/reply flow
|
||
- File open via palette (but shallow assertion: tab exists)
|
||
- Terminal panel toggles (but doesn’t assert Ghostty mounted)
|
||
- Context panel open
|
||
|
||
We want to add a focused smoke layer that increases coverage of the most regression-prone UI paths.
|
||
|
||
---
|
||
|
||
### Proposed Tests
|
||
|
||
All tests should use the shared fixtures in:
|
||
|
||
- `packages/app/e2e/fixtures.ts` (for `sdk`, `directory`, `gotoSession`)
|
||
- `packages/app/e2e/utils.ts` (for `modKey`, `promptSelector`, `terminalToggleKey`)
|
||
|
||
Prefer creating new spec files rather than overloading existing ones, so it’s easy to run these tests as a group via grep.
|
||
|
||
Suggested file layout:
|
||
|
||
- `packages/app/e2e/settings.spec.ts`
|
||
- `packages/app/e2e/prompt-slash-open.spec.ts`
|
||
- `packages/app/e2e/prompt-mention.spec.ts`
|
||
- `packages/app/e2e/model-picker.spec.ts`
|
||
- `packages/app/e2e/file-viewer.spec.ts`
|
||
- `packages/app/e2e/terminal-init.spec.ts`
|
||
|
||
Name each test with a “smoke” prefix so CI can run only this suite if needed.
|
||
|
||
#### 1) Settings dialog open / switch / close
|
||
|
||
Purpose: catch regressions in dialog infra, settings rendering, tabs.
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Open settings via keybind (preferred for stability): `await page.keyboard.press(`${modKey}+Comma`)`.
|
||
3. Assert dialog visible (`page.getByRole('dialog')`).
|
||
4. Click the "Shortcuts" tab (role `tab`, name "Shortcuts").
|
||
5. Assert shortcuts view renders (e.g. the search field placeholder or reset button exists).
|
||
6. Close with `Escape` and assert dialog removed.
|
||
|
||
Notes:
|
||
|
||
- If `Meta+Comma` / `Control+Comma` key name is flaky, fall back to clicking the sidebar settings icon.
|
||
- Favor role-based selectors over brittle class selectors.
|
||
- If `Escape` doesn’t dismiss reliably (tooltips can intercept), fall back to clicking the dialog overlay.
|
||
|
||
Implementation: `packages/app/e2e/settings.spec.ts`
|
||
|
||
Acceptance criteria:
|
||
|
||
- Settings dialog opens reliably.
|
||
- Switching to Shortcuts tab works.
|
||
- Escape closes the dialog.
|
||
|
||
#### 2) Prompt slash command path: `/open` opens file picker
|
||
|
||
Purpose: validate contenteditable parsing + slash popover + builtin command dispatch (distinct from `mod+p`).
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Click prompt (`promptSelector`).
|
||
3. Type `/open`.
|
||
4. Press `Enter` (while slash popover is active).
|
||
5. Assert a dialog appears and contains a textbox (the file picker search input).
|
||
6. Close dialog with `Escape`.
|
||
|
||
Acceptance criteria:
|
||
|
||
- `/open` triggers `file.open` and opens `DialogSelectFile`.
|
||
|
||
#### 3) Prompt @mention inserts a file pill token
|
||
|
||
Purpose: validate the most fragile prompt behavior: structured tokens inside contenteditable.
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Focus the prompt.
|
||
3. Type `@packages/app/package.json`.
|
||
4. Press `Tab` to accept the active @mention suggestion.
|
||
5. Assert a pill element is inserted:
|
||
- `page.locator('[data-component="prompt-input"] [data-type="file"][data-path="packages/app/package.json"]')` exists.
|
||
|
||
Acceptance criteria:
|
||
|
||
- A file pill is inserted and has the expected `data-*` attributes.
|
||
- Prompt editor remains interactable (e.g. typing a trailing space works).
|
||
|
||
#### 4) Model selection UI works end-to-end
|
||
|
||
Purpose: validate model list rendering, selection wiring, and prompt footer updating.
|
||
|
||
Implementation approach:
|
||
|
||
- Use `/model` to open the model selection dialog (builtin command).
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Focus prompt, type `/model`, press `Enter`.
|
||
3. In the model dialog, pick a visible model that is not the current selection (if available).
|
||
4. Use the search field to filter to that model (use its id from the list item's `data-key` to avoid time-based model visibility drift).
|
||
5. Select the filtered model.
|
||
6. Assert dialog closed.
|
||
7. Assert the prompt footer now shows the chosen model name.
|
||
|
||
Acceptance criteria:
|
||
|
||
- A model can be selected without requiring provider auth.
|
||
- The prompt footer reflects the new selection.
|
||
|
||
#### 5) File viewer renders real file content
|
||
|
||
Purpose: ensure file search + open + file.read + code viewer render all work.
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Open file picker (either `mod+p` or `/open`).
|
||
3. Search for `packages/app/package.json`.
|
||
4. Click the matching file result.
|
||
5. Ensure the new file tab is active (click the `package.json` tab if needed so the viewer mounts).
|
||
6. Assert the code viewer contains a known substring:
|
||
- `"name": "@opencode-ai/app"`.
|
||
7. Optionally assert the file tab is active and visible.
|
||
|
||
Acceptance criteria:
|
||
|
||
- Code view shows expected content (not just “tab exists”).
|
||
|
||
#### 8) Terminal init + create new terminal
|
||
|
||
Purpose: ensure terminal isn’t only “visible”, but actually mounted and functional.
|
||
|
||
Steps:
|
||
|
||
1. `await gotoSession()`.
|
||
2. Open terminal with `terminalToggleKey` (currently `Control+Backquote`).
|
||
3. Assert terminal container exists and is visible: `[data-component="terminal"]`.
|
||
4. Assert Ghostty textarea exists: `[data-component="terminal"] textarea`.
|
||
5. Create a new terminal via keybind (`terminal.new` is `ctrl+alt+t`).
|
||
6. Assert terminal tab count increases to 2.
|
||
|
||
Acceptance criteria:
|
||
|
||
- Ghostty mounts (textarea present).
|
||
- Creating a new terminal results in a second tab.
|
||
|
||
---
|
||
|
||
### CI Stability + Flake Avoidance
|
||
|
||
These tests run with `fullyParallel: true` in `packages/app/playwright.config.ts`. Keep them isolated and deterministic.
|
||
|
||
- Avoid ordering-based assertions: never assume a “first” session/project/file is stable unless you filtered by unique text.
|
||
- Prefer deterministic targets:
|
||
- use `packages/app/package.json` rather than bare `package.json` (multiple hits possible)
|
||
- for models, avoid hardcoding a single model id; pick from the visible list and filter by its `data-key` instead
|
||
- Prefer robust selectors:
|
||
- role selectors: `getByRole('dialog')`, `getByRole('textbox')`, `getByRole('tab')`
|
||
- stable data attributes already present: `promptSelector`, `[data-component="terminal"]`
|
||
- Keep tests local and fast:
|
||
- do not submit prompts that require real model replies
|
||
- avoid `page.waitForTimeout`; use `expect(...).toBeVisible()` and `expect.poll` when needed
|
||
- Watch for silent UI failures:
|
||
- capture `page.on('pageerror')` and fail test if any are emitted
|
||
- optionally capture console errors (`page.on('console', ...)`) and fail on `type==='error'`
|
||
- Cleanup:
|
||
- these tests should not need to create sessions
|
||
- if a test ever creates sessions or PTYs directly, clean up with SDK calls in `finally`
|
||
|
||
---
|
||
|
||
### Validation Plan
|
||
|
||
Run locally:
|
||
|
||
- `cd packages/app`
|
||
- `bun run test:e2e:local -- --grep smoke`
|
||
|
||
Verify:
|
||
|
||
- all new tests pass consistently across multiple runs
|
||
- overall e2e suite time does not increase significantly
|
||
|
||
---
|
||
|
||
### Open Questions
|
||
|
||
- Should we add a small helper in `packages/app/e2e/utils.ts` for “type into prompt contenteditable” to reduce duplication?
|
||
- Do we want to gate these smoke tests with a dedicated `@smoke` naming convention (or `test.describe('smoke', ...)`) so CI can target them explicitly?
|