Testing your chat
The hooks an end-to-end test can rely on — the chat, the field, Send and Stop, a message by role, a reply still being written, a failed turn — the same in every framework, with a Playwright example.
An end-to-end test drives the chat as a person does: it types a message, sends it, waits for the reply, stops one, and reads what the transcript shows. Six things need finding, and they are found the same way in React, Vue, Svelte, Angular and plain HTML.
The hooks
Section titled “The hooks”| A test looks for | The hook |
|---|---|
| The chat | [data-aparte-chat] |
| The field | getByRole('textbox', { name }), where name is the placeholder |
| Send, Stop and Submit | aparte-composer-send[data-act]: send, stop or submit |
| A message, by who wrote it | aparte-chat-bubble[data-role]: user or assistant |
| A reply still being written | [aria-busy="true"], inside the message |
| A turn that failed | [data-error-code], in the reply |
| The value a select shows | aparte-select [role="combobox"], whose text is the shown value and nothing else (a searchable select shows its button, [aria-haspopup="listbox"]) |
These are a promise like the rest of what will not change: a hook moves only
in a major version, after a notice. data-aparte-chat, data-act and data-role are attributes of
elements, so that list names them. The field’s role and name, aria-busy and data-error-code sit on
parts inside an element, so no element lists them; the example below holds them, and this repository
runs it against its six example apps with every change.
A class name is not a hook. Classes are styling, and they can change in a minor release.
Why a role and a name first
Section titled “Why a role and a name first”Look for what a person looks for. The field is a textbox named by its placeholder, and
getByRole('textbox', { name }) finds exactly that. A screen reader says the same words, so a test
written this way fails when the field stops being findable for a reader too, and it survives a
redesign, which a class name does not. getByPlaceholder is not the way to the field: it matches the
placeholder attribute of an <input> or a <textarea>, and the field is neither. While the menu of
Triggers is open, the field is a combobox, as the ARIA pattern for a field
with a popup asks, and it is a textbox again once the menu closes.
The other hooks cover what no role names, and what a label cannot carry because it is translated.
What the button does now is data-act, the same in every language, where its name follows the locale.
Who wrote a message is data-role. Whether a reply is still being written is aria-busy, the state a
screen reader waits on to read the reply once it is complete
(Accessibility). Why a turn failed is data-error-code.
The chat, whatever the framework
Section titled “The chat, whatever the framework”React, Vue and Svelte render a <div data-aparte-chat> in place of the element. Angular’s component
and the plain <aparte-chat> carry the same attribute on themselves, so one selector finds the chat in
all five:
const chat = page.locator('[data-aparte-chat]');A page with several chats has several matches: scope each one from something you own, such as the
<section> you put it in.
A rule of your own that meant “the <div> a framework renders” and was written [data-aparte-chat]
now matches <aparte-chat> as well. Write div[data-aparte-chat] to keep it to the <div>.
Send, Stop and Submit
Section titled “Send, Stop and Submit”aparte-composer-send is one button with three meanings, and its data-act says which one it has
now: send at rest, stop while a reply is being written, submit while a question in the composer
waits for an answer that the button sends. The element has no box of its own, so press the button
inside it:
await chat.locator('aparte-composer-send[data-act="send"] button').click();
await expect(chat.locator('aparte-composer-send[data-act="stop"]')).toBeVisible();After a press of Stop, the button says send again a moment before the reply stops being aria-busy:
wait for the second, not only the first, before you ask what the reply holds.
A message, and a reply still being written
Section titled “A message, and a reply still being written”aparte-chat-bubble[data-role] is one message. Read its text with toContainText rather than an
exact match: a message holds more than its words, such as who wrote it and when.
While a reply is being written, its message is aria-busy="true", from the moment it appears to its
last word. The mark goes when the turn ends, however it ended: finished, failed or stopped. Ask for it
inside the message. The transcript is aria-busy while a conversation loads, and a card or a chart in
the reply is aria-busy while it loads, which is what you want to wait for before you read the reply.
const reply = chat.locator('aparte-chat-bubble[data-role="assistant"]').last();
await expect(reply).toBeVisible();
await expect(reply.locator('[aria-busy="true"]')).toHaveCount(0);Wait for the reply to appear first. Before it exists nothing is busy, and the second wait passes at once.
A failed turn
Section titled “A failed turn”A turn that fails leaves an error in the reply, and data-error-code on it holds the code of the
failure: PROVIDER_ERROR for a 5xx from the model, NET_ERROR when it cannot be reached, and the
others Error lists. The person reads a sentence chosen by that code. A test can
ask for the code itself:
await expect(chat.locator('[data-error-code]')).toHaveAttribute('data-error-code', 'PROVIDER_ERROR');An app that draws its own error with setErrorRenderer draws the markup too: put data-error-code
on it, from the code the renderer receives, and the same line works.
An example
Section titled “An example”This is the file the repository runs against its six example apps: type, send, wait for the reply to finish and read it, stop a reply, and see a failed turn. It reaches the chat through the hooks above and nothing else.
/**
* Testing your chat: the guide's example, and the file the repository runs on its six example
* apps (vanilla, React, Vue, Svelte 4 and 5, Angular), with the same lines in each. It reaches
* the chat through the testing contract and through nothing else: no class, no tag a wrapper
* adds. So what it proves is what a test of your own app can rely on, and a hook that moves
* is what turns this file red.
*/
import { test, expect } from '@playwright/test';
// This repo's stand-in for the model: it answers the API the example apps call. Yours is
// whatever fakes your app's backend: a `page.route`, a test server, a recorded stream.
import { installLlmMock } from '../helpers/mock-llm.js';
test('type, send, wait for the reply to finish, read it', async ({ page }) => {
await installLlmMock(page, { scenario: 'text' });
await page.goto('/');
const chat = page.locator('[data-aparte-chat]');
// The field is a textbox, and its accessible name is the placeholder.
await chat.getByRole('textbox', { name: 'Ask ChatClone' }).fill('Say hello');
// `data-act` says what the button does now: `send`. Press the button inside it.
await chat.locator('aparte-composer-send[data-act="send"] button').click();
// A message is told by its role.
await expect(chat.locator('aparte-chat-bubble[data-role="user"]')).toContainText('Say hello');
const reply = chat.locator('aparte-chat-bubble[data-role="assistant"]').last();
await expect(reply).toBeVisible();
// The reply is finished when nothing in it is `aria-busy`. Read it then.
await expect(reply.locator('[aria-busy="true"]')).toHaveCount(0);
await expect(reply).toContainText('aparte e2e mock');
});
test('stop a reply that is still being written', async ({ page }) => {
await installLlmMock(page, { scenario: 'slow', delayMs: 15_000 });
await page.goto('/');
const chat = page.locator('[data-aparte-chat]');
await chat.getByRole('textbox', { name: 'Ask ChatClone' }).fill('Take your time');
await chat.locator('aparte-composer-send[data-act="send"] button').click();
// While the reply is written, the same button says `stop`, and the reply says it is busy.
const stop = chat.locator('aparte-composer-send[data-act="stop"]');
const reply = chat.locator('aparte-chat-bubble[data-role="assistant"]').last();
await expect(stop).toBeVisible();
await expect(reply.locator('[aria-busy="true"]')).toHaveCount(1);
await stop.locator('button').click();
// Stopped: the button is Send again and nothing is being written.
await expect(chat.locator('aparte-composer-send[data-act="send"]')).toBeVisible();
await expect(reply.locator('[aria-busy="true"]')).toHaveCount(0);
});
test('see a failed turn', async ({ page }) => {
await installLlmMock(page, { scenario: 'http-500' });
await page.goto('/');
const chat = page.locator('[data-aparte-chat]');
await chat.getByRole('textbox', { name: 'Ask ChatClone' }).fill('This one fails');
await chat.locator('aparte-composer-send[data-act="send"] button').click();
// A failed turn leaves an error in the reply, and `data-error-code` says why: here the model
// answered with a 500.
const failure = chat.locator('aparte-chat-bubble[data-role="assistant"] [data-error-code]');
await expect(failure).toBeVisible();
await expect(failure).toHaveAttribute('data-error-code', 'PROVIDER_ERROR');
// And the chat is usable again: nothing is written, the button is Send.
await expect(chat.locator('[aria-busy="true"]')).toHaveCount(0);
await expect(chat.locator('aparte-composer-send[data-act="send"]')).toBeVisible();
});installLlmMock is this repository’s stand-in for the model API that the example apps call. Replace it
with however your app fakes its backend: a page.route, a test server, a recorded stream. The name in
getByRole('textbox', { name: 'Ask ChatClone' }) is the placeholder of the example apps; use yours.