End-to-end tests verify that the entire application works from the user’s perspective. For AI-powered products, this means testing the full flow: user submits input, the system processes it through retrieval, inference, and post-processing, and the user sees a meaningful response in the UI. E2E tests are the most expensive and slowest tests in the pyramid, but they catch integration failures that no other layer can.

Testing AI Chatbot UIs with Playwright

Playwright is the preferred tool for E2E testing AI applications because of its superior support for network interception, streaming responses, and async operations.

import pytest
from playwright.sync_api import expect

def test_chatbot_responds_to_question(page):
    page.goto("http://localhost:3000/chat")
    page.get_by_placeholder("Ask a question").fill("What is machine learning?")
    page.get_by_role("button", name="Send").click()

    # Wait for the response to appear (AI responses take time)
    response_bubble = page.locator(".response-message").last
    expect(response_bubble).to_be_visible(timeout=30000)

    # Assert response is non-empty and contains relevant content
    text = response_bubble.inner_text()
    assert len(text) > 20, "Response too short to be meaningful"

Testing Streaming Responses

Many AI UIs stream responses token-by-token via Server-Sent Events (SSE). Testing streaming requires waiting for the stream to complete rather than checking for immediate content.

def test_streaming_response_completes(page):
    page.goto("http://localhost:3000/chat")
    page.get_by_placeholder("Ask a question").fill("Explain neural networks")
    page.get_by_role("button", name="Send").click()

    response = page.locator(".response-message").last

    # Wait for streaming indicator to disappear (signals completion)
    page.locator(".streaming-indicator").wait_for(state="hidden", timeout=60000)

    text = response.inner_text()
    assert len(text) > 50
    # Verify no partial JSON or broken formatting leaked into UI
    assert "```" not in text or text.count("```") % 2 == 0

Testing Async AI Workflows

Some AI features are asynchronous: the user submits a document, processing happens in the background, and results appear later. Test these with polling patterns.

def test_document_analysis_workflow(page):
    page.goto("http://localhost:3000/upload")

    # Upload a document
    page.set_input_files("input[type=file]", "tests/fixtures/sample_report.pdf")
    page.get_by_role("button", name="Analyze").click()

    # Wait for processing status
    expect(page.get_by_text("Processing")).to_be_visible(timeout=5000)

    # Wait for completion (may take up to 2 minutes for large documents)
    expect(page.get_by_text("Analysis complete")).to_be_visible(timeout=120000)

    # Verify results are displayed
    results = page.locator(".analysis-results")
    expect(results).to_be_visible()
    assert "Summary" in results.inner_text()

Handling Non-Deterministic Outputs

The central challenge of E2E testing AI products: the same input produces different outputs on each run. Use these strategies.

Assert structure, not content. Verify that the response contains a heading, a paragraph, and a citation. Do not assert the exact text.

def test_response_has_expected_structure(page):
    page.goto("http://localhost:3000/chat")
    page.get_by_placeholder("Ask a question").fill("Summarize the Q3 report")
    page.get_by_role("button", name="Send").click()
    page.locator(".streaming-indicator").wait_for(state="hidden", timeout=60000)

    response = page.locator(".response-message").last
    # Check structural elements, not exact content
    assert len(response.inner_text()) > 100
    expect(response.locator(".citation")).to_have_count(1, timeout=5000)

Mock the AI backend for deterministic E2E tests. Intercept API calls in Playwright and return fixture responses. This lets you test the full UI flow without model non-determinism.

def test_chatbot_ui_with_mocked_backend(page):
    # Intercept the AI API call and return a fixture
    page.route("**/api/chat", lambda route: route.fulfill(
        status=200,
        content_type="application/json",
        body='{"response": "Machine learning is a subset of AI that learns from data."}'
    ))

    page.goto("http://localhost:3000/chat")
    page.get_by_placeholder("Ask a question").fill("What is ML?")
    page.get_by_role("button", name="Send").click()

    response = page.locator(".response-message").last
    expect(response).to_contain_text("Machine learning is a subset of AI")

This is the recommended approach for most E2E tests. Run a small set of tests against the real backend on a schedule, but use mocked backends for the PR-level E2E suite.

Visual Regression Testing

AI-generated content can break layouts: unexpectedly long responses, malformed markdown, or special characters. Visual regression testing catches these.

def test_chat_layout_visual_regression(page):
    page.route("**/api/chat", lambda route: route.fulfill(
        status=200,
        content_type="application/json",
        body='{"response": "A short answer."}'
    ))
    page.goto("http://localhost:3000/chat")
    page.get_by_placeholder("Ask a question").fill("Test question")
    page.get_by_role("button", name="Send").click()
    page.locator(".response-message").last.wait_for(state="visible")

    expect(page).to_have_screenshot("chat-short-response.png", max_diff_pixel_ratio=0.01)

Create visual snapshots for different response types: short answers, long answers, responses with code blocks, responses with tables, and error states. Update snapshots intentionally when the UI design changes.

Cypress vs Playwright for AI Apps

Playwright has significant advantages for AI application testing. It supports multiple browser contexts in parallel, handles network interception more flexibly (critical for mocking AI APIs), and provides better async/await patterns for long-running AI operations. Cypress’s automatic retry and built-in waiting are convenient but less flexible when you need to wait for streaming responses or background processing. The playwright-testing-guide covers Playwright patterns in depth, and the playwright-vs-cypress comparison provides a detailed feature comparison.

CI Integration

E2E tests are expensive. Run the mocked-backend E2E suite on every PR (takes 2-5 minutes). Run the full E2E suite with real AI backends on merge to main or on a nightly schedule. Use parallelization (Playwright supports sharding across CI workers) to keep execution time under 10 minutes.

# GitHub Actions example for E2E tests
e2e-tests:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: npm ci
    - run: npx playwright install --with-deps
    - run: npx playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }}
  strategy:
    matrix:
      shard: [1, 2, 3, 4]