CI integration turns individual automated checks into regular feedback. Code changes trigger a reproducible suite, results belong to a specific build, and the team gets enough evidence to decide. The goal is to identify change risk quickly and address failure causes.
Adding a test command to a pipeline is insufficient. The suite needs a ready environment, independent data, clear blocking rules, and accessible artifacts. Otherwise, CI automates waiting and reruns while confidence stays low.
Separate suites by purpose
Fast, focused checks help on every change request. Service checks verify contracts and important business rules. A small UI smoke suite can cover key user journeys. Run expensive browser or device matrices on agreed triggers.
Nightly runs cannot compensate for missing critical-risk checks before release. Define the decision each suite supports and how long the team can wait. Choose triggers according to failure cost as well as duration.
A reproducible environment
Install dependencies from the committed lock file and pin the runtime and application version. Browser tools need matching browsers and system libraries. Caching should speed setup without silently changing versions or hiding missing dependencies.
Check service readiness instead of waiting an arbitrary thirty seconds. Readiness must cover the components the scenario requires: an open port does not establish that migrations are finished or a queue is ready. Distinguish setup failure from a product assertion failure.
A minimal GitHub Actions example
Assume Playwright Test is already configured, package-lock.json and .nvmrc specify a supported Node version, and tests use a local form through page.setContent. This example needs no application server. Save the workflow as .github/workflows/tests.yml.
name: Browser checks
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version-file: .nvmrc
cache: npm
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test --reporter=html
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: playwright-report/
retention-days: 7Expect execution on a pull request or manual trigger, a separate job result, and an HTML artifact if report creation completed. A failing assertion must fail the job while the report-upload step still runs. The example timeout is a training budget to adjust for the suite.
Connecting a real application
Use controlled service startup or a dedicated preview of the required build. Pass the address explicitly so the test knows which version it checks. If backend and frontend deploy separately, retain both versions; otherwise, a green result may describe an incompatible combination.
Create a separate user and order per run. Smoke-test login, a sample order, and reading its result. Expect specific status and data without real charges. Then intentionally break an expectation in a training branch and verify that the pipeline actually blocks the incorrect result.
Secrets and permissions
Store credentials in CI's secret mechanism and expose them only to jobs that need them. Keep tokens out of command lines, reports, and network artifacts. Standard log masking does not guarantee sanitized screenshots, JSON attachments, or traces.
Plan separately for external pull requests: untrusted code must not receive privileged credentials. Avoid switching to a more privileged trigger merely to make tests pass. Use a secret-free suite for such contributions and a separate trusted integration process.
Parallel execution and resources
Workers and shards help when data and resources are independent. Shared accounts, fixed ports, and global cleanup can create races. Establish independence before increasing concurrency, then measure the benefit.
Measure agent queues, dependency installation, application startup, and investigation alongside test duration. Reducing expensive setup can help more than doubling workers. CI resource changes should preserve comparable results.
Blocking checks and exceptions
Every required check needs an owner and response policy. Product failures need a fix or explicit risk decision; infrastructure failures need environment recovery. Skipped must not automatically mean a critical check was completed.
Document temporary exceptions with a reason, expiry, and compensating check. Keep quarantined unstable scenarios visible separately. Silently applying continue-on-error to the whole suite makes testing an optional log.
Test the integration's negative paths too: a wrong dependency, unavailable service, and failed assertion should leave distinct useful evidence. Confirm that renamed workflows or jobs have not left required-check rules waiting for old names. Branch protection must reference results that actually execute. A pipeline that runs successfully but does not participate in the merge decision required by policy has not fulfilled that responsibility.
Artifacts and exit codes
Save reports, relevant logs, and diagnostics after failure without overwriting the runner's exit code with a successful later command. Explicitly verify error and artifact-publication behavior when using shell scripts.
Every result should identify the commit, environment, dependency versions, and run link. Keep different builds' reports separate. Retain artifacts long enough for investigation without accumulating unnecessary data.
Common mistakes and lessons
Common problems include testing an arbitrary shared environment, hidden local-machine dependencies, missing readiness checks, and rerunning the entire pipeline until green. Another is treating green CI as proof of quality without knowing which tests actually ran.
Good CI integration gives fast, trustworthy feedback about a specific product version. Start with a small independent suite, prove it detects an intentional violation, and support clear investigation. Then expand coverage and the matrix while preserving trust.