Xray is a test management application for Jira. It links requirements to checks and shows execution results in the context of delivery. This article concerns test management, rather than security analysis tools with the same name. Xray's strength for QA is traceability: following a user-facing promise through to an actual check and defect. That chain is useful only when relationships are correct, tests are meaningful, and the report context is appropriate.
Entities to distinguish
Test describes a check, Precondition describes a reusable prerequisite, and Test Set is a logical collection of tests. Test Execution represents a particular execution of a collection, while Test Run stores an individual test's result within that execution. Test Plan defines the planned scope and groups related executions. Some entities are Jira issues, but a Test Run should not be treated as a separate Jira issue. Installed types and capabilities depend on configuration; consult the appropriate documentation for Cloud or Server/Data Center.
Why Test Set does not replace Test Plan
A Test Set is useful as a collection of checks, such as "Critical payment operations." Membership alone does not tell you whether a test passed on the current version. Test Plan tracks planned tests and results from associated Test Executions. Using a collection as a readiness report can display a list of good scenarios without any recent execution. Start with the management question, then choose the entity that actually contains the answer.
Test status versus execution result
The status of a Test Jira issue may describe the instruction's lifecycle: draft, review, or ready to use. A Test Run result describes an observation: passed, failed, not executed, or blocked according to project settings. These are separate dimensions. A fully approved test can fail on a new build; an outdated test can pass while missing the relevant rule. Avoid mechanically linking these statuses without understanding the team's process.
Writing a manual Test
Express the title as verifiable behavior, preconditions as a reproducible state, and steps as actions with clear expected results. For a payment, specify the currency, payment method, order state, and how to verify the outcome. When using Precondition, ensure the link does not conceal critical differences: new and existing users may have different restrictions. A single long test case for the entire checkout makes failures difficult to isolate. Separate checks by purpose while keeping a distinct critical end-to-end scenario.
Exercise: refunding a partially paid order
A Jira story covers refunds. The rule is that a refund cannot exceed the amount actually paid, and a repeated request must not create a duplicate refund. The test environment contains an order paid with 6000 minor currency units and a payment provider sandbox. You need to verify the calculation, limits, and repeated operation, as well as the button's presence.
- 1.Create Tests for a refund of 2000, exceeding the available amount, and repeating a request. Link them to the requirement and describe the side effects to verify.
- 2.Add the checks to the release Test Plan and create a Test Execution with build and environment context. The result should be an explicit planned scope and a place for new results.
- 3.Refund 2000. Check the operation record and the remaining refundable balance of 4000; record the Test Run with the actual values.
- 4.After the first refund, try refunding 5000. The product rules should reject it without a second financial transaction.
- 5.Repeat the original request with the agreed idempotency key. The contract-defined handling should occur without a second refund; if it fails, link the bug to the failed execution.
- 6.After the fix, arrange the appropriate retest within the agreed process and record the new build. Make sure the report does not present an old result as current.
Requirement coverage and environments
Traceability begins with relationships, but test content determines quality. Check which Jira issue types count as requirements and which relationships Xray considers in the project. Then clarify the coverage calculation context: version, Test Plan, Test Environment, and related executions. One successful Chrome result does not verify a mobile scenario. Latest-result statistics can also hide a previously failed configuration unless the team agrees on how to distinguish environments and reruns.
Automated tests, BDD, and stable identity
Xray can link automated checks and import results in supported formats. The specific import and matching behavior depends on the format, edition, and adapter. For BDD, agree on the direction of scenario synchronization: two people independently editing a feature file and a Jira test create competing sources of truth. Method names, Test keys, and scenario identifiers serve different purposes. Renaming should not create new Tests uncontrollably and break coverage history.
Check the integration before bulk imports
Run a small collection containing a passing test, an intentionally failing test, and a skipped test. Check which Test Execution receives the results, how steps and attachments appear, and which Tests are created or matched. Verify endpoints and authentication methods before copying them between Cloud and Data Center. Preserve build metadata while removing secrets from requests and logs. A missing result is an integration gap, rather than an automatic pass.
Maintaining history and the test repository
Regularly identify requirements without checks, Tests without owners, and scenarios that no longer match the product. Editing a test after execution should not leave readers guessing what was previously tested: use the available history and attach the necessary context. Across multiple projects, agree on access rules and links in advance. If a colleague can see a requirement but cannot open its execution or attachment, nominal traceability offers little practical value for shared decisions.
Mistakes and practical takeaways
Typical mistakes include confusing Test Set with a release plan, treating Test status as an execution result, and trusting an overall green chart without checking its filters. Xray does not guarantee complete coverage; it displays the data the team has linked and executed. A good outcome consists of traceable requirements, current Tests, and verified Test Runs on the required build. QA can then explain both what passed and which risks remain before release.