QC Jira + Zephyr Essential Collector — Runbook
How to deploy the QC Jira collector, configure it for a customer's Jira Server/DC + Zephyr Essential, and drive the pipeline via REST endpoints.
This guide shows how to deploy the QC collector, configure it against a customer's Jira Server / Data Center instance (with Zephyr Essential), and trigger every collection step manually via REST endpoints. Use it for onboarding a new customer, debugging a sync, or validating data before enabling scheduled collection.
Architecture in one line: one container that reads bugs, tests, and Zephyr test executions from a single Jira DC instance, normalizes them, and writes to MongoDB. The dashboard / API service reads from Mongo — the collector never talks to Jira on the API path.
1. What the collector does
Project discovery
qc_project_discovery
Every Jira project visible to the PAT
qc_projects, qc_collector_state
Field discovery
qc_field_discovery
Customfield IDs (Sprint, Story Points, Zephyr Teststep)
qc_field_map, qc_collector_state
Issue sync
qc_issues_sync
Bugs + Test issues with expand=changelog,names,schema. Stores raw payload, derives normalized view, status history, and link graph.
qc_issues_raw, qc_issues, qc_issue_history, qc_issue_links, qc_project_state, qc_collector_state
Zephyr sync (runs after issue sync)
qc_zephyr_sync
Test cycles + executions from /rest/zapi/latest/*. Enriches qc_issues.test.* on linked Test issues.
qc_test_cycles, qc_test_executions, qc_issues.test.* (in place)
All 11 qc_* collections live in the shared dev-sec-ops-db MongoDB.
The collector is read-only against Jira. It does not write labels, comments, or escalation tickets back. If a customer asks for writeback, push back to the design discussion — it's intentionally out of scope.
2. Prerequisites
Before you start:
Customer-provided:
Jira Server or Data Center base URL (e.g.
https://stg-jira.nomura.com)A service-account Personal Access Token (PAT) with Browse Projects permission on every project to be ingested
Confirmation that Zephyr Essential is installed and licensed (test artifacts won't surface otherwise)
Network access — typically VPN or IP-allowlist to reach the customer's Jira
On the SEC1 side:
MongoDB reachable at
host.docker.internal:27018(SSH tunnel to the shared dev/test Mongo) OR a production Mongo URIDocker / Docker Compose (Compose v2)
This repo cloned and built (
mvn -DskipTests package) — or the prebuilt image pulled
3. Environment variables — manual-only setup
This config disables crons so you can drive the pipeline by hand the first time. Once you've verified the data, flip QC_SCHEDULER_ENABLED=true to let the schedulers take over.
Required
Optional — tuning knobs (defaults are sensible)
Docker Compose $-escaping: if your Mongo password (or JIRA_API_TOKEN) contains a literal $, double it in .env ($$) and URL-encode it as %24 in MONGODB_URI. The two values diverge intentionally — one is read raw by the JVM, the other is parsed as a URL.
4. Deployment
Option A — Production: prebuilt image via docker-compose
The qc-jira-collector service is already defined in docker-compose.yml. Drop a tailored env file at .env.qc-jira-collector and bring it up:
You should see, within ~30 seconds:
If health check fails (Cannot connect to Jira, Jira authentication failed, etc.), the container will exit unless JIRA_FAIL_ON_HEALTH_CHECK_ERROR=false. Fix credentials / network and restart.
Option B — Local development with a Dockerised Jira test stack
For local iteration without touching a customer's instance, use docker-compose.testing.yml which spins up a Jira DC 10.3 container, Postgres, and the collector. See TESTING.md for the wizard / Zephyr-plugin install steps.
5. Manual collection — REST endpoints
Two ways to drive collection:
The simple "do everything" trigger (most common — same naming as other collectors)
Per-step admin endpoints under
/api/jira/qc/(for debugging or partial runs)
Port 8086 in both production and local testing.
5.0 The single "kick off everything" trigger
This is the endpoint most people want. Mirrors the convention used by every other collector in this repo (/api/v1/collector/<system>/collect).
Returns immediately (async — the work continues in the background):
Under the hood it runs, in order: project discovery → field discovery → issue sync → Zephyr sync. Monitor progress via:
You'll see each job's status advance through running → idle, with lastSuccessAt timestamps updating. Total time depends on issue count — for Nomura-scale (~100K issues) expect 10–20 minutes on a cold first run; subsequent calls take seconds because of the hash-skip delta.
There's also a project-only variant if you only want to refresh the project list:
Both /api/v1/collector/jira/collect* endpoints are async — they return 200 immediately and the work continues. The /api/jira/qc/sync/* endpoints in §5.2–5.5 are synchronous — they block until the cycle finishes. Use the async one in scripts and CI; use the sync ones when you're debugging and want a single-step return.
5.1 Health & status
Expected on a fresh deploy: qcEnabled: true, activeProjects: 0, fieldMapEntries: 0, all jobs with lastSuccessAt: null.
5.2 Project discovery — populate qc_projects
Returns:
Inspect what was discovered:
If you want to scope which projects to ingest, either:
Set
QC_PROJECT_ALLOWLIST=PAY,AUTHin env and restart, ORToggle individual projects via:
5.3 Field discovery — populate qc_field_map
Returns the count of logical names mapped to customfield IDs (expected 2–8 depending on what's installed):
View the resolved map:
If a Zephyr field you expect isn't resolved, see §7 Troubleshooting.
5.4 Issue sync — populate qc_issues_raw, qc_issues, qc_issue_history, qc_issue_links
This is the main workhorse. On a cold start it does a full pull; on subsequent calls it's delta-driven.
Returns:
Time scales roughly linearly with issue count + JQL chunk size. For Nomura-scale (~100K issues) expect the first cycle to run 10–20 minutes; subsequent delta cycles drop to seconds because the hash-skip path identifies unchanged payloads and skips derivation.
Zephyr sync runs automatically after issue sync when the scheduler is on. To trigger it manually for verification, see 5.5.
5.5 Zephyr sync — populate qc_test_cycles, qc_test_executions, enrich qc_issues.test.*
Returns:
If licenseDisabled: true, Zephyr Essential's REST API rejected with errorId: 12 ("Please enter valid license"). The Jira-issue side still works (Test issues are ingested by the issue sync), but execution data won't be collected. Once the customer applies a Zephyr license:
5.6 Health-check the collected data — /stats
Use this anytime you want a one-shot answer to "did the data collection actually work?". No Jira call; pure Mongo aggregation.
Returns a complete snapshot:
Use the sanity block as your acceptance test. All seven flags green = collection worked. Any true on allIssuesUnmapped / allSeveritiesMedium, or any false on the has* flags after a full run, means something needs investigation — see §7 Troubleshooting.
The issueType / severity / status / testStatus / linkType distributions are useful for spot-checks too — if a customer expects "lots of bugs" but you're seeing 90% STORY, the issuetype-map config may need an override.
5.7 Re-derive without re-collecting — /reprocess
If you change the normalizer (e.g. add a new severity bucket, fix a Zephyr field mapping), bump the DERIVER_VERSION constant in JiraQcIssuesSyncService and call:
Returns:
This makes zero Jira API calls. The customer's instance is untouched. Derivation runs against the locally-stored verbatim payloads in qc_issues_raw.
5.8 Wipe all collected data — /wipe
DESTRUCTIVE. This drops every qc_* collection in Mongo. Use only when:
Starting fresh against a new customer
Recovering from a bad ingestion run
Tearing down a test environment
The endpoint refuses to run without an explicit confirm=DELETE query parameter — a hand-typed token automation won't accidentally produce.
Returns:
Zero Jira calls — only Mongo collections are dropped. The customer's Jira instance is untouched. Indexes are recreated automatically on the next collector startup (via QcTtlReconciler). To rebuild the data right away:
6. Going from manual to scheduled
Once the data looks right in Mongo (or in the dashboard):
Set
QC_SCHEDULER_ENABLED=trueRestart the container
Verify in
/api/jira/qc/statusthatlastRunAtadvances on the configured cron cadence
The default crons are conservative and won't overwhelm Jira:
Project discovery: every 6 hours
Field discovery: daily at 03:00
Issue sync (with Zephyr appended): every 10 minutes — delta-driven, so steady-state cost is small
7. Troubleshooting
Container exits at startup with Cannot connect to Jira at https://...
Network / VPN / token
Verify the URL is reachable from inside the container: docker exec qc-test-jira-collector curl -fsS https://stg-jira.<customer>.com/rest/api/2/serverInfo
Health check passes but QC: PAT can see 0 project(s)
PAT doesn't have Browse Projects on any project
Have the customer scope the service account broader, or set QC_PROJECT_ALLOWLIST to a known-good project
Connection refused on Mongo
SSH tunnel down OR password not URL-encoded
Check nc -zv localhost 27018; ensure $ in password is %24 in MONGODB_URI
licenseDisabled: true from /sync/zephyr
Customer hasn't applied a Zephyr Essential license
Customer applies a license; call /sync/zephyr/reset-license, retry
Issue sync returns error: ... '%' is a reserved JQL character
Re-encoding bug (resolved in current build)
Pull latest; this was a known double-encode that's now fixed
qc_issue_history rows duplicating
Unique compound index missing
Check db.qc_issue_history.getIndexes() — should include unique_transition. If missing, restart the container; QcTtlReconciler will create it.
Sync cycle takes >30 minutes
First cold start at Nomura-scale, or a slow Jira instance
Inspect qc_collector_state.lastRunDurationMs; lower QC_PROJECT_CHUNK_SIZE if individual JQL calls are timing out
A Zephyr field we expect (e.g. test status) isn't resolved
Customer named it differently from our heuristic
Add a QC_CUSTOM_FIELD_OVERRIDES config entry, or extend the heuristic in JiraFieldDiscoveryService
Spring binding error like No setter found for property: qc-enabled
Old env var name
Use QC_* (top-level) not JIRA_QC_* — that prefix was renamed
Useful one-liners
8. Endpoint reference
"Kick off everything" — aggregate triggers (async)
POST
/api/v1/collector/jira/collect
The main entry point. Runs project + field discovery, issue sync, Zephyr sync — all in one background job.
POST
/api/v1/collector/jira/collect-projects
Just refresh the project list (background)
Per-step admin endpoints (synchronous)
All under /api/jira/qc/.
GET
/status
Overall health + per-job last-run stats
GET
/projects
Discovered project list (filter ?archived=true for archived)
POST
/projects/{key}/enabled?value={true|false}
Toggle a project's enabled flag
GET
/field-map
Active customfield map + raw entries
GET
/project-state
Per-project sync watermarks
GET
/project-state/{key}
Single project watermark
POST
/project-state/{key}/reset
Reset a project's cursor (force re-pull)
POST
/discover/projects
Run project discovery now
POST
/discover/fields
Run customfield discovery now
POST
/sync/issues
Run an issue sync cycle now
POST
/sync/zephyr
Run Zephyr cycles + executions sync now
POST
/sync/zephyr/reset-license
Clear cached "Zephyr disabled" sentinel after a license is applied
POST
/reprocess?onlyStale={true|false}
Re-derive normalized from raw with zero Jira calls
GET
/stats
Sanity snapshot: per-collection counts, freshness, distributions, sanity flags. Use as acceptance test after a run.
POST
/wipe?confirm=DELETE
DESTRUCTIVE. Drops every qc_* collection (refuses without the confirm token). Used for clean restarts.
9. Companion docs
TEST_RESULTS.md — what was verified end-to-end against a local Jira + Zephyr Essential stack
QC_API_HANDOFF.md — schema reference for the API service that reads the Mongo collections
CLAUDE.md — architectural decisions and conventions for this codebase
TESTING.md — how to bring up the local Jira test stack
Last updated