debug

Reading Logs from Podman

The two collectors run as separate containers:

Container	Service
gitlab-collector	GitLab DevInsight collection (GitLabCollectorService)
ci-pipeline-collector	Pipeline detail enrichment (GitLabPipelineCollectorService)

Verify they're running:

podman ps --format "{{.Names}}\t{{.Status}}" | grep -E "gitlab-collector|ci-pipeline-collector"

macOS note: these commands use mktime, which requires GNU awk. Install with brew install gawk and replace awk with gawk.

---

A. GitLab Full Collection Cycle — Wall-Clock per Run

podman logs gitlab-collector 2>&1 | awk '
  /Starting GitLab DevInsight data collection cycle/ {
    start_ts = $1 " " $2
    start_epoch = mktime(gensub(/[-:]/," ","g",start_ts))
  }
  /Completed GitLab DevInsight data collection cycle/ && start_epoch {
    end_ts = $1 " " $2
    end_epoch = mktime(gensub(/[-:]/," ","g",end_ts))
    printf "%s -> %s   %d sec\n", start_ts, end_ts, end_epoch - start_epoch
    start_epoch = 0
  }'

B. GitLab Per-Job Aggregated Stats (runs / avg / min / max in ms)

podman logs gitlab-collector 2>&1 | awk -F'[][]' '/GitLabCollectorService - Job \[/ {
    job=$2
    match($0, /in ([0-9]+)ms/, m); ms=m[1]
    n[job]++; sum[job]+=ms
    if (ms<min[job]||min[job]==0) min[job]=ms
    if (ms>max[job]) max[job]=ms
}
END {
    printf "%-25s %6s %10s %10s %10s\n","job","runs","avg_ms","min_ms","max_ms"
    for (j in n) printf "%-25s %6d %10d %10d %10d\n", j, n[j], sum[j]/n[j], min[j], max[j]
}' | sort

C. Pipeline Collector — Per-Cycle Wall-Clock

podman logs ci-pipeline-collector 2>&1 | awk '
  /GitLabPipelineCollectorService - Refreshing ALL pipeline details/ {
    s_ts = $1 " " $2
    s = mktime(gensub(/[-:]/," ","g",s_ts))
  }
  /GitLabPipelineCollectorService - Stage 2: enriched/ && s {
    e_ts = $1 " " $2
    e = mktime(gensub(/[-:]/," ","g",e_ts))
    match($0, /enriched ([0-9]+) pipelines/, m)
    printf "%s -> %s   %4d sec   (%s pipelines)\n", s_ts, e_ts, e-s, m[1]
    s = 0
  }'

D. Pipeline Collector — Summary Stats

podman logs ci-pipeline-collector 2>&1 | awk '
  /GitLabPipelineCollectorService - Refreshing ALL pipeline details/ {
    s = mktime(gensub(/[-:]/," ","g",$1" "$2))
  }
  /GitLabPipelineCollectorService - Stage 2: enriched/ && s {
    e = mktime(gensub(/[-:]/," ","g",$1" "$2))
    d = e - s; n++; sum += d
    if (min==0 || d<min) min=d
    if (d>max) max=d
    s = 0
  }
  END {
    if (n) printf "pipeline cycles: %d   avg=%.1fs   min=%ds   max=%ds   total=%ds\n",
                  n, sum/n, min, max, sum
  }'

---

Capturing Errors & Warnings

Run these against whichever container you're inspecting. Replace <container> with gitlab-collector or ci-pipeline-collector.

E. All ERROR / WARN lines

# GitLab collector
podman logs gitlab-collector 2>&1 | grep -E " (ERROR|WARN) "

# Pipeline collector
podman logs ci-pipeline-collector 2>&1 | grep -E " (ERROR|WARN) "

F. Errors with stack-trace context (15 lines after each ERROR)

podman logs <container> 2>&1 | grep -A 15 " ERROR "

G. Distinct errors, deduplicated and counted

podman logs <container> 2>&1 \
  | grep -E " ERROR " \
  | sed -E 's/^[0-9-]+ [0-9:]+ \[[^]]+\] ERROR //' \
  | sort | uniq -c | sort -rn

H. HTTP failure status codes from the GitLab API

podman logs <container> 2>&1 \
  | grep -oE "[0-9]{3} (Not Found|Unauthorized|Forbidden|Bad Request|Too Many Requests|Internal Server Error|Bad Gateway|Service Unavailable|Gateway Timeout)" \
  | sort | uniq -c | sort -rn

I. Per-project soft failures (404, skipped resources)

podman logs <container> 2>&1 \
  | grep -oE "Could not fetch [a-z_ ]+ for project [0-9]+: [0-9]+ [A-Za-z ]+" \
  | sort | uniq -c | sort -rn

J. Rate-limiting, timeouts, connection issues

podman logs <container> 2>&1 \
  | grep -iE "rate.?limit|429|timeout|timed out|connection reset|connection refused|socket" \
  | grep -vE " DEBUG "

K. Cycle health summary — errors per GitLab cycle

podman logs gitlab-collector 2>&1 | awk '
  /Starting GitLab DevInsight data collection cycle/ {
    in_cycle = 1; start = $1 " " $2; errs = 0
  }
  in_cycle && / ERROR / { errs++ }
  /Completed GitLab DevInsight data collection cycle/ && in_cycle {
    printf "%s -> %s   errors=%d\n", start, $1 " " $2, errs
    in_cycle = 0
  }'

---

Tips for Podman Log Capture

• Limit time range: podman logs --since 1h gitlab-collector 2>&1 | …, or since a timestamp: --since 2026-04-24T09:00:00.

• Tail recent only: podman logs --tail 5000 gitlab-collector 2>&1 | … for quick spot checks.

• Live monitor errors: podman logs -f gitlab-collector 2>&1 | grep --line-buffered -E " (ERROR|WARN) ".

• Save a snapshot for offline analysis:

  podman logs gitlab-collector       > gitlab-collector.log 2>&1
  podman logs ci-pipeline-collector  > ci-pipeline-collector.log 2>&1

  Then run the same awk/grep commands against the files (drop the podman logs … | prefix and pass the file as the last argument).

• Why 2>&1: Spring Boot writes logs to stderr by default; without 2>&1, grep/awk only see stdout.

• Podman timestamps vs app timestamps: the awk scripts above parse the app's own YYYY-MM-DD HH:MM:SS prefix from $1 $2. Avoid --timestamps (it adds an RFC3339 prefix that shifts the columns and breaks the scripts).

Pipeline Categorization Rules

Runtime-mutable rules stored in MongoDB (pipeline_category_rules collection). They're evaluated by the collector during pipeline enrichment and by the /pipelines/recategorize endpoint when re-stamping stored pipelines.

• Editable at runtime via POST /api/v1/collector/gitlab/classification/rules — no container rebuild, no redeploy.

• Applied to stored data via POST /api/v1/collector/gitlab/pipelines/recategorize?daysBack=N — pure Mongo, no GitLab calls.

• Evaluated in priority order (ascending). First matching rule sets pipelineCategory. Every matching rule contributes labels.

---

Quick Reference

Priority	Name	Match type	Matched against	Sub-label dimensions
10	NPC2	namespace	eis-terraform-npc-provisioning (substring on namespace path)	env
20	TOM	namespace	eis-grafana-tom (substring)	subteam
30	Database	namespace	EIS-DBMW-DBENG (substring)	—
40	NCD	include	.gitlab-ci.yml include.project matches .*ncd.*pipeline.*	subtype

Fallthrough: pipelines matching no rule receive pipelineCategory = "custom". Pipelines with no .gitlab-ci.yml and no namespace match receive pipelineCategory = "none".

---

Rule 1 — NPC2 (priority 10)

Namespace-based. Categorizes any project whose namespace path contains eis-terraform-npc-provisioning. Works for both lab (gitlab.com) and production (gitlab.nomura.com) URLs.

JSON

{
  "name": "NPC2",
  "matchType": "namespace",
  "namespacePattern": "eis-terraform-npc-provisioning",
  "priority": 10,
  "enabled": true,
  "description": "NPC2 — env from namespace suffix on first path segment",
  "subRules": [
    { "field": "namespace", "pattern": "-(nonprodtest|nonprod)(/|$)", "label": "nonprod", "key": "env", "enabled": true },
    { "field": "namespace", "pattern": "-(prodtest|prod)(/|$)",       "label": "prod",    "key": "env", "enabled": true },
    { "field": "namespace", "pattern": "-qa(/|$)",                    "label": "qa",      "key": "env", "enabled": true }
  ]
}

Sub-label table

Namespace suffix	env label
-prodtest	prod
-prod	prod
-nonprodtest	nonprod
-nonprod	nonprod
-qa	qa

Sub-rule order matters: nonprod is listed before prod because -prod would otherwise substring-match -nonprodtest.

Example outputs

URL	pipelineCategory	pipelineLabels	pipelineLabelMap
gitlab.com/eis-terraform-npc-provisioning-prodtest/foo	NPC2	[NPC2, prod]	{category: NPC2, env: prod}
gitlab.nomura.com/eis-terraform-npc-provisioning-nonprod/bar	NPC2	[NPC2, nonprod]	{category: NPC2, env: nonprod}
gitlab.nomura.com/eis-terraform-npc-provisioning-qa/x	NPC2	[NPC2, qa]	{category: NPC2, env: qa}

---

Rule 2 — TOM (priority 20)

Namespace-based. Matches eis-grafana-tom and distinguishes the rules vs eng sub-teams.

JSON

{
  "name": "TOM",
  "matchType": "namespace",
  "namespacePattern": "eis-grafana-tom",
  "priority": 20,
  "enabled": true,
  "description": "TOM grafana/observability — sub-team from first-segment suffix (rules / eng)",
  "subRules": [
    { "field": "namespace", "pattern": "-rules(/|$)", "label": "rules", "key": "subteam", "enabled": true },
    { "field": "namespace", "pattern": "-eng(/|$)",   "label": "eng",   "key": "subteam", "enabled": true }
  ]
}

Sub-label table

Namespace suffix	subteam label
-rules	rules
-eng	eng

Example outputs

URL	pipelineCategory	pipelineLabels	pipelineLabelMap
gitlab.nomura.com/eis-grafana-tom-rules/rules-repo	TOM	[TOM, rules]	{category: TOM, subteam: rules}
gitlab.nomura.com/eis-grafana-tom-eng/grafana-ui-automation-job	TOM	[TOM, eng]	{category: TOM, subteam: eng}
gitlab.nomura.com/eis-grafana-tom/whatever	TOM	[TOM]	{category: TOM}

---

Rule 3 — Database (priority 30)

Namespace-based. Catches the DBMW database engineering group. No sub-rules.

JSON

{
  "name": "Database",
  "matchType": "namespace",
  "namespacePattern": "EIS-DBMW-DBENG",
  "priority": 30,
  "enabled": true,
  "description": "DBMW database engineering pipelines"
}

Example outputs

URL	pipelineCategory	pipelineLabels	pipelineLabelMap
gitlab.nomura.com/EIS-DBMW-DBENG/dbmw-housekeeping	Database	[Database]	{category: Database}

---

Rule 4 — NCD (priority 40)

Include-based. Matches any .gitlab-ci.yml whose include.project value contains ncd.*pipeline.* (e.g. gts-cta-strategy-innersource/ncd/pipeline-templates). Sub-rules examine include.file to identify Helm CI / Application CI / Dependency CI.

JSON

{
  "name": "NCD",
  "matchType": "include",
  "includeProjectPattern": ".*ncd.*pipeline.*",
  "priority": 40,
  "enabled": true,
  "description": "NCD template family — Helm sub-rule listed first so it outranks Application",
  "subRules": [
    { "field": "templateFile", "pattern": "NCD-Build\\.helm\\.local\\.gitlab-ci\\.yml", "label": "Helm CI",        "key": "subtype", "enabled": true },
    { "field": "templateFile", "pattern": "NCD-Dependency\\.local\\.gitlab-ci\\.yml",   "label": "Dependency CI",  "key": "subtype", "enabled": true },
    { "field": "templateFile", "pattern": "NCD-Build\\.local\\.gitlab-ci\\.yml",        "label": "Application CI", "key": "subtype", "enabled": true }
  ]
}

Sub-label table

include.file	subtype label
NCD-Build.helm.local.gitlab-ci.yml	Helm CI
NCD-Dependency.local.gitlab-ci.yml	Dependency CI
NCD-Build.local.gitlab-ci.yml	Application CI

Sub-rule order matters: Helm CI is checked before Application CI because NCD-Build.helm.local.gitlab-ci.yml would otherwise substring-match the Application pattern.

Example outputs

include.project	include.file	pipelineCategory	pipelineLabels	pipelineLabelMap
gts-cta-strategy-innersource/ncd/pipeline-templates	NCD-Build.helm.local.gitlab-ci.yml	NCD	[NCD, Helm CI]	{category: NCD, subtype: Helm CI}
gts-cta-strategy-innersource/ncd/pipeline-templates	NCD-Build.local.gitlab-ci.yml	NCD	[NCD, Application CI]	{category: NCD, subtype: Application CI}
gts-cta-strategy-innersource/ncd/pipeline-dependency	NCD-Dependency.local.gitlab-ci.yml	NCD	[NCD, Dependency CI]	{category: NCD, subtype: Dependency CI}

---

Installation — POST all rules in one go

COLLECTOR=http://localhost:8088

# NPC2
curl -sX POST "$COLLECTOR/api/v1/collector/gitlab/classification/rules" \
  -H 'Content-Type: application/json' -d '{
  "name": "NPC2",
  "matchType": "namespace",
  "namespacePattern": "eis-terraform-npc-provisioning",
  "priority": 10,
  "enabled": true,
  "description": "NPC2 — env from namespace suffix on first path segment",
  "subRules": [
    { "field": "namespace", "pattern": "-(nonprodtest|nonprod)(/|$)", "label": "nonprod", "key": "env", "enabled": true },
    { "field": "namespace", "pattern": "-(prodtest|prod)(/|$)",       "label": "prod",    "key": "env", "enabled": true },
    { "field": "namespace", "pattern": "-qa(/|$)",                    "label": "qa",      "key": "env", "enabled": true }
  ]
}'

# TOM
curl -sX POST "$COLLECTOR/api/v1/collector/gitlab/classification/rules" \
  -H 'Content-Type: application/json' -d '{
  "name": "TOM",
  "matchType": "namespace",
  "namespacePattern": "eis-grafana-tom",
  "priority": 20,
  "enabled": true,
  "description": "TOM grafana/observability — sub-team from first-segment suffix (rules / eng)",
  "subRules": [
    { "field": "namespace", "pattern": "-rules(/|$)", "label": "rules", "key": "subteam", "enabled": true },
    { "field": "namespace", "pattern": "-eng(/|$)",   "label": "eng",   "key": "subteam", "enabled": true }
  ]
}'

# Database
curl -sX POST "$COLLECTOR/api/v1/collector/gitlab/classification/rules" \
  -H 'Content-Type: application/json' -d '{
  "name": "Database",
  "matchType": "namespace",
  "namespacePattern": "EIS-DBMW-DBENG",
  "priority": 30,
  "enabled": true,
  "description": "DBMW database engineering pipelines"
}'

# NCD
curl -sX POST "$COLLECTOR/api/v1/collector/gitlab/classification/rules" \
  -H 'Content-Type: application/json' -d '{
  "name": "NCD",
  "matchType": "include",
  "includeProjectPattern": ".*ncd.*pipeline.*",
  "priority": 40,
  "enabled": true,
  "description": "NCD template family — Helm sub-rule listed first so it outranks Application",
  "subRules": [
    { "field": "templateFile", "pattern": "NCD-Build\\.helm\\.local\\.gitlab-ci\\.yml", "label": "Helm CI",        "key": "subtype", "enabled": true },
    { "field": "templateFile", "pattern": "NCD-Dependency\\.local\\.gitlab-ci\\.yml",   "label": "Dependency CI",  "key": "subtype", "enabled": true },
    { "field": "templateFile", "pattern": "NCD-Build\\.local\\.gitlab-ci\\.yml",        "label": "Application CI", "key": "subtype", "enabled": true }
  ]
}'

Verify

curl -s "$COLLECTOR/api/v1/collector/gitlab/classification/rules" | python3 -c "
import json, sys
d = json.load(sys.stdin)['data']['activeRules']
print(f\"source={d['source']}  count={d['count']}\")
for r in d['rules']:
    print(f\"  {r['priority']:>3}  {r['name']:<10}  matchType={r['matchType']}  subRules={len(r['subRules'])}\")
"

Expected:

source=mongo  count=4
   10  NPC2        matchType=namespace  subRules=3
   20  TOM         matchType=namespace  subRules=2
   30  Database    matchType=namespace  subRules=0
   40  NCD         matchType=include    subRules=3

Apply to stored data

curl -sX POST "$COLLECTOR/api/v1/collector/gitlab/pipelines/recategorize?daysBack=90"

Runs in seconds — pure Mongo + config, no GitLab calls.

---

Sub-rule field reference

The field attribute on a sub-rule decides which raw fact the pattern is matched against:

field value	Source fact	Notes
templateProject	include.project from .gitlab-ci.yml	The template repo path
templateRef	include.ref	The branch / tag of the included template
templateFile	include.file	The specific file inside the template repo
namespace	parsed from repoUrl	E.g. eis-terraform-npc-provisioning-prod/some-repo
repoUrl	full project URL	Useful for rare URL-based regexes

Sub-rule semantics

• All sub-rule patterns are case-insensitive and use Matcher.find() (substring match).

• Sub-rules are evaluated in array order. List the most specific patterns first when they can overlap (e.g. Helm before Application; nonprod before prod).

• Each matching sub-rule contributes its label to pipelineLabels. If key is set, it also lands in pipelineLabelMap[key] (last writer wins on key collisions).

• Set enabled: false to keep a sub-rule on record without applying it.

---

Operational notes

• The classifier caches rules in-process for 60 s. Mutating endpoints invalidate the cache automatically.

• Namespace rules require pipeline.repoUrl to be populated. That field is stamped during enrichment from the project record — which means projects must already exist in dev_insight_projects_collection for namespace rules to fire. Run /gitlab/collect once to seed projects, then /pipelines/refresh for enrichment.

• For include-based rules, only pipelineTemplateProject / pipelineTemplateRef / pipelineTemplateFile are needed — these are captured directly during pipeline enrichment from the parsed .gitlab-ci.yml.

Investigation — Pipeline categorization showing 81% "custom"

Reporter: ops, 2026-05-21 Symptom: On the Nomura GitLab-Demo dataset (4288 pipelines, 30-day window), the collector + dashboard reports:

• NCD = 342 (8%)

• NPC2 = 393 (9%)

• TOM = 80 (2%)

• Database = 1 (0%)

• custom = 3472 (81%)

• none = 0

Spot-check: GM-EUC-QIS-Structuring/renovate-config shows .gitlab-ci.yml with include.project = gts-cta-strategy-innersource/ncd/pipeline-dependency and include.file = NCD-Dependency.local.gitlab-ci.yml. This should classify as NCD / Dependency CI per the active rules. Two rows of renovate-config in the dashboard are tagged NCD / Dependency CI correctly; three other renovate-config rows are tagged custom — same YAML shape, different fork instances.

Additional context:

• 30-day enrichment cycle on this dataset took ~11 hours.

• Running /pipelines/recategorize after collection only changed a handful of records; most stayed custom.

Conclusion: there's a real bug. This document narrows down which of three hypotheses is responsible before any code change.

---

Hypotheses

A. Per-project cache contamination across pipelines

PipelineCategoryClassifier.CACHE is keyed by projectId alone. The cache stores a Classification object that carries the first include's templateProject / templateRef / templateFile. When a project has multiple pipelines on different refs (e.g. main, release-2.x, feature branches), they all share the same cached Classification from whichever ref was fetched first — even if their actual .gitlab-ci.yml could differ on different refs.

If this is the cause:

• A subset of pipelines would have wrong template values stamped (matching the first-cached ref, not their own).

• Affected pipelines: those in projects with diverse refs.

B. Stale enrichment from before classification was wired up

Stage 1 (pipelines_collector) creates pipeline docs with no template fields. Stage 2 enrichment runs only when explicitly triggered (/pipelines/refresh). If force=false, terminal pipelines hit isFullyEnrichedTerminal short-circuit and are skipped. Pipelines enriched once before the categorization rules existed (or before the multi-include/namespace fix) carry pipelineCategory = "custom" indefinitely.

/pipelines/recategorize works off stored pipelineTemplateProject / pipelineTemplateRef / pipelineTemplateFile. If those fields are null (because enrichment never captured them), recategorize has nothing to derive from and the doc remains custom.

If this is the cause:

• Many "custom" pipelines would have null pipelineTemplate* fields.

• Recategorize wouldn't help — only a force=true re-enrichment would, because that re-fetches the YAML.

• Distribution by collectedAt would show "custom" docs clustered in older cycles.

C. YAML fetch failing for many projects

fetchGitLabCiYaml returns null on 404, network error, or any non-2xx. When YAML is null, classifier returns noYamlCategory (default "none"). The dashboard shows 0 "none", so this isn't a primary cause — but partial failures (e.g. some refs return 200, others 404) could still contribute via the cache.

---

Investigation queries

Run these against the Mongo backing the affected environment. Do not commit the output to git — it may contain repo names/IDs. Paste back as plain text in the investigation thread.

Query 1 — How many "custom" pipelines actually have template facts captured?

db.dev_insight_pipelines_collection.aggregate([
  { $match: { pipelineCategory: "custom" } },
  { $group: {
      _id: {
        hasProject: { $cond: [{ $ifNull: ["$pipelineTemplateProject", false] }, "yes", "no"] },
        hasFile:    { $cond: [{ $ifNull: ["$pipelineTemplateFile",    false] }, "yes", "no"] }
      },
      count: { $sum: 1 }
  }},
  { $sort: { count: -1 } }
])

Interpretation:

• hasProject=no, hasFile=no dominant → Hypothesis B (template facts never captured). /pipelines/recategorize cannot help; a full /pipelines/refresh?force=true&daysBack=30 is needed.

• hasProject=yes, hasFile=yes dominant → Hypothesis A or C (facts captured but rules didn't fire — either wrong values or fetch returned partial data).

Query 2 — Sample a "custom" pipeline that should be NCD

db.dev_insight_pipelines_collection.findOne(
  { pipelineCategory: "custom",
    repoUrl: { $regex: "renovate-config", $options: "i" } },
  { _id: 0, pipelineId: 1, repoUrl: 1, ref: 1, status: 1,
    pipelineCategory: 1, pipelineLabels: 1, pipelineLabelMap: 1,
    pipelineTemplateProject: 1, pipelineTemplateRef: 1, pipelineTemplateFile: 1,
    collectedAt: 1, updatedAt: 1, projectId: 1 }
)

Interpretation:

• All pipelineTemplate* fields null → enrichment never captured the YAML for this pipeline. Confirms Hypothesis B.

• pipelineTemplateProject set to the wrong project (not the one in the actual .gitlab-ci.yml) → cache contamination, Hypothesis A.

• Fields set correctly but pipelineCategory = custom → rule isn't matching. Check active rules via GET /api/v1/collector/gitlab/classification/rules.

Query 3 — Is there a YAML snapshot for that project?

Take projectId from Query 2's result, then:

db.gitlab_ci_yaml.find(
  { gitlabProjectId: <projectId> },
  { _id: 0, ref: 1, includeProject: 1, includeFile: 1, fetchedAt: 1, contentHash: 1 }
).sort({ fetchedAt: -1 })

Interpretation:

• No snapshot → the collector never successfully fetched .gitlab-ci.yml for this project. Network issue, 404, or wrong path. Hypothesis C variant.

• One snapshot per ref → check whether includeProject matches what the dashboard shows.

• Multiple snapshots with different includeProject values → confirms that refs diverge and cache-by-projectId (Hypothesis A) would cause wrong stamping.

Query 4 — When were the "custom" pipelines categorized?

db.dev_insight_pipelines_collection.aggregate([
  { $match: { pipelineCategory: "custom" } },
  { $project: { day: { $dateToString: { format: "%Y-%m-%d", date: "$collectedAt" } } } },
  { $group: { _id: "$day", n: { $sum: 1 } } },
  { $sort: { _id: -1 } },
  { $limit: 15 }
])

Interpretation:

• All custom docs clustered in dates before the categorization rules were uploaded → Hypothesis B is definitive.

• custom docs spread across recent days, including post-rule-upload → enrichment is producing custom even with current rules. Hypothesis A or C.

Query 5 — Distribution of pipelineTemplateFile values among "custom"

db.dev_insight_pipelines_collection.aggregate([
  { $match: { pipelineCategory: "custom", pipelineTemplateFile: { $ne: null } } },
  { $group: { _id: "$pipelineTemplateFile", n: { $sum: 1 } } },
  { $sort: { n: -1 } },
  { $limit: 20 }
])

Interpretation:

• If any of these _id values match patterns from the NCD/NPC2/TOM/Database rules → confirms rule didn't fire even when facts were captured. Cache contamination (A) or rules mismatch.

• If all values are unfamiliar → those projects use different templates not yet in the rule set.

Query 6 — Current active rules (sanity check)

curl -s "http://<collector-host>:8080/api/v1/collector/gitlab/classification/rules" | jq

db.dev_insight_pipelines_collection.aggregate([
  { $match: {
      collectedAt: { $gte: ISODate("2026-05-26T00:00:00Z"), $lt: ISODate("2026-05-27T00:00:00Z") }
  }},
  { $group: {
      _id: {
        hasProject: { $cond: [{ $ifNull: ["$pipelineTemplateProject", false] }, "yes", "no"] },
        hasJobs:    { $cond: [{ $gt: [{ $size: { $ifNull: ["$jobs", []] } }, 0] }, "yes", "no"] }
      },
      count: { $sum: 1 }
  }}
])
podman logs ci-pipeline-collector 2>&1 | grep -E "Refreshing ALL pipeline details" | head -10

Confirm:

• data.activeRules.source is "mongo" (not "yaml-config" fallback).

• data.activeRules.count is 4.

• Each rule has the expected matchType, regex, and sub-rules.

---

Decision tree

Query 1 says	Query 4 says	Likely cause	Fix
hasProject=no dominant	custom clustered in old dates	B — stale enrichment	Run POST /api/v1/collector/gitlab/pipelines/refresh?daysBack=30&force=true — full re-enrich captures fresh template facts
hasProject=yes dominant	custom includes recent dates	A — cache contamination	Code change: key classifier cache by (projectId, ref) instead of projectId alone
Query 3 shows no snapshot	—	C — YAML fetch failing	Check collector logs for "Could not fetch .gitlab-ci.yml" WARNs; inspect for 401/403/404 patterns
Query 5 shows known patterns	—	Rule isn't firing despite facts	Check Query 6 — verify rules loaded and sub-rule order

---

Side observation — 11h cycle time

30 days × ~4000 pipelines took ~11 hours. For reference, the previous baseline was 1h 44m for 19k pipelines on gitlab.com. The discrepancy suggests:

• Stage 1 (pipelines_collector) was the bottleneck — paginated pipeline-list fetch for ~4000 active projects, serially.

• Or Nomura's internal GitLab has higher per-request latency than gitlab.com.

• Or rate-limiting was triggering global pauses.

To diagnose:

# Per-job timings from collector logs
podman logs gitlab-collector 2>&1 | grep -E "completed successfully in [0-9]+ms" | tail -30

# Look for rate-limit pauses
podman logs gitlab-collector 2>&1 | grep -E "rate limit|429|Pausing"

If pipelines_collector (Stage 1) is the longest job, the project-level parallelism added in this release will help once GITLAB_COLLECTOR_PROJECT_PARALLELISM=4 is set. But categorization correctness comes first — don't bump parallelism until the categorization bug is understood.

---

What to send back

When investigating, paste the output of:

1. Query 1 (counts by hasProject/hasFile)

2. Query 4 (counts by day)

3. Query 6 (active rules listing)

That's enough to identify the hypothesis. Queries 2/3/5 are follow-ups depending on what 1+4 show.

Don't change code yet. The fix for B (full re-enrich) doesn't need code changes — it's an operational command. The fix for A is a small code change; the fix for C is environment troubleshooting. Picking the wrong fix risks burning another 11-hour cycle.