ROX-32846: Add repository-to-CPE mapping API to indexer by BradLugo · Pull Request #18705 · stackrox/stackrox · GitHub
Skip to content

ROX-32846: Add repository-to-CPE mapping API to indexer#18705

Merged
BradLugo merged 15 commits into
masterfrom
blugo/ROX-32846-GetRepositoryToCPEMapping
May 5, 2026
Merged

ROX-32846: Add repository-to-CPE mapping API to indexer#18705
BradLugo merged 15 commits into
masterfrom
blugo/ROX-32846-GetRepositoryToCPEMapping

Conversation

@BradLugo

@BradLugo BradLugo commented Jan 27, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a new RPC, GetRepositoryToCPEMapping, to the Indexer service that returns the Red Hat repository-to-CPE mapping used for RHEL package vulnerability matching. The matcher's ScanSBOM API will need this mapping to enrich RHEL packages with CPE information during SBOM vulnerability scanning.

The Indexer acts as a stateless proxy to the repository-to-CPE mapping file from Central, while the matcher caches the data locally with periodic background refresh. The GetRepositoryToCPEMapping supports conditional fetch via If-Modified-Since to minimize data transfer on refresh.

Alternatives

  • Individual repository-to-cpe lookup: Frankly, this would be my long-term solution if we weren't planning on moving toward other indexing improvements (i.e., "slim index reports"), which is really my preferred solution here. This is a valid option, and we could prove whether it's better than sending the entire mapping over with performance testing, but the current changes seemed easier to implement at the time. We can revisit this during the Tech Preview implementation.
  • repository-to-cpe mapping in Matcher: We'd need to have the same repository-to-cpe related configurations in the Matcher, which requires deployment configuration changes. Given that this is meant to be a somewhat temporary solution, I didn't see the benefit in that effort.
  • SBOM parsing in the Indexer: Some of the SBOMs are already too large to send over the wire as-is. Sending that data between the Indexer and Matcher seemed like too much. The repository-to-cpe data is much smaller.
  • Eager initialization in the Matcher: The updater could fetch the mapping on startup, but lazy initialization avoids unnecessary work for matchers that never use the repo-to-CPE data. Especially relevant, since not all (read: most) users won't use this feature initially, and I expect the implementation to change substantially over time.

PR stack:

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • added unit tests
  • added e2e tests
  • added regression tests
  • added compatibility tests
  • modified existing tests

How I validated my change

  1. Deploy stackrox
  2. Optionally set the ROX_SBOM_SCANNING feature flag:
    ❯ kubectl set env deploy/scanner-v4-indexer ROX_SBOM_SCANNING=true
    
  3. Port forward Indexer
    ❯ kubectl port-forward svc/scanner-v4-indexer 8443
    
  4. Hit the endpoint
    ❯ grpcurl -insecure \
        -import-path proto \
        -proto internalapi/scanner/v4/indexer_service.proto \
        localhost:8443 scanner.v4.Indexer/GetRepositoryToCPEMapping | head
    {
      "modified": true,
      "lastModified": "Mon, 13 Apr 2026 00:45:11 GMT",
      "mapping": {
        "3scale-amp-2-for-rhel-8-ppc64le-debug-rpms": {
          "cpes": [
            "cpe:/a:redhat:3scale:2.13::el8",
            "cpe:/a:redhat:3scale:2.14::el8",
            "cpe:/a:redhat:3scale:2.15::el8",
            "cpe:/a:redhat:3scale:2.16::el8",
    

@rhacs-bot

rhacs-bot commented Jan 27, 2026

Copy link
Copy Markdown
Contributor

@codecov

codecov Bot commented Jan 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.75806% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.58%. Comparing base (d49ad42) to head (566e043).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
pkg/scannerv4/client/client.go 0.00% 31 Missing ⚠️
scanner/indexer/indexer.go 0.00% 26 Missing ⚠️
scanner/indexer/remote.go 0.00% 12 Missing ⚠️
scanner/indexer/repositorytocpefetcher.go 90.32% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #18705      +/-   ##
==========================================
+ Coverage   49.56%   49.58%   +0.02%     
==========================================
  Files        2794     2797       +3     
  Lines      212573   212821     +248     
==========================================
+ Hits       105359   105530     +171     
- Misses      99483    99557      +74     
- Partials     7731     7734       +3     
Flag Coverage Δ
go-unit-tests 49.58% <69.75%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch 2 times, most recently from 1d03f53 to 3a3b390 Compare January 27, 2026 23:09
Comment thread scanner/internal/httputil/updater.go Outdated
Comment thread scanner/indexer/repositorytocpeupdater.go Outdated
@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 3a3b390 to 9bff2aa Compare January 28, 2026 05:52
@BradLugo BradLugo changed the title ROX-32846: add repository-to-CPE mapping API to indexer ROX-32846: Add repository-to-CPE mapping API to indexer Jan 28, 2026
@dcaravel dcaravel force-pushed the dc/sbom-ingest-central-api branch from 01e65a6 to 09f9e5f Compare February 13, 2026 20:18
@dcaravel dcaravel requested a review from a team as a code owner February 13, 2026 20:18
Base automatically changed from dc/sbom-ingest-central-api to master February 13, 2026 23:36

@vikin91 vikin91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks like it would not affect any scanning feature developed in the Sensor team, so it is good to go.
Feel free to request a re-review once the CI checks are resolved.

@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 9bff2aa to 0ea00c1 Compare April 9, 2026 17:54
@BradLugo BradLugo marked this pull request as draft April 9, 2026 17:55
@github-actions

github-actions Bot commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

🚀 Build Images Ready

Images are ready for commit df7bf05. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-867-gdf7bf05676

@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 0ea00c1 to 53ef512 Compare April 13, 2026 18:25
@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch 2 times, most recently from 8f167dd to 6a3585d Compare April 14, 2026 18:22
@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 6a3585d to 113ae4b Compare April 27, 2026 16:02
@BradLugo BradLugo marked this pull request as ready for review April 27, 2026 16:02
@BradLugo

Copy link
Copy Markdown
Contributor Author

/retest

Comment thread scanner/indexer/indexer.go Outdated
Comment thread scanner/matcher/repo2cpe/updater.go Outdated
Comment thread scanner/matcher/repo2cpe/updater_test.go Outdated
Comment thread scanner/matcher/matcher.go Outdated
Comment thread scanner/indexer/repositorytocpefetcher.go Outdated
@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 4d2e343 to 4733fa8 Compare April 29, 2026 19:05
@BradLugo BradLugo requested a review from dcaravel April 29, 2026 21:37
@BradLugo

Copy link
Copy Markdown
Contributor Author

Gonna take another look at repo2cpe updater again tomorrow morning. I think there are some slight improvements we can make before merging.

@BradLugo

Copy link
Copy Markdown
Contributor Author

Ok, the repo2cpe.Updater kind of got away from me, but I like it 🤷 Happy to make it simpler if we don't want all that complexity. Should note that I believe a lot of this will be ripped out at some point for a better Claircore API.

@dcaravel dcaravel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 3a847e9 to 43fa702 Compare May 5, 2026 00:38
@BradLugo

BradLugo commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

Rebasing and incorporating #20259.

@openshift-ci

openshift-ci Bot commented May 5, 2026

Copy link
Copy Markdown

@BradLugo: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ocp-4-20-operator-e2e-tests 9bff2aa link false /test ocp-4-20-operator-e2e-tests
ci/prow/ocp-4-20-nongroovy-e2e-tests 9bff2aa link false /test ocp-4-20-nongroovy-e2e-tests
ci/prow/ocp-4-20-ui-e2e-tests 9bff2aa link false /test ocp-4-20-ui-e2e-tests
ci/prow/ocp-4-20-qa-e2e-tests 9bff2aa link false /test ocp-4-20-qa-e2e-tests
ci/prow/ocp-4-20-scanner-v4-install-tests 9bff2aa link false /test ocp-4-20-scanner-v4-install-tests
ci/prow/ocp-4-12-scanner-v4-install-tests 43fa702 link false /test ocp-4-12-scanner-v4-install-tests
ci/prow/ocp-4-21-scanner-v4-install-tests 43fa702 link false /test ocp-4-21-scanner-v4-install-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

BradLugo added 15 commits May 5, 2026 10:08
Adds a new RPC, GetRepositoryToCPEMapping, to the Indexer service that
returns the Red Hat repository-to-CPE mapping used for RHEL package
vulnerability matching. This mapping will be needed by the matcher's
ScanSBOM API to enrich RHEL packages with CPE information during SBOM
vulnerability scanning.

Assisted-by: Claude Opus 4.5 (200k) <noreply@anthropic.com>
The indexer now acts as a simple proxy to upstream (Central/Red Hat),
while the matcher caches the data locally with periodic refresh.

Key changes:
- Add conditional fetch support (If-Modified-Since) to gRPC API
- New RepositoryToCPEFetcher in indexer as a stateless proxy
- New repo2cpe.Updater in matcher with lazy initialization
  - Only fetches data on first Get() call
  - Background refresh every 24 hours using conditional requests
  - Close() method for graceful shutdown
- Remove httputil.Updater and RepositoryToCPEUpdater (no longer needed)

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The repo2cpe updater now initializes lazily on first Get() call instead
of requiring an explicit Start(). This avoids fetching mapping data and
starting background refresh for matchers that never use it.

Changes:
- Use sync.Once to trigger init on first access
- Replace blocking Start() with Close() for lifecycle management
- Use done channel instead of context for shutdown signaling

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Particularly to avoid panics for logging during testing.
Don't reach out to WAN by default. This was a result of copy+pasting the
Claircore implementation at the beginning of this branch.
Also changes the related log warning to an error.
Mostly hardening the repo2cpe.Updater lifecycle, adjust a log, and nil
checking.
@BradLugo BradLugo force-pushed the blugo/ROX-32846-GetRepositoryToCPEMapping branch from 43fa702 to 566e043 Compare May 5, 2026 17:08
@BradLugo

BradLugo commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

@BradLugo BradLugo merged commit df7bf05 into master May 5, 2026
105 of 107 checks passed
@BradLugo BradLugo deleted the blugo/ROX-32846-GetRepositoryToCPEMapping branch May 5, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants