ci: route devbox/Nix package resolution through Artifactory (#1302) · segmentio/analytics-react-native@3fb51ae · GitHub
Skip to content

Commit 3fb51ae

Browse files
abueideabueide
authored
ci: route devbox/Nix package resolution through Artifactory (#1302)
All supply-chain vectors are required to go through Artifactory's scanning/curation proxy, not just npm - this repo's devbox/Nix packages (nodejs, yarn-berry, jq, cocoapods, etc.) were still being fetched straight from cache.nixos.org, unfiltered. Adopts the same mechanism already shipped in segmentio/ajs-renderer (#515): upstream devbox hardcodes cache.nixos.org as its fetchClosure source and ignores any override, so a patched devbox binary is built from source (same patch ajs-renderer uses - devbox's own DEVBOX_NIX_BINARY_CACHE override, proposed upstream at jetify-com/devbox#2891, not yet merged) and installed over whatever devbox-install-action just put on PATH, immediately after every existing "Install devbox" step in ci.yml and release.yml. ajs-renderer's version of this involves a whole custom CI base Docker image, an apt-installed nix-bin, and a disabled flake registry - that complexity exists specifically to work around Twilio Buildkite's locked-down egress. GitHub Actions runners have open internet, so none of that bootstrap is needed here - just the patch + nix.conf substituter + DEVBOX_NIX_BINARY_CACHE. Verified locally before pushing: - the patch applies cleanly to devbox 0.16.0, the version this repo's own devbox.lock nixpkgs-unstable pin (afce96367b) already provides - the patched expression actually builds (nix build --impure --expr), and the resulting binary's compiled string table contains DEVBOX_NIX_BINARY_CACHE, confirming the patch compiled in rather than silently no-op'ing NOT verified (needs a real CI run, flagging explicitly rather than claiming more confidence than earned): - whether segmentio's GitHub-Actions-OIDC-authenticated identity (or anonymous access) can actually reach https://otk.artifactory.twilio.com/artifactory/remote-nix-cache-nixos - this is a different Artifactory repo than the npm proxy, onboarded separately, and I found no IPD doc confirming general segmentio access to it the way trusted_builds covers npm - whether that substituter actually serves the specific package/version closures devbox.lock resolves (nodejs 22, yarn-berry, cocoapods, jq, treefmt, nixfmt, shfmt) - if any are missing, that job falls back to a local build instead of a hard failure, but will be much slower
1 parent 0f4dbda commit 3fb51ae

4 files changed

Lines changed: 214 additions & 0 deletions

File tree

‎.github/actions/devbox-artifactory-cache/action.yml‎

Lines changed: 54 additions & 0 deletions

‎.github/actions/devbox-artifactory-cache/configurable-nix-binary-cache.patch‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
diff --git a/internal/devpkg/narinfo_cache.go b/internal/devpkg/narinfo_cache.go
2+
index e39fd2a..5c79300 100644
3+
--- a/internal/devpkg/narinfo_cache.go
4+
+++ b/internal/devpkg/narinfo_cache.go
5+
@@ -15,15 +15,29 @@
6+
"github.com/pkg/errors"
7+
"go.jetify.com/devbox/internal/debug"
8+
"go.jetify.com/devbox/internal/devbox/providers/nixcache"
9+
+ "go.jetify.com/devbox/internal/envir"
10+
"go.jetify.com/devbox/internal/goutil"
11+
"go.jetify.com/devbox/internal/lock"
12+
"go.jetify.com/devbox/internal/nix"
13+
"golang.org/x/sync/errgroup"
14+
)
15+
16+
-// binaryCache is the store from which to fetch this package's binaries.
17+
-// It is used as FromStore in builtins.fetchClosure.
18+
-const binaryCache = "https://cache.nixos.org"
19+
+// defaultBinaryCache is the public Nix binary cache that Devbox queries by
20+
+// default for prebuilt package outputs.
21+
+const defaultBinaryCache = "https://cache.nixos.org"
22+
+
23+
+// BinaryCache is the store from which to fetch a package's binaries. It is
24+
+// used as the FromStore in builtins.fetchClosure and as the first cache
25+
+// queried when checking whether a package's outputs are already built.
26+
+//
27+
+// It defaults to the public cache (https://cache.nixos.org) but can be
28+
+// overridden with the DEVBOX_NIX_BINARY_CACHE environment variable. This is
29+
+// useful in network-restricted environments where the public cache is
30+
+// unreachable and an internal mirror/proxy serving the same store paths must
31+
+// be used instead.
32+
+func BinaryCache() string {
33+
+ return envir.GetValueOrDefault(envir.DevboxNixBinaryCache, defaultBinaryCache)
34+
+}
35+
36+
// useDefaultOutputs is a special value for the outputName parameter of
37+
// fetchNarInfoStatusOnce, which indicates that the default outputs should be
38+
@@ -320,7 +334,7 @@ func(o *s3.Options) {
39+
var nixCacheIsConfigured = goutil.OnceValueWithContext(nixcache.IsConfigured)
40+
41+
func readCaches(ctx context.Context) ([]string, error) {
42+
- cacheURIs := []string{binaryCache}
43+
+ cacheURIs := []string{BinaryCache()}
44+
if !nixCacheIsConfigured.Do(ctx) {
45+
return cacheURIs, nil
46+
}
47+
diff --git a/internal/envir/env.go b/internal/envir/env.go
48+
index 5cf0c37..382ec20 100644
49+
--- a/internal/envir/env.go
50+
+++ b/internal/envir/env.go
51+
@@ -6,6 +6,13 @@
52+
const (
53+
DevboxCache = "DEVBOX_CACHE"
54+
DevboxGateway = "DEVBOX_GATEWAY"
55+
+ // DevboxNixBinaryCache overrides the default Nix binary cache that Devbox
56+
+ // queries for prebuilt package outputs (https://cache.nixos.org). This is
57+
+ // useful in network-restricted environments where the public cache is
58+
+ // unreachable and an internal mirror/proxy must be used instead. The value
59+
+ // should be a substituter URL serving the same store paths (e.g. an
60+
+ // Artifactory/Nexus generic remote that mirrors cache.nixos.org).
61+
+ DevboxNixBinaryCache = "DEVBOX_NIX_BINARY_CACHE"
62+
// DevboxLatestVersion is the latest version available of the devbox CLI binary.
63+
// NOTE: it should NOT start with v (like 0.4.8)
64+
DevboxLatestVersion = "DEVBOX_LATEST_VERSION"
65+
diff --git a/internal/shellgen/generate.go b/internal/shellgen/generate.go
66+
index 838152f..16c3052 100644
67+
--- a/internal/shellgen/generate.go
68+
+++ b/internal/shellgen/generate.go
69+
@@ -17,6 +17,7 @@
70+
"github.com/pkg/errors"
71+
"go.jetify.com/devbox/internal/cuecfg"
72+
"go.jetify.com/devbox/internal/debug"
73+
+ "go.jetify.com/devbox/internal/devpkg"
74+
"go.jetify.com/devbox/internal/redact"
75+
)
76+
77+
@@ -161,9 +162,41 @@ func toJSON(a any) string {
78+
}
79+
80+
var templateFuncs = template.FuncMap{
81+
- "json": toJSON,
82+
- "contains": strings.Contains,
83+
- "debug": debug.IsEnabled,
84+
+ "json": toJSON,
85+
+ "contains": strings.Contains,
86+
+ "debug": debug.IsEnabled,
87+
+ "fetchClosureStore": fetchClosureStore,
88+
+ "nixString": nixString,
89+
+}
90+
+
91+
+// fetchClosureStore returns the store URL to use as the fromStore of a
92+
+// builtins.fetchClosure in the generated flake.
93+
+//
94+
+// fetchClosure only supports http(s) stores, so when a package's outputs were
95+
+// found in an http(s) cache we use that cache directly (it's where the path
96+
+// actually lives, which may be the public cache or a configured mirror via
97+
+// DEVBOX_NIX_BINARY_CACHE). For s3 caches fetchClosure can't fetch directly, so
98+
+// we fall back to the configured binary cache as a placeholder store — the path
99+
+// is already built locally, so fetchClosure won't actually fetch from it.
100+
+func fetchClosureStore(cacheURI string) string {
101+
+ if strings.HasPrefix(cacheURI, "http://") || strings.HasPrefix(cacheURI, "https://") {
102+
+ return cacheURI
103+
+ }
104+
+ return devpkg.BinaryCache()
105+
+}
106+
+
107+
+// nixString renders s as a double-quoted Nix string literal, escaping the
108+
+// characters that are special inside one: backslash, double quote, and the
109+
+// "${" antiquotation (interpolation) start. This keeps values that originate
110+
+// from configuration (e.g. a cache URL from DEVBOX_NIX_BINARY_CACHE) from
111+
+// breaking flake evaluation or being interpreted as Nix interpolation.
112+
+func nixString(s string) string {
113+
+ r := strings.NewReplacer(
114+
+ `\`, `\\`,
115+
+ `"`, `\"`,
116+
+ `${`, `\${`,
117+
+ )
118+
+ return `"` + r.Replace(s) + `"`
119+
}
120+
121+
func makeFlakeFile(d devboxer, plan *flakePlan) error {
122+
diff --git a/internal/shellgen/tmpl/flake.nix.tmpl b/internal/shellgen/tmpl/flake.nix.tmpl
123+
index 3bd0513..cbe1321 100644
124+
--- a/internal/shellgen/tmpl/flake.nix.tmpl
125+
+++ b/internal/shellgen/tmpl/flake.nix.tmpl
126+
@@ -42,13 +42,16 @@
127+
{{ if $output.CacheURI -}}
128+
(builtins.trace "downloading {{ $pkg.Versioned }}" (builtins.fetchClosure {
129+
{{/*
130+
- HACK HACK HACK! fetchClosure only supports http(s) caches and not
131+
- s3 caches. Until we implement that, we put a fake store here.
132+
- Since we pre-build everything, fetchClosure will not actually
133+
- fetch anything and just use the local version. This may break
134+
- if user somehow removes the local store path.
135+
+ fetchClosure only supports http(s) caches and not s3 caches.
136+
+ fetchClosureStore returns the http(s) cache the output was found
137+
+ in (the public cache or a configured mirror via
138+
+ DEVBOX_NIX_BINARY_CACHE), falling back to the default binary
139+
+ cache for s3 caches. Since we pre-build everything, fetchClosure
140+
+ will not actually fetch anything for an s3 cache and just use the
141+
+ local version. This may break if the user somehow removes the
142+
+ local store path.
143+
*/}}
144+
- fromStore = "https://cache.nixos.org";
145+
+ fromStore = {{ fetchClosureStore $output.CacheURI | nixString }};
146+
fromPath = "{{ $pkg.InputAddressedPathForOutput $output.Name }}";
147+
inputAddressed = true;
148+
}))

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ jobs:
1818
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
1919
- name: Install devbox
2020
uses: jetify-com/devbox-install-action@a0d2d53632934ae004f878c840055956d9f741b0 # v0.14.0
21+
- name: Route Nix/devbox packages through Artifactory
22+
uses: ./.github/actions/devbox-artifactory-cache
2123
- name: Install dependencies
2224
run: devbox run ci:install
2325
- name: Lint (eslint)
@@ -39,6 +41,8 @@ jobs:
3941
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
4042
- name: Install devbox
4143
uses: jetify-com/devbox-install-action@a0d2d53632934ae004f878c840055956d9f741b0 # v0.14.0
44+
- name: Route Nix/devbox packages through Artifactory
45+
uses: ./.github/actions/devbox-artifactory-cache
4246
- name: Install dependencies
4347
run: devbox run ci:install
4448
- name: Validate PR title

‎.github/workflows/release.yml‎

Lines changed: 8 additions & 0 deletions

0 commit comments

Comments
 (0)