{{ message }}
Repository navigation
cmd/go: clean GOCACHE based on disk usage #29561
Description
The GOCACHE appears to lack a disk size limit - this is a problem in a space constrained environment and/or when running go on a disk that is nearing capacity. For example, on the openbsd/arm builder (which runs on a USB stick), the ~/.cache/go-build directory runs past several GB in a very short time, which then leads to various failures (git clone or go builds). The only option that I currently appear to have is to run go cache -clean regularly, in order to keep the cache at a respectable size. It seems that having a configurable upper bound would be preferable and/or free disk space based checks that prevent writes (e.g. evict then write) to the cache from failing when the disk is full due, partly due to a large GOCACHE:
[gopher@cubox ~ 102]$ du -csh ~/.cache/go-build
2.2G /home/gopher/.cache/go-build
2.2G total
[gopher@cubox ~ 103]$ df -h /home
Filesystem Size Used Avail Capacity Mounted on
/dev/sd1m 9.0G 8.5G -6.0K 100% /home
[gopher@cubox ~ 104]$ go build -o /tmp/main /tmp/main.go
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
/home: write failed, file system is full
[gopher@cubox ~ 105]$ du -csh ~/.cache/go-build
2.2G /home/gopher/.cache/go-build
2.2G total
[gopher@cubox ~ 106]$ df -h /home
Filesystem Size Used Avail Capacity Mounted on
/dev/sd1m 9.0G 8.5G -6.0K 100% /home
Activity
jayconrod commentedon Jan 4, 2019
I think this would be a good thing to have, especially since the cache will be required starting in Go 1.12. I don't think there's a fixed maximum cache size that would work for everyone, but maybe we could at least make eviction more aggressive when the disk is low on space.
mvdan commentedon Jan 4, 2019
We're no longer allowing
GOCACHE=offstarting in 1.12, so it might be a bit tricky to allow an arbitrary size limit. One could simply swapGOCACHE=offwithGOCACHELIMIT=0, for example.I seem to remember that @rsc designed the cache to automatically evict based on modification time. Seems sound to me to also automatically evict if the disk is full enough that it might realistically hit errors if it builds a few large packages (say, if there's less than 500MB left).
bitfield commentedon Jan 5, 2019
This sounds like a bit of a can of worms. How do you check for remaining disk space in a cross-platform way? How much free space should trigger a cache eviction? How do you guarantee that, having evicted some cache, the build process won't still run out of disk anyway?
If you're building on a small disk, then wouldn't running
go cache -cleanbefore each build be a better solution than trying to build disk space management into the build tool itself?mvdan commentedon Jan 5, 2019
I personally don't know how easy it would be to control "does the disk have enough space left" in a sane and portable way. I was simply pointing out that since the current eviction algorithm takes into account timestamps, perhaps it should also use some data from the filesystem or disk. If we can make that work, we could avoid adding more knobs to the go tool.
That might not be a great solution - what if you're building a very large project like Kubernetes? It might produce many hundreds of megabytes of build cache, so it's not unreasonable to think that it could on its own be enough to fill up some filesystems.
bitfield commentedon Jan 5, 2019
Indeed. So this proposal would, at best, kick the problem slightly further down the road.
[-]cmd/go/internal/cache: GOCACHE appears to lack disk size limit[/-][+]cmd/go: clean GOCACHE based on disk usage[/+]31 remaining items
jakebailey commentedon Nov 16, 2025
My team has been hitting this repeatedly, especially now with things like agent mode or similar where an LLM can be churning away trying things, or even just a normal dev scenario with a build in a watch mode. I can myself produce something like 200GB of files in the build cache a day. I have the space to not run out, but others on my team have less space and can actually fill up their drives.
In CI, I have a build cache set up for the typescript-go repo that keeps the most recent build main for other jobs, and that cache for just one build is some 11GB on disk (big enough that it runs out of space in GHA runners because they have so little free space).
It would be nice if there were some way to have this pruned more aggressively.
Perhaps our repo is just super bad when it comes to cache sizes due to one of the issues linked above, too.
seankhliao commentedon Nov 16, 2025
Now that caching is pluggable via a cacheprog: https://pkg.go.dev/cmd/go/internal/cacheprog
I think it's something people can experiment with to manage their caches in different ways (offloading to shared storage in ci, or just pruning more aggressively locally)
josharian commentedon Nov 16, 2025
It’s good that people can mess around with cacheprog, but that is a stopgap at best.
The existence of tools to fix your tools does not mean that your core tools don’t need to work well out of the box.
We should not ask users to manage this themselves.
seankhliao commentedon Nov 16, 2025
I mentioned it more as an easier way people experiment / prove that there's a good / cross platform algorithm for managing the cache size, since the thread seems to be a lot of ideas and complaints, but very little in the way data on what would actually work.
jakebailey commentedon Nov 16, 2025
Yes, I plan to try and build a GOCACHEPROG for this, both for the cleaning, but also because I've found that the actual cached files themselves are super compressible. One file that's part of the cache for TypeScript's AST package is 200MB, but compresses with zstd to just 14MB!
I may have avoided this problem on my own personal machines just because I use btrfs with transparent zstd compression. My work machines do not do this.
josharian commentedon Nov 16, 2025
Oooooh, transparent zstd is a fascinating idea.
josharian commentedon Nov 16, 2025
Also, because GitHub ever-so-helpfully hides comments, I'm going to repeat in full here something Russ wrote above, for visibility.
jakebailey commentedon Nov 16, 2025
Unfortunately, it feels like
GOCACHEPROGcannot be used to do this since the protocol expectsgetto have written files to disk. So, no compression there; it would have to be done at the places the Go toolchain caches things. And it doesn't seem like the stdlib has zstd compression, only decompression. At best, aGOCACHEPROGcould compress its own assets.For actually managing the lifetime of assets, it's not clear how a
GOCACHEPROGcan achieve this either without a lot of complication, given the requirement to write files back to disk. Since aGOCACHEPROGexe can be run from multiple concurrent unrelated builds, you basically end up redoing everything the Go disk cache already does to make that work, at least in my mind.I think it's probably the best situation if the built in disk cache got some sort of generational caching as linked above, and then some sort of compression where possible.
josharian commentedon Nov 16, 2025
#62513 Is about adding zstd support to std in general. I believe @klauspost to be actively working on it (?). Nice to have this as a possible internal use case for it.
jakebailey commentedon Nov 17, 2025
gzip seems to do alright at 19MB for my example, though in quite literally 20x more time (70ms vs 1.4s!).
But, somewhat offtopic to the cleanup issue. I was just hoping that making the cache in general 10x+ smaller would help with this problem 😄
AlekSi commentedon Nov 17, 2025
This discussion prompted me to finally cleanup our internal tool and publish it: https://github.com/AlekSi/hardcache
4a6f656c commentedon Jan 1, 2026
Still hit this on a regular basis...
bysiber commentedon Mar 2, 2026
This is one of those issues that seems niche but affects basically every Go developer over time. The build cache just keeps growing with no upper bound, and
go clean -cacheis too aggressive iut wipes everything.On macOS with a 256GB or 512GB SSD, GOCACHE can quietly eat 5-10GB over a few months of active development. Adding GOMODCACHE on top of that and you're looking at 15GB+ of Go-related caches.
What I've been doing until this gets addressed:
For macOS users dealing with this: ClearDisk is a free open-source menu bar tool that monitors GOCACHE, GOMODCACHE, and
~/go/pkg/modalongside other developer caches (Rust target/, node_modules, Docker, Xcode DerivedData, pip). It shows exactly how much space each cache uses so you can make informed cleanup decisions.Having the
go cleancommand support something like--max-sizeor--older-thanwould be great. The current all-or-nothing approach makes developers hesitant to clean anything, so caches just grow forever.jakebailey commentedon Jun 26, 2026
I forgot I wrote this, and in the context of #76337 I actually wrote a GOCACHEPROG to do this to a reasonable effect: https://github.com/jakebailey/gocachez
Part of it is that it removes old files automatically up to a configurable size, which is harder to reach with compression as well.
Cap the Go build cache at 20G