https://vivax.dev Viv's blog about software engineering Zola en Sat, 28 Mar 2026 00:00:00 +0000 Sat, 28 Mar 2026 00:00:00 +0000 Unknown https://vivax.dev/blog/ci-caching/ https://vivax.dev/blog/ci-caching/ <p>I have been writing complex integration suites for my projects for a while, the most complex by far being the one for <a rel="external" href="https://github.com/serpent-Tools/natrix">Natrix</a>, which involves running around 4 different kinds of integration tests, alongside two unit test suites, and a whole range of linters.</p> <p>And through all of that I jumped between a few different tools for this, raw github actions, dagger, and eventually writing my own. There were many ergonomics issues and similar with the previous tools that didn't work for my kind of integration tests, but the thing that was ultimately the biggest limiting factor was caching, especially in github runners.</p> <h2 id="caching-should-be-invisible">Caching should be invisible.</h2> <p>This is, in my opinion, one of the most important aspects of a good cache. The only signs that a cache was used should be lower execution time, and less log output. There should be no way to tell that the cache was used based on the final output. And it's at this point a lot of edge cases pop up, but we won't get into them here.</p> <p>If you in your user code of a system ever have to write <code>if is_cached(): ...</code> something has gone very wrong with the cache design in said system. As I said before, at its core caching is an optimization, and optimizations shouldn't affect semantics.</p> <p>Naturally this stuff can never be fully perfect, for example docker assumes any step can be cached, but ultimately <code>apt install ...</code> can't be correctly cached because it talks to an external system, in general stuff that talks to external systems is harder to cache correctly, but in general we bite the bullet and do so anyway because in most cases it's okay, especially if you pin versions. In other words making caches invisible isn't just on the cache implementation itself but also on its users to write code that is able to be cached.</p> <blockquote class="markdown-alert-note"> <p>I am speaking from the perspective of most caches one touches during development work, in other domains this constraint might not hold.</p> </blockquote> <h3 id="what-should-we-cache">What should we cache?</h3> <p>In general this is a question that has been solved before, if you are looking at your CI and wondering what to cache, a good first bet is to cache the caches. What I mean is, most build systems maintain caches of artifacts built to avoid work, you certainly notice this locally. So a good first candidate for caching in your CI, is your build system's caches, because those are already designed to be persisted between runs. And here is a very important abstraction boundary to consider when it comes to caching in CI, which comes in two flavors.</p> <ol> <li>Caching info so that you can explicitly skip steps on cache hit.</li> <li>Caching a tools cache so it can skip steps automatically for you.</li> </ol> <p>The second one is almost always better, because you can then entrust a widely tested tool to have figured out the cache invalidation for you, your job is just persisting its cache between runs. This is the path cargo, npm, docker, lend themselves to.</p> <h2 id="caching-in-github-actions">Caching in GitHub Actions</h2> <p><a rel="external" href="https://docs.github.com/en/actions">GitHub Actions</a> are maybe one of the most popular CI platforms, and as such it naturally has a caching story.</p> <h3 id="actions-cache"><a rel="external" href="https://github.com/actions/cache"><code>actions/cache</code></a></h3> <p>GitHub's official cache action is amazing for caching caches like we talked about above, this action lets you very easily store and restore a given folder or file, which is exactly what you need for caching build caches. It's easy to use, and when given a good cache key will mostly just work. So whenever what you need caching is nicely exposed as a well documented folder or file the solution is simple and easy.</p> <h3 id="swatinem-rust-cache"><a rel="external" href="https://github.com/Swatinem/rust-cache"><code>swatinem/rust-cache</code></a></h3> <p>Now sometimes it's nice to have abstractions that hide what folders and files to hide, and use domain knowledge to make the cache smaller. For example this rust cache strips out unneeded data from <code>./target</code>, but at its core it's still just caching a folder or file, simple easy.</p> <h2 id="caching-docker-layers">Caching docker layers?</h2> <p>What if your CI builds docker images? Well as you might know docker does cache layers, but where? Well docker stores its metadata in a few internal places, and trying to cache these would be a losing battle, luckily docker uses buildkit, which does expose some <a rel="external" href="https://docs.docker.com/build/cache/backends/">options</a> here:</p> <h3 id="local-files"><a rel="external" href="https://docs.docker.com/build/cache/backends/local/">Local Files</a></h3> <p>Docker supports exporting its cache to a local file, or more correctly <a rel="external" href="https://docs.docker.com/build/buildkit/">buildkit</a> does, and docker exposes wrapper options for it.</p> <blockquote> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">docker</span><span style="color: #A6E3A1;"> buildx build --push -t</span><span style="color: #94E2D5;"> &lt;</span><span style="color: #A6E3A1;">registr</span><span>y</span><span style="color: #94E2D5;">&gt;</span><span style="color: #A6E3A1;">/</span><span style="color: #94E2D5;">&lt;</span><span style="color: #A6E3A1;">imag</span><span>e</span><span style="color: #94E2D5;">&gt;</span><span style="color: #F5C2E7;"> \</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> --cache-to type=local,dest=path/to/local/dir[,parameters...]</span><span style="color: #F5C2E7;"> \</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> --cache-from type=local,src=path/to/local/dir .</span></span></code></pre></blockquote> <p>And now you can use <code>actions/cache</code> to save and restore this folder to gain the benefit of dockers layer caching in your CI, or in any CI for that matter.</p> <h3 id="github-actions-cache"><a rel="external" href="https://docs.docker.com/build/cache/backends/gha/">GitHub Actions Cache</a></h3> <p>Buildkit/docker also supports talking to the GitHub Actions cache directly, this is easiest done via the <a rel="external" href="https://github.com/docker/build-push-action">docker/build-push-action</a> action as it will ensure the proper GitHub tokens are set in the docker arguments etc. This is more efficient because instead of needing to save and restore the entire cache to and from the local runner, this action will lazily pull layer data as it gets cache hits, and only upload the new layers.</p> <h2 id="dagger"><a rel="external" href="https://dagger.io/">Dagger</a></h2> <p>Now GitHub Actions is all fine and good, until you want to run CI locally as well, here we have some interesting tools, one I really enjoyed for a while was Dagger. Its programming based interface for docker was really nice to work with, and let me do some stuff I had been wanting to express in docker files for a while. Eventually I ran into some UX issues that drove me away, but their core tool is really unique and cool!</p> <p>Now let's look at their caching story, at its core it's essentially the same caching system as buildkit and docker, because well dagger uses buildkit internally. Locally all works well, now since it's based on buildkit in theory the CI caching story should be the same!</p> <p>To quote their <a rel="external" href="https://docs.dagger.io/getting-started/ci-integrations/github-actions">docs</a>:</p> <blockquote> <p>Dagger has also partnered with Depot to provide managed, Dagger Powered GitHub Actions runners. These runners, which serve as drop-in replacements for GitHub's own runners, come with Dagger pre-installed and pre-configured to best practices, <em>automatic persistent layer caching</em>, and multi-architecture support. They make it faster and easier to run Dagger pipelines in GitHub repositories.</p> </blockquote> <p>Oh, well I guess they need to make money somehow, that's fair! Let's just find where they talk about gha or local file based caches so we can make this work on GitHub runners... okay nothing in the main docs, let's check the docs for <a rel="external" href="https://github.com/dagger/dagger-for-github">dagger/dagger-for-github</a>, hmm nothing there. Okay for these, apparently very niche use cases, GitHub issues is the goto! Aha! found it, <a rel="external" href="https://github.com/dagger/dagger-for-github/issues/39">#39</a></p> <blockquote> <p>jpadams closed this as not plannedon Feb 13, 2023</p> </blockquote> <p>Oh... well I guess moving on then.</p> <h2 id="how-about-my-tool">How about my tool?</h2> <p>I am developing my own workflow runner called <a rel="external" href="https://github.com/Serpent-Tools/serpentine">Serpentine</a>, let's take a look at how it handles caching in CI.</p> <p>By default serpentine stores a <em>non-portable</em> cache in the platforms default cache directory, okay so no just slapping <code>actions/cache</code> on it right away, but it does expose some cli arguments to make the cache portable between systems, <code>--standalone-cache</code>, and naturally also a flag to control the cache location, now we can <code>actions/cache</code> it!</p> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">serpentine</span><span style="color: #A6E3A1;"> run --cache /tmp/serpentine.cache --standalone-cache --ci</span></span></code></pre> <blockquote class="markdown-alert-note"> <p>The reason serpentine's cache isn't portable by default is that it's actually storing most of its data in a <a rel="external" href="https://containerd.io/">containerd</a> daemon, what the standalone flag does is instruct serpentine to export this data to the cache file, which on local only runs would be wasted time.</p> </blockquote> <h2 id="so-whats-the-hardest-thing-in-cs">So whats the hardest thing in CS?</h2> <p>Honestly cache invalidation is hard, I won't deny that, but tools keep getting it right, what we do see big tools still fail at is the user facing <em>cache api design</em>. Caching is an optimization, and if your optimization requires user collaboration it better be easy and clear what they need to do. <code>cargo</code> just asks you to save a target directory, <code>npm</code> asks you to save <code>node_modules</code>, and <code>dagger</code> just asks you to please pay them.</p> <h1 id="further-reading">Further reading</h1> <ul> <li><a rel="external" href="https://en.wikipedia.org/wiki/Cache_(computing)">https://en.wikipedia.org/wiki/Cache_(computing)</a></li> <li><a rel="external" href="https://docs.docker.com/build/cache/">https://docs.docker.com/build/cache/</a></li> </ul>