Skip to content
Try Gitea Cloud ☁️ for 30 days → Accelerate your Development & Deploys!
Support

Storage and Caching

Runner pods keep their caches on persistent volumes, which outlive the pods: when a runner pod is removed its volumes are kept, and when the runner pod is recreated it mounts them again, with the images and actions it downloaded before. This page describes those volumes, how long they live, and the workflow cache.

  • Dynamic provisioning of ReadWriteOnce volumes. The volumes are created as runner pods are added, so a StorageClass must provision them on demand. Volumes created in advance by hand are not supported.
  • allowVolumeExpansion on the StorageClass is recommended, so that the volumes can grow in place later, see Changing StorageClass or size.
  • volumeBindingMode: WaitForFirstConsumer is recommended, so that a volume is created where its pod is scheduled.
  • Volumes that can be attached on another node. A runner pod recreated after a scale-down mounts its previous volumes, on whichever node it is scheduled. Node-local provisioners such as local-path-provisioner pin each runner to one node, and the runner cannot start again once that node is gone. Use them for single-node trials only.
  • Topology. Volumes bound to a zone, or to a subset of the nodes, keep their runner pods there. If the cluster adds nodes on demand, the component that does so must take the topology of the volumes into account, or pods stay Pending with volume node affinity conflict. If it cannot, keep the runner pods in one zone with nodeSelector or affinity.

Each backend adds its own requirements: the Docker data root of the Docker-in-Docker backend must be on block storage, and the volumes of the Kubernetes backend must support fsGroup.

Check the StorageClass with:

Terminal window
kubectl get storageclass

ALLOWVOLUMEEXPANSION should preferably be true, and VOLUMEBINDINGMODE WaitForFirstConsumer.

Volume Claim name Mounted at Holds Size StorageClass
Docker data root (Docker-in-Docker only) <runner pod>-cache dockerStorageMountPath, /var/lib/docker by default Images and layers pulled by Docker, job workspaces cacheVolumeSize, 100Gi dockerStorageClassName, else className
Action cache <runner pod>-action-cache /cache The code of the actions jobs use, and the staging area of S3 caching actionCacheVolumeSize, 10Gi actionCacheClassName, else className

The action cache holds the actions that jobs use with uses:, so they are not downloaded again for every job. The workflow cache is separate, see Workflow cache.

className is the StorageClass named standard by default, not the default StorageClass of the cluster. Set it, or the more specific fields, to a StorageClass that exists: without one, the claims stay Pending.

With the Docker-in-Docker backend, everything a job container writes, such as its workspace and the tools that setup-* actions install, is written to the Docker data root. The action cache only holds what the runner itself downloads, which is light on I/O. Therefore, the action cache can be on network storage, while the Docker data root should be on fast block storage.

Plan the storage for the peak, not the average:

total ≈ maxSize × (Docker data root + action cache) + cache server
  • The default sizes are large. The Docker data root is 100Gi for each runner pod and the cache server 500Gi. Size them to the jobs.
  • Volumes are not freed when scaling down. The volumes of a runner are kept for storage.cacheTTL after their last use, a week by default. A Runner that once scaled to maxSize keeps that storage for a week.
  • With S3 caching there is no cache server volume, but the action cache also stages each cache before upload.
storage:
cacheTTL: 168h # deleted this long after the last use, 168h by default
cacheMaxAge: 720h # recreated empty this long after creation, 720h by default
  • The volumes of a runner are deleted once they have not been used for cacheTTL. While its runner pod runs, they are in use and kept. The volume of the built-in cache server follows cacheTTL too, and is deleted once the cache server has been unused for that long, for example after scaling to 0 or switching to another cache.
  • cacheMaxAge caps the total age of the volumes of a runner pod. A volume that reaches it is deleted and created again empty, even if the runner is always in use, like the runners kept by minSize. Such a runner pod is rebuilt with the new volume once it is idle, one runner pod of a Runner at a time, with a CachePVCExpired event. A larger cacheTTL therefore does not keep a cache forever. The cache server volume is not affected, as the cache server expires its own entries.
  • The registrations of the runners are kept in Secrets that do not expire.
  • Two cases delete the volumes of a runner immediately. Lowering maxSize deletes those of the runners above it, along with their registrations. A runner pod that does not come online is deleted with its volumes, see Runner pods that do not come online.

The StorageClass of a volume cannot be changed, and Kubernetes cannot shrink a volume. To avoid emptying caches by accident, the controller only applies such a change when storage.allowDeleteCachePVC is true:

Change allowDeleteCachePVC: false (default) allowDeleteCachePVC: true
Larger size, StorageClass allows expansion Expanded in place (CachePVCExpand) Expanded in place (CachePVCExpand)
Larger size, StorageClass does not allow expansion Ignored, CachePVCExpandUnsupported warning Volume deleted and created again, empty (CachePVCRecreate)
Smaller size Ignored, CachePVCShrinkIgnored warning Volume deleted and created again, empty (CachePVCRecreate)
Different StorageClass Ignored, CachePVCClassChangeIgnored warning Volume deleted and created again, empty (CachePVCRecreate)

The volumes of a runner pod are recreated once it is idle, as for any configuration change. The volume of the built-in cache server is recreated immediately, together with its pod.

When a volume is deleted, the StorageClass decides whether the storage behind it is freed. With a Retain policy, deleted volumes leave Released persistent volumes behind. To free them, set:

storage:
reclaimPolicy: Delete # or Retain; unset keeps the policy of the StorageClass

The controller then sets this policy on the persistent volumes provisioned for the Runner, but only while a volume is bound and in use by a runner pod or the cache server. The volumes of runners already scaled down are not in use, and keep the policy of the StorageClass when the Runner is deleted. Therefore, set it when you create the Runner, not just before deleting it. Statically provisioned volumes are never changed.

The workflow cache, which actions/cache and actions such as setup-go and setup-node use, needs a cache server. There are three ways to provide it, and S3-compatible storage is recommended for production:

  • The cache is kept in a bucket, so neither the pod of the built-in cache server nor its volume, 500Gi by default, is needed.
  • Capacity grows with the bucket, without planning or resizing a cache server volume.
  • Several Runner resources can share a cache by using the same prefix, while the built-in cache server serves a single Runner.

Read the notes in S3-compatible storage before using it. The built-in cache server needs no configuration, which suits a trial.

S3-compatible storage (recommended) Built-in cache server (default) External cache server
Configured with cacheServer.s3 Nothing cacheServer.externalServerURL
Where the cache lives A bucket A volume of a pod the controller creates Your cache server

The cache traffic of jobs goes through the runner, so the cache server only needs to be reachable from the runner pods.

The cache can live in a bucket of an S3-compatible service, such as AWS S3, MinIO or RustFS. The runners then read and write the bucket directly, the controller creates no cache server, and runners with the same bucket and prefix share their cache.

Create a Secret with the access key:

Terminal window
kubectl -n <namespace> create secret generic runner-cache-s3 \
--from-literal=access-key=<access key ID> \
--from-literal=secret-key=<secret access key>
cacheServer:
s3:
endpoint: "https://s3.us-west-2.amazonaws.com" # starts with http:// or https://
region: "us-west-2" # see below
bucket: "runner-cache"
prefix: "" # <namespace>/<name> of the Runner when empty
# pathStyle: true
accessKeyRef:
name: runner-cache-s3
key: access-key
secretKeyRef:
name: runner-cache-s3
key: secret-key
  • region is required, and signs every request. On AWS it is the region of the bucket. MinIO and RustFS usually use us-east-1, Cloudflare R2 uses auto.
  • pathStyle can usually be left unset: the runner then addresses buckets on amazonaws.com by host name and others by path. Set it to true on AWS for a bucket name with dots, and to false for a service that only supports host names.
  • Set lifecycle rules on the bucket. The runners never delete caches, and storage.cacheTTL does not apply to the bucket. Add at least a rule that expires the objects under the prefix after some days, for example 7, and one that aborts incomplete multipart uploads, for example after 1 day. Expiry counts from the creation of an object, so caches in use also expire and are created again on the next run. Unlike the built-in cache server, the bucket keeps every cache saved within the expiry, including those replaced by newer ones, and has no size limit per repository. Keep the expiry short.
  • Size the action cache for the largest cache. A runner writes each cache in full to its action cache volume before uploading it, and a multipart upload briefly takes about twice that while its parts are combined. The action cache should hold the largest single cache twice, plus the downloaded actions. The default 10Gi fits single caches of up to about 4GiB. When it runs out of space the cache is not saved, and the job is not affected. On network storage, saving caches is noticeably slower.
  • Only static access keys are supported, not IRSA, EKS Pod Identity or instance roles. The keys need s3:ListBucket on the bucket, which a condition can limit to the prefix, and s3:GetObject, s3:PutObject and s3:DeleteObject on the objects under the prefix. s3:AbortMultipartUpload is optional: it lets a runner clean up a failed upload, which the lifecycle rule does otherwise. Changing the Secret, for example to rotate the keys, rebuilds the runner pods so that they read the new keys.
  • With the Docker-in-Docker backend, including jobs that run on the host, job code may be able to read the access keys. Grant only the permissions above, limited to the prefix, and restrict the keys to the network of the cluster where the storage service allows it. Use separate keys for separate trust domains, such as Runner resources of different Gitea instances.
  • The runner verifies the certificate of the endpoint with the certificates of its image, or the bundle in SSL_CERT_FILE as in the init container example. gitea.skipTlsVerify does not apply.
  • Only the runners access the bucket, the job containers do not.
  • The default prefix differs per Runner. Runner resources with the same prefix share the cache. Caches are told apart by repository name only, not by Gitea instance, so Runner resources of different Gitea instances should not share a prefix.
  • s3 cannot be combined with externalServerURL. host and port have no effect with it.
  • Caches are not moved when switching from the built-in cache server. Until its volume is deleted after storage.cacheTTL, removing s3 switches back to the previous cache.

By default, the controller creates a cache server for each Runner: a pod named <name>-cache-server with a Service, a Secret holding the key the runners use to authenticate, and a volume. The cache is kept on the volume <name>-cache-server-data, of storage.cacheServerVolumeSize, 500Gi by default, and of the StorageClass className. Size it to the cache use of the repositories. With S3, an external cache server or your own runner configuration, neither the cache server nor its volume is created.

cacheServer:
# host: cache.example.internal # <name>-cache-server-svc.<namespace>.svc.cluster.local when unset
# port: 8081 # 80 by default, 8080 with the Kubernetes backend, where it must be 1024 or above
  • The runners reach it by the host name of its Service, which assumes the cluster domain cluster.local. On a cluster with another domain, set host, or the workflow cache misses every time without failing the job.
  • While a Runner has no runner pods, the pod and Service of the cache server are removed, and its volume is kept for storage.cacheTTL.
  • It runs the runner image, with no resource requests or limits.
  • The runners authenticate to it with a key that the controller generates and keeps in the Secret <name>-cache-server-secret, mounted into the runner pods as a file.
  • To change the key, delete that Secret. The controller generates a new key, rebuilds the cache server immediately, and then the runner pods as for any configuration change. Until a runner pod is rebuilt, the cache reads and writes of its jobs fail, which jobs treat as a cache miss.
  • When its configuration changes, for example the runner image or the port, the cache server pod is rebuilt immediately, and its data stays on the volume. Cache reads and writes in flight fail once, which jobs treat as a cache miss.
  • Its volume follows cacheServerVolumeSize and className like the volumes of the runner pods.
  • After switching to an external cache server, S3 or runnerConfigMapName, the built-in cache server is kept until every runner pod uses the new configuration, so running jobs are not affected, and is then deleted. Its volume is deleted after storage.cacheTTL.

To use a cache server of your own, such as one started with gitea-runner cache-server, set its URL and the shared key. The controller then creates no built-in cache server.

cacheServer:
externalServerURL: "http://cache.example.com/"
externalSecretRef:
name: cache-server-secret
key: secret

externalSecretRef reads the key from a Secret in the namespace of the Runner. The deprecated externalSecret takes the key as plain text instead, and is written into the generated configuration. Only one of the two can be set.

  • When the Secret or its key is missing or empty, the controller creates no runner pod and reports a BuildRunnerWorkerFailed warning that mentions cache server external secret.
  • Changing the Secret, for example to rotate the key, rebuilds the runner pods so that they read the new key.