Skip to content

Backup and recovery

taloscluster stores the cluster's desired state in git and a handful of generated files in the cluster directory that it never commits. Which of those files are irreplaceable, which are derived, and how the cluster's own state is preserved is the difference between recovering a lost machine in minutes and losing the ability to touch a live cluster at all. This page tells you what to back up and how to recover.

The categories, in order of how bad it is to lose them:

What Where it lives Loss is Recoverable?
Provider credentials secrets.yaml plus the cloud account behind it locked out of the cloud re-create the credential and restore the file
Talos identity talossecrets.yaml control of an existing cluster no — restore from backup or rebuild
Client configuration talosconfig, kubeconfig no clients for a still-running cluster re-generated by converge
etcd snapshot a file you write elsewhere cluster control-plane state restore the snapshot
Application data inside the cluster (etcd, volumes) the workloads themselves restore from a Kubernetes backup

Run taloscluster status to see what a cluster currently owns. The shared boot image is not per-cluster (it is keyed by talos.version and the base schematic and can be reused), so it does not need backing up.

What to back up

A cluster directory holds cluster.yaml (your desired state, committed to git), secrets.yaml (provider credentials, mode 0600), talossecrets.yaml (the Talos identity, mode 0600), and the derived client files talosconfig and kubeconfig. taloscluster init writes a .gitignore that keeps secrets.yaml, talossecrets.yaml, talosconfig, and kubeconfig out of git precisely because they hold credentials or identity.

Back these up off the management machine and out of git:

  • secrets.yaml — the OpenStack application credential, the Proxmox API token, and the Tailscale pre-auth key. It is needed to talk to the provider; it is not the cluster's identity. See the OpenStack setup and Proxmox setup pages.
  • talossecrets.yaml — the one file that cannot be regenerated. It holds the cluster CA, the etcd CA, and the join tokens that make the machines part of the same cluster. Converge refuses to mint a new one when the cluster's machines already exist, so restore this from backup before the first converge of a recovered machine. See Talos identity below.
  • cluster.yaml and the git history — the desired state. Even without git, keeping a copy of cluster.yaml lets you rebuild an identical cluster; with git you also keep every previous state.

talosconfig and kubeconfig are derived and safe to regenerate, but they still hold the cluster's client certificate and API credentials, so treat them as secrets too. Keep their backups only if you want clients working before a converge.

An application credential or API token is created and scoped by the cloud, not by taloscluster. secrets.yaml only stores it: if the machine that holds it is lost but the credential still exists in the cloud, restoring secrets.yaml is enough. If the credential itself is gone, rotate a new one in the cloud and update secrets.yaml; the clustered nodes do not depend on it after bootstrap because the Talos identity lives in talossecrets.yaml, not in the provider credential.

Talos identity

talossecrets.yaml is the cluster's cryptographic identity. Losing it means no tool, including taloscluster itself, can authenticate to the existing nodes — the Talos API only accepts mutual TLS with the cluster's own certificates, and the machine configs nodes run were built with these secrets. It cannot be regenerated. This is why converge hard-fails with "restore it from backup" when the file is missing but the cluster's machines already exist, rather than silently minting a fresh identity against a live cluster.

Back up talossecrets.yaml as soon as the first converge writes it (see Day 1). converge mints the identity in the secrets phase, before it creates the network, creates the machines or bootstraps the cluster, and configures the nodes from that same bundle; bootstrap never replaces it. A backup that predates the bootstrap is therefore already the current identity — take it as soon as the file appears in the cluster directory, not after the cluster finishes building.

Client configuration

talosconfig and kubeconfig are derived client configs, not backup-class artifacts. converge rewrites talosconfig (from talossecrets.yaml and controlplane-01's endpoint) and the kubeconfig, so deleting them costs nothing but the next unscheduled reconcile. Doing it by hand:

rm -f mycluster/talosconfig mycluster/kubeconfig
taloscluster converge -C mycluster

converge requires talossecrets.yaml to regenerate the talosconfig context, so this only works when the Talos identity is present. It also recovers a kubeconfig that went missing (for example a lost management machine) by re-fetching it from the cluster through the restored identity, so the cluster is reconciled as existing instead of being misread as a fresh bootstrap. After a destroy, converge wipes both derived files along with talossecrets.yaml so clients do not point stale credentials at a cluster that no longer exists.

etcd snapshot

talossecrets.yaml and the derived configs let you talk to nodes; they do not back up the cluster's own state inside etcd. The control-plane data layer — Kubernetes objects, secrets, and the state of workloads the API server wrote — lives in etcd. Take a snapshot of a healthy control plane:

talosctl --talosconfig talosconfig -n mycluster-controlplane-01 etcd snapshot etcd-snapshot-$(date +%F).db

-n targets a real control-plane node; the talosconfig context carries the endpoint, so no -e is needed. Run it from controlplane-01, whose etcd is authoritative. Restore is a low-level operation on an etcd member rather than something converge does; see the Talos documentation for etcd maintenance and restore procedures.

An etcd snapshot is only consistent for the moment it was taken. Workloads that write between snapshots are not covered, so snapshot on a schedule, not once.

Application data

etcd snapshots protect cluster control-plane state. They are not a backup of your data: PersistentVolumes, uncommitted writes, and anything managed by external controllers live elsewhere. Back up application data with a real Kubernetes backup tool (for example Velero), taking the running cluster's storage, CSI snapshots, and restore procedures into account. Verify and practice a restore before you need it.

The Talos host firewall allows intra-cluster traffic from network.cidr by default, so in-cluster backup pods can reach their storage; out-of-cluster sources need the security allowlists to reach port 6443.

Recovery: lost management machine

A new management machine recovers a cluster in the same way you back it up, in reverse: restore the backed-up files to a fresh cluster directory, reconnect to the tailnet, then let converge rebuild the derived state.

  1. Install taloscluster and connect the machine to the cluster's tailnet (or a reachable route to the private network). taloscluster does not join the tailnet for you; see Machines and access.
  2. Restore cluster.yaml, secrets.yaml, and talossecrets.yaml from backup into the cluster directory. Keep the file modes: secrets.yaml and talossecrets.yaml should be 0600.
  3. Run taloscluster plan to confirm the configuration matches what exists, then taloscluster converge. Converge regenerates talosconfig and kubeconfig from the restored identity and reconciles the cluster, so the new machine can manage it again.

A missing kubeconfig is not treated as proof that the cluster was never bootstrapped. Because the identity and machines exist, converge first recovers the kubeconfig from the first control plane through the restored identity and only then decides whether the cluster is fresh. A recovered cluster is reconciled as an existing one -- scale-down, machine-config apply and the Kubernetes upgrade all run, and any node scaled up in the same run boots at the upgraded version rather than a newer one joining an older cluster. Only when the recovery produces no kubeconfig (a first run that never reached bootstrap) does converge treat the cluster as fresh and bootstrap it. Recovery reaches the first control plane the way every talosctl call does: by its MagicDNS name when Tailscale is enabled, or its real provider-reported address otherwise (no hostname to resolve when Tailscale is off).

taloscluster plan -C mycluster
taloscluster converge -C mycluster
taloscluster status -C mycluster

If talossecrets.yaml was lost, converge refuses to proceed once it sees the cluster's machines exist (see Talos identity); there is no way around restoring the identity or rebuilding the cluster. If only the provider credential is lost, rotate it in the cloud and put the new value in secrets.yaml before converging.

Recovery: interrupted bootstrap

A converge interrupted between creating machines and bootstrapping etcd is not a broken cluster — it is an unfinished first run, and converge self-heals. The create phases (image, network, machines) are idempotent re-runs, and because a never-bootstrapped cluster has no kubeconfig, converge recognizes there was no bootstrap and completes it on the next run rather than trying to recreate the machines or mint a new identity. See Day 1. That interrupted first run already generated talossecrets.yaml in its secrets phase, so keep the backup you took of that pre-bootstrap identity: the re-run bootstraps the cluster from the same bundle without re-minting it, so the pre-bootstrap backup matches the running cluster.

taloscluster converge -C mycluster   # re-run after an interruption

That clears up after an interruption before the bootstrap. An interruption during upgrade, drain, or scale-down is different: converge re-runs and picks the safe reconciliation from wherever it stopped, but for control-plane work it refuses to progress past a half-done operation (for example a failed graceful reset) rather than risk losing quorum. Investigate with taloscluster status, re-run converge once the phase completes, and see Troubleshooting.