Skip to main content
Version: 1.19

Disaster Recovery

API Version Information

This guide applies to Spaces v1.9.0 and later (Self-Hosted only). Space Backups is a alpha feature enabled by default starting in v1.14.0.

  • v1.9.0-v1.13.x: Alpha feature (requires manual enablement)
  • v1.14.0+: GA (enabled by default)

For control-plane backups, see Backup and Restore.

important

For Connected and Disconnected Spaces, this feature requires Spaces v1.9.0 and, starting with v1.14.0, Spaces enables it by default.

To enable it on versions earlier than v1.14.0, set features.alpha.spaceBackup.enabled=true when you install Spaces.

up space init --token-file="${SPACES_TOKEN_PATH}" "v${SPACES_VERSION}" \
...
--set "features.alpha.spaceBackup.enabled=true"

Upbound's Space Backups is a built-in Space-wide backup and restore feature. This guide explains how to configure Space Backups and how to restore from one of them in case of disaster recovery.

This feature for Space administrators. Group or Control Plane users can leverage Shared Backups to backup and restore their ControlPlanes.

Benefits​

The Space Backups feature provides the following benefits:

  • Automatic backups for all resources in a Space and all resources in control planes, without any operational overhead.
  • Backup schedules.
  • Selectors to specify resources to backup.

Prerequisites​

Enabled the Space Backups feature in the Space:

  • Cloud Spaces: Not accessible to users.
  • Connected Spaces: Space administrator must enable this feature.
  • Disconnected Spaces: Space administrator must enable this feature.

Configure a Space Backup Config​

SpaceBackupConfig is a cluster-scoped resource. This resource configures the storage details and provider. Whenever a backup executes (either by schedule or manually initiated), it references a SpaceBackupConfig to tell it where store the snapshot.

Backup config provider​

The spec.objectStorage.provider and spec.objectStorage.config fields configures:

  • The object storage provider
  • The path to the provider
  • The credentials needed to communicate with the provider

You can only set one provider. Upbound currently supports AWS, Azure, and GCP as providers.

spec.objectStorage.config is a freeform map of configuration options for the object storage provider. See Thanos object storage for more information on the formats for each supported cloud provider. spec.bucket and spec.provider overrides the required values in the config.

AWS as a storage provider​

This example demonstrates how to use AWS as a storage provider for your backups:

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupConfig
metadata:
name: default
spec:
objectStorage:
provider: AWS
bucket: spaces-backup-bucket
config:
endpoint: s3.eu-west-2.amazonaws.com
region: eu-west-2
credentials:
source: Secret
secretRef:
name: bucket-creds
namespace: upbound-system
key: creds

This example assumes you've already created an S3 bucket called spaces-backup-bucket in the eu-west-2 AWS region. To access the bucket, define the account credentials as a Secret in the specified Namespace (upbound-system in this example).

Azure as a storage provider​

This example demonstrates how to use Azure as a storage provider for your backups:

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupConfig
metadata:
name: default
namespace: default
spec:
objectStorage:
provider: Azure
bucket: upbound-backups
config:
storage_account: upbackupstore
container: upbound-backups
endpoint: blob.core.windows.net
credentials:
source: Secret
secretRef:
name: bucket-creds
namespace: upbound-system
key: creds

This example assumes you've already created an Azure storage account called upbackupstore and blob upbound-backups. To access the blob, define the account credentials as a Secret in the specified Namespace (upbound-system in this example).

GCP as a storage provider​

This example demonstrates how to use Google Cloud Storage as a storage provider for your backups:

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupConfig
metadata:
name: default
namespace: default
spec:
objectStorage:
provider: GCP
bucket: spaces-backup-bucket
credentials:
source: Secret
secretRef:
name: bucket-creds
namespace: upbound-system
key: creds

This example assumes you've already created a Cloud bucket called "spaces-backup-bucket" and a service account with access to this bucket. Define the key file as a Secret in the specified Namespace (upbound-system in this example).

Use workload identity instead of a Secret​

Set spec.objectStorage.credentials.source to InjectedIdentity to authenticate with the identity of the spaces-controller pod instead of a Secret. See Space Backups Workload ID for the cloud-provider setup.

Configure a Space Backup Schedule​

SpaceBackupSchedule is a cluster-scoped resource. This resource defines a backup schedule for the whole Space.

Below is an example of a Space Backup Schedule running every day. It backs up all groups having environment: production labels and all control planes in those groups having backup: please labels.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupSchedule
metadata:
name: daily-schedule
spec:
schedule: "@daily"
configRef:
kind: SpaceBackupConfig
name: default
match:
groups:
labelSelectors:
- matchLabels:
environment: production
controlPlanes:
labelSelectors:
- matchLabels:
backup: please

Define a schedule​

The spec.schedule field is a Cron-formatted string. Some common examples are below:

EntryDescription
@hourlyRun once an hour.
@dailyRun once a day.
@weeklyRun once a week.
0 0/4 * * *Run every 4 hours.
0/15 * * * 1-5Run every fifteenth minute on Monday through Friday.
@every 1h30m10sRun every 1 hour, 30 minutes, and 10 seconds. Hour is the largest measurement of time for @every.

Suspend a schedule​

Use spec.suspend field to suspend the schedule. It creates no new backups, but allows running backups to complete.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupSchedule
metadata:
name: daily-schedule
spec:
suspend: true
...

Garbage collect backups when the schedule gets deleted​

Set the spec.useOwnerReferencesInBackup to garbage collect associated SpaceBackup when a SpaceBackupSchedule gets deleted. If set to true, backups are garbage collected when the schedule gets deleted.

Set the time to live​

Set the spec.ttl field to define the time to live for the backup. After this time, the backup is eligible for garbage collection. If this field isn't set, the backup isn't garbage collected.

The time to live is a duration, for example, 168h for 7 days.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackupSchedule
metadata:
name: daily-schedule
spec:
ttl: 168h # Backup is garbage collected after 7 days
...

Selecting space resources to backup​

By default, a SpaceBackup selects all groups and, for each of them, all control planes, secrets, and any other group-scoped resources.

By setting spec.match, you can include only specific groups, control planes, secrets, or other Space resources in the backup.

By setting spec.exclude, you can filter out some matched Space API resources from the backup.

Including space resources in a backup​

Different fields are available to include resources based on labels or names:

  • spec.match.groups to include only some groups in the backup.
  • spec.match.controlPlanes to include only some control planes in the backup.
  • spec.match.secrets to include only some secrets in the backup.
  • spec.match.extras to include only some extra resources in the backup.
apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackup
metadata:
name: my-backup
spec:
configRef:
kind: SpaceBackupConfig
name: default
match:
groups:
labelSelectors:
- matchLabels:
environment: production
controlPlanes:
labelSelectors:
- matchLabels:
backup: please
secrets:
names:
- my-secret
extras:
- apiGroup: "spaces.upbound.io"
kind: "SharedBackupConfig"
names:
- my-shared-backup

Excluding Space resources from the backup​

Use the spec.exclude field to exclude matched Space API resources from the backup.

Different fields are available to exclude resources based on labels or names:

  • spec.exclude.groups to exclude some groups from the backup.
  • spec.exclude.controlPlanes to exclude some control planes from the backup.
  • spec.exclude.secrets to exclude some secrets from the backup.
  • spec.exclude.extras to exclude some extra resources from the backup.
apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackup
metadata:
name: my-backup
spec:
ttl: 168h # Backup is garbage collected after 7 days
configRef:
kind: SpaceBackupConfig
name: default
match:
groups:
labelSelectors:
- matchLabels:
environment: production
exclude:
groups:
names:
- not-this-one-please

Exclude resources in control planes' backups​

By default, it backs up all resources in a selected control plane.

Use the spec.controlPlaneBackups.excludedResources field to exclude resources from control planes' backups.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackup
metadata:
name: my-backup
spec:
ttl: 168h # Backup is garbage collected after 7 days
configRef:
kind: SpaceBackupConfig
name: default
controlPlaneBackups:
excludedResources:
- secrets
- buckets.s3.aws.upbound.io

Create a manual backup​

SpaceBackup is a cluster-scoped resource that causes a single backup to occur for the whole Space.

Below is an example of a manual SpaceBackup:

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackup
metadata:
name: my-backup
spec:
configRef:
kind: SpaceBackupConfig
name: default
deletionPolicy: Delete

The backup specification DeletionPolicy defines backup deletion actions, including the deletion of the backup file from the bucket. The Deletion Policy value defaults to Orphan. Set it to Delete to remove uploaded files in the bucket. For more information on the backup and restore process, review the Spaces API documentation.

Set the time to live​

Set the spec.ttl field to define the time to live for the backup. After this time, the backup is eligible for garbage collection. If this field isn't set, the backup isn't garbage collected. The time to live is a duration, for example, 168h for 7 days.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceBackup
metadata:
name: my-backup
spec:
ttl: 168h # Backup is garbage collected after 7 days
...

Restore from a space backup​

Space Backup and Restore focuses only on disaster recovery. The restore procedure assumes a new Space installation with no existing resources. The restore procedure is idempotent, so you can run it multiple times without any side effects in case of failures.

To restore a Space from an existing Space Backup, follow these steps:

  1. Install Spaces from scratch as needed.
  2. Create a SpaceBackupConfig as needed to access the SpaceBackup from the object storage, for example named my-backup-config.
  3. Select the backup you want to restore from, for example my-backup.
  4. Create a SpaceRestore to restore the Space:
apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceRestore
metadata:
name: my-restore
spec:
source:
kind: SpaceBackup
name: my-backup
configRef:
kind: SpaceBackupConfig
name: my-backup-config

Before Spaces v1.19, run the restore command instead.

Restore with a SpaceRestore​

important

This feature is available from Spaces v1.19.

SpaceRestore is a cluster-scoped resource that restores a Space from a SpaceBackup. Like a SpaceBackup, you can create it with a GitOps tool, review it, and follow the restore progress in its status.

spec.source names the SpaceBackup to restore. The SpaceBackup doesn't have to exist in the Space: Spaces uses its name to find the backup in the object storage the spec.configRef SpaceBackupConfig points to. If you create the SpaceRestore and the SpaceBackupConfig at the same time, the restore waits for the SpaceBackupConfig to exist.

Spaces runs one restore at a time. A SpaceRestore created while another restore is in progress waits for it to finish, and pending restores start in creation order.

A SpaceRestore restores the Space once, and you can't change its spec after you create it. To restore again, create a new SpaceRestore. Spaces never retries a failed restore on its own. Restoring again imports the backed up state again into the control planes the earlier restore restored, overwriting any later change to them. Spaces keeps completed SpaceRestores, as a record of what it restored and when, until you delete them.

tip

Suspend your SpaceBackupSchedules while restoring, so that they don't back up a Space the restore hasn't finished restoring.

Restore specific control planes​

Instead of restoring the whole Space, you can choose to restore specific control planes from a backup with spec.controlPlanes. You can also set spec.controlPlanesOnly to skip restoring Space objects, such as groups and secrets, which must already exist. This lets Space administrators restore individual control planes without restoring the entire Space.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceRestore
metadata:
name: my-restore
spec:
source:
kind: SpaceBackup
name: my-backup
configRef:
kind: SpaceBackupConfig
name: my-backup-config
controlPlanes:
- group: default
name: ctp1
- group: default
name: ctp2
controlPlanesOnly: true

Spaces skips the Space objects that already exist, but restores the state of the control planes that already exist, overwriting any change made to them after the backup.

Tolerate control plane failures​

By default, any control plane failing to restore fails the whole restore. Set spec.failures.controlPlanes to tolerate some failures, as a number or as a percentage of the control planes selected for restore. When the failures exceed it, Spaces stops starting new control plane restores, waits for the ones in progress, and fails the restore.

A control plane counts as failed when it doesn't become ready and healthy, and get its state restored, within spec.controlPlaneTimeout, one hour by default. Raise it for large control planes.

spec.maxParallelControlPlanes sets how many control planes Spaces restores in parallel, five by default.

apiVersion: admin.spaces.upbound.io/v1alpha1
kind: SpaceRestore
metadata:
name: my-restore
spec:
source:
kind: SpaceBackup
name: my-backup
configRef:
kind: SpaceBackupConfig
name: my-backup-config
maxParallelControlPlanes: 5
controlPlaneTimeout: 2h
failures:
controlPlanes: 10%

Follow a restore​

List the SpaceRestores to see their phase and how many control planes they restored:

kubectl get spacerestores
NAME PHASE SOURCE CONFIG RESTORED FAILED AGE
my-restore InProgress my-backup my-backup-config 7 1 12m
PhaseDescription
PendingWaiting for another restore to finish, or for the SpaceBackupConfig to exist.
InProgressRestoring the Space objects, then the control planes.
CompletedThe restore completed, with no more control plane failures than tolerated.
FailedThe restore failed. The Succeeded condition tells why.

The status also records when the restore started and finished, and the Succeeded condition tells what the restore is waiting for or doing. status.controlPlanes counts the control planes selected for restore, and the ones restored, skipped, for example because the backup has no state for them, and failed. It also lists the control planes not restored and why, up to 100 of them.

Spaces runs each restore in a Job in the upbound-system namespace, named after the SpaceRestore followed by a hash, and keeps it until you delete the SpaceRestore. Its logs list every control plane not restored:

kubectl get jobs -n upbound-system
kubectl logs -n upbound-system job/my-restore-1a2b3c4d

Kubernetes can garbage collect the pod of a finished Job, and its logs with it.

Stop a restore​

Delete the SpaceRestore to stop its restore. Spaces deletes the restore Job, waits for its pod to stop, and then removes the SpaceRestore. What the restore restored before stopping stays in place.

warning

Don't disable Space Backups while a restore is in progress. Without them, Spaces can't report the restore outcome when the restore itself can't. It also can't stop the restore when you delete its SpaceRestore, which stays until you enable Space Backups again.

Configure the restore Job​

Each restore runs in a Job in the upbound-system namespace. To reach the object storage and the Space, the Job's pod gets the spaces-controller pod settings. These include its image, service account, image pull secrets, security contexts and resources. They also include its affinity, node selector, tolerations, pod labels and annotations, extra environment variables, volumes and volume mounts. The workload identity set up for Space Backups applies to restores too.

Override them, and configure the rest, through the controller.restore.job Helm values:

ValueDefaultDescription
controller.restore.job.resources{}Requests and limits of the Job's pod. When empty, the spaces-controller ones apply. The restore loads the whole backup in memory, so size them for your largest backup.
controller.restore.job.affinity{}Affinity of the Job's pod, for example to keep it away from the spaces-controller node. When empty, the spaces-controller one applies.
controller.restore.job.podLabels{}Labels added to the Job's pod, besides the spaces-controller pod labels.
controller.restore.job.startupTimeout10mHow long the Job's pod can take to start, for example to get scheduled and pull its image, before the restore fails.
controller.restore.job.podDisruptionBudget.enabledtruePrevents evicting the Job's running pod, for example when draining its node, which would fail the restore.

The Spaces Helm chart README lists the other controller.restore.job values, such as nodeSelector and tolerations, which replace the spaces-controller settings when set.

The PodDisruptionBudget blocks draining the node running a restore until the restore finishes, which some managed node upgrades only wait for up to a timeout. It doesn't prevent involuntary disruptions, such as a node failure, or Kubernetes evicting the Job's pod when its node runs out of memory, which fail the restore.

Admission policies and resource quotas applied to the upbound-system namespace must admit the restore Job and its pod.

Restore permissions​

Creating a SpaceRestore requires the create permission on spacerestores in the admin.spaces.upbound.io API group, and the get permission on the SpaceBackupConfig it references.

warning

The restore runs with the spaces-controller service account and creates whatever the backup has. Treat the permissions to create SpaceRestores, or to create and update SpaceBackupConfigs, as granting the same access as the spaces-controller service account.

Restore with the hyperspace restore command​

Before Spaces v1.19, or to restore a Space by hand, run the restore command in the spaces-controller pod:

export SPACE_BACKUP_CONFIG=my-backup-config
export SPACE_BACKUP=my-backup
kubectl exec -ti -n upbound-system deployments/spaces-controller -c spaces -- hyperspace restore $SPACE_BACKUP $SPACE_BACKUP_CONFIG

The command flags match the SpaceRestore fields:

SpaceRestore fieldCommand flagAvailable from
spec.controlPlanes--controlplanesv1.11
spec.controlPlanesOnly--controlplanes-onlyv1.11
spec.maxParallelControlPlanes--max-parallel-controlplanesv1.9
spec.controlPlaneTimeout--controlplane-timeoutv1.19
spec.failures.controlPlanes--tolerated-controlplane-failuresv1.19

For example, to restore specific control planes:

export SPACE_BACKUP_CONFIG=my-backup-config
export SPACE_BACKUP=my-backup
kubectl exec -ti -n upbound-system deployments/spaces-controller -c spaces \
-- hyperspace restore $SPACE_BACKUP $SPACE_BACKUP_CONFIG --controlplanes default/ctp1,default/ctp2 --controlplanes-only