What’s Changed

• Fix label selectors in backups, policies, telemetry, eso
• fix: missing wiring for controlplanes uxp metrics flag
• Revert “fix(apollo): specify schema creating indexes to get ready for pg17”

What’s Changed

Warning - Breaking changes

Please be aware of the following changes:

• ClientCertFromHeader authenticator at spaces-router has been removed and it can no longer authenticate requests from a client certificate it finds in the Ssl-Client-Cert HTTP request header. spaces-router now requires SSL-passthrough to be enabled for the ingress-nginx controller if:

  • The Spaces v1.10.0 installation is still using the Ingress API (this is still the default although we now support the Gateway API) and,
  • spaces-router is running in secure mode (the default) and,
  • Hub identities are enabled via the authentication.hubIdentities Spaces Helm chart parameter (the default) and,
  • You would like to be able to authenticate Spaces clients using the client certificates issued by the host cluster’s signer.

  You can enable the SSL-passthrough mode for the ingress-nginx controller by passing the --enable-ssl-passthrough=true command-line option to it. Please also see the official self-hosted Spaces deployment guides.

• If you are using the Gateway API with Spaces and your chosen Gateway API implementation is Envoy Gateway, please note that the short name ctp now belongs to the clienttrafficpolicies.gateway.envoyproxy.io custom resources. If you have any scripts that use this short name for controlplanes.spaces.upbound.io, you will need to update them to use the long name controlplane if you are using Envoy Gateway.

Features and Enhancements

• Observability:

  • Added an option to reference sensitive data in SharedTelemetryConfig configuration through a secret.

• Query API:

  • Pin in-chart PostgreSQL cluster to 16 and wire down image pull secrets to it too.

• Security:

  • Spaces chart now installs a network policy that allows ingress traffic to the spaces-router pod’s port 8443 only from the ingress-nginx controller pod or the connect agent pod. The namespace and the pod labels of the ingress-nginx controller can be specified using the ingress.namespaceLabels and the ingress.podLabels Helm chart parameters, respectively. The pod labels for the connect agent can be specified using the connect.agent.podLabels Helm chart parameter.
  • Spaces now supports the Kubernetes Gateway API in addition to the Ingress API. The ClientCertFromHeader authenticator has been removed and when using a secured spaces-router with client certificate authentication, TLS traffic needs to be terminated at spaces-router.

• Other:

  • Add schema validation for helm chart values.
  • ControlPlanes now expose the time at which they first became Available.
  • Bumped latest supported Crossplane minor version to v1.18.
  • ClusterType is removed from the Spaces helm chart values

Fixed Bugs

• Fixed storage class parameter visibility for control plane etcd and vector in values.yaml
• Avoid noisy restarts of apollo-syncer by retrying with backoff before erroring out.
• Avoid resetting apollo syncers’ passwords on every reconciliation, preventing frequent syncers restarts.
• Drop controlPlanes.uxp.repository from values, always use registry.
• Fix an issue preventing to pull xpkg.upbound.io/spaces-artifacts/external-secrets-chart when Shared Secrets was enabled.
• Fixed a security issue with how the internal tokens are validated by mxp-gateway.
• Fixed a timeout issue in spaces-router while doing authorization checks against the host cluster for ControlPlane requests when hub RBAC is enabled via the authorization.hubRBAC Helm parameter.
• Fixed duplicate probe definitions in spaces controller deployment.
• Move to domain-qualified finalizer for controlplane provisioner reconciler, while dropping old ones allowing ControlPlanes deletion after Spaces upgrade.
• Properly handle and validate imagePullSecrets passed to the helm chart.
• Revert in-cluster host port used from 9091 to 8443. This led Argo to not be able to reach controlplanes.
• SpaceBackups now will only skip just created controlplanes instead of the ones not ready.
• Spaces-router can now reload without a restart renewed spaces-ca certificate which it uses to validate the upstream server certificates.

Bugs fixed

• Fixed a timeout issue in spaces-router while doing authorization checks against the host cluster for ControlPlane requests when hub RBAC is enabled via the authorization.hubRBAC Helm parameter
• Fixed an issue causing frequent syncers restarts in Apollo due to password resets
• Reduced the unnecessary restarts of Apollo syncers in case of transient errors
• Pin in-chart PostgreSQL cluster to version 16 and pass image pull secrets, if specified.

Security

• Spaces chart now installs a network policy that allows ingress traffic to the spaces-router pod’s port 8443 only from the ingress-nginx controller pod or the connect agent pod. The namespace and the pod labels of the ingress-nginx controller can be specified using the ingress.namespaceLabels and the ingress.podLabels Helm chart parameters, respectively. The pod labels for the connect agent can be specified using the connect.agent.podLabels Helm chart parameter

Enhancements

• ControlPlanes now expose the time at which they first became Available at .status.firstAvailableAt
• SpaceBackups now will only skip just created controlplanes instead of the ones not ready
• UXP images and associated registry actions are now configured with registry authentication for Spaces deployments using private registries, where the public UXP image is behind auth in the private registry.

What’s Changed

• Revert in-cluster host port used from 9091 to 8443. This led Argo to not be able to reach controlplanes.

What’s Changed

• Fixed duplicate probe definitions in spaces controller deployment.

What’s Changed

• Fixed an issue preventing to pull xpkg.upbound.io/spaces-artifacts/external-secrets-chart when Shared Secrets was enabled.

What’s Changed

Features and Enhancements

• Added optional insecure mode for all Spaces endpoints to allow finer grain control around mTLS policies.
• Added class dimension to all control plane metrics.
• Added new alpha feature, Space Backup, to allow backing up and restoring a Space in case of DR.
• Added a configuration option for enabling Control Plane Crossplane dependency version upgrades.
• Bumped latest supported Crossplane minor version to v1.18.
• External-secrets operator Helm chart is bumped to version 0.10.4.

Fixed Bugs

• Added the controlPlanes.syncer.extraSyncLabels spaces Helm chart parameter so that any extra labels that you specify with a DeploymentRuntimeConfig for a Crossplane provider/function can be configured to properly sync in the host cluster for the ControlPlane. You may need this for certain workload identity-based authentication schemes for authenticating Crossplane provider/function workloads.
• Drop controlPlanes.uxp.repository from values, always use registry.
• Fixed Query API failing to define necessary custom functions due to inconsistent ordering.
• Fixed a race when restarting spaces-controller admission webhooks fail with a certificate error.
• Fixed control plane and namespace listing on the ingress if additional non-RBAC authorization is configured on the host Kubernetes.
• Fixed the Spaces API endpoint for namespace to make kubectl get namespaces work.
• Fixed an issue where SharedTelemetryConfig would endlessly reconcile.
• Fixed an issue with SharedTelemetryConfig datadog exporter failing with 413 error code.
• Move to domain-qualified finalizer for control plane provisioner reconciler, while dropping old ones allowing ControlPlanes deletion after Spaces upgrade.
• Respect disabled features for discovery, avoiding unnecessary noise when using kubectl.

What’s Changed

Warning

Please be aware of the following changes:

• Spaces is no longer published to Google Artefact Registry and can only be accessed via xpkg.upbound.io.
• We’ve removed the following unused fields from the ControlPlane CRD:
  • spec.managementPolicies
  • spec.deletionPolicy
  • spec.publishConnectionDetailsTo
• User name, groups, uid and extra keys of user.Info originating from the host cluster (i.e. any host cluster identity) are all now prefixed with upbound:spaces:host: when that identity is used within a control plane. In Spaces v1.7 and below, no such prefix was added to groups from host cluster client certificates and tokens, so any RBAC rules within a control plane that refers to a user group from a host cluster identity need to be updated to add that prefix.

Features and Enhancements

• MCPs:

  • Crossplane v1.17 is added to the list of supported MCP versions.
  • Dropped the HEALTHY and added more details to MESSAGE column in the control plane get/list output to better communicate the status of the control planes.

• Query API:

  • Added ability to deploy and wire a CloudNativePG-powered Postgres Cluster for Query API directly from the helm chart.

• IAM:

  • The controlPlanes.mxpController.pod.customLabels Helm parameter was added to help configure workload identities for shared secrets on EKS, AKS & GKE clusters.
  • The controlPlanes.sharedSecrets.serviceAccount.customAnnotations and controlPlanes.sharedSecrets.pod.customLabels Helm parameters was added to help configure workload identities for shared secrets on EKS, AKS & GKE clusters.
  • Support was added for workload identity-based authentication schemes for Spaces billing on EKS, AKS & GKE clusters.
  • Added support for authenticating Host Kubernetes ServiceAccounts in Spaces API and Control Planes
  • Filter lists of namespaces and controlplanes in Spaces API to those the user has access to.
  • Mxp-gateway now uses the original user of the request being forwarded to a ControlPlane or xgql.
  • There is no more any need to populate the upbound.io/aud userinfo extra in structured auth config for OIDC.

• Backup & restore:

  • Deletion policy Delete for Backups using Secrets for credentials will now be respected.

• Shared Secrets:

  • External-secrets operator Helm chart is bumped to version 0.9.20.

• Administration:

  • Health checks were added to the spaces-controller pods.
  • Added control plane state metrics to track the number of all / synced / ready / healthy / deleting / degraded and stuck in provisioning or deleting control planes.

Fixed Bugs

• Added missing priorityClass for telemetry-spaces-logs daemonset.
• Fixed an issue where SharedTelemetryConfig would endlessly reconcile.
• Fixed an issue with SharedTelemetryConfig Datadog exporter failing with 413. The issue is not fully fixed but has been remediated by removing the metrics that were too big for Datadog to handle.

What’s Changed

• Fixed a bug in Spaces Chart’s pre-upgrade hook where the backOffLimit was incorrectly declared.

What’s Changed

• Fixed a bug in Apollo where the right column for owners query was not displayed correctly.
• Fixed a bug in Apollo to ensure that requests do not fail when debug output encounters an issue.
• Fixed a bug in MXP-Gateway to prevent logging of PII, and granted admins & editors the privilege to view logs for controlplane pods.
• Fixed a bug in Spaces Chart related to the pre-upgrade hook.
• Fixed a bug in Crossplane-Versions-Public configmap to allow public querying.

API Changes

• Added v1alpha2 of the Query API, which supports a richer set of filters.

What’s New

• OCI Artifact Support in Upbound Registry: We are excited to announce that the Spaces Helm Chart and images are now shipped as OCI artifacts by default, hosted in the Upbound central registry. You can access these at xpkg.upbound.io/spaces/artifacts.

  Important: To pull the Helm Chart and images from the new OCI location, you will need to obtain a new pull token from your Upbound account representative.

  To update your pull secret, follow these steps:

  1. Delete the existing pull secret:

     kubectl delete -n upbound-system upbound-pull-secret
     

  2. Re-create the upbound-pull-secret:

     kubectl -n upbound-system create secret docker-registry upbound-pull-secret \
     --docker-server=https://xpkg.upbound.io \
     --docker-username="$(jq -r .accessId $SPACES_TOKEN_PATH)" \
     --docker-password="$(jq -r .token $SPACES_TOKEN_PATH)"
     

  Start the Helm-Chart upgrade:

  1. Log in to the OCI Registry:

     jq -r .token $SPACES_TOKEN_PATH | helm registry login xpkg.upbound.io -u $(jq -r .accessId $SPACES_TOKEN_PATH) --password-stdin
     

  2. Upgrade Spaces software from the new location:

     helm -n upbound-system upgrade --install spaces \
       oci://xpkg.upbound.io/spaces-artifacts/spaces \
       --version "${SPACES_VERSION}" \
       --set "ingress.host=${SPACES_ROUTER_HOST}" \
       --set "clusterType=${SPACES_CLUSTER_TYPE}" \
       --set "account=${UPBOUND_ACCOUNT}" \
       --set "authentication.hubIdentities=true" \
       --set "authorization.hubRBAC=true"
     

• Helm Repository Deprecation: This release marks the final time the Spaces Helm Chart will be published to the Upbound Helm repository. All users are encouraged to migrate to the new OCI artifact format to ensure uninterrupted access to future updates.

  If you need additional time to prepare for this transition, you can still use the old registry with this release. To do so, you must explicitly set the registry:

  helm -n upbound-system upgrade --install spaces \
    oci://us-west1-docker.pkg.dev/orchestration-build/upbound-environments/spaces \
    --version "${SPACES_VERSION}" \
    --set "registry=us-west1-docker.pkg.dev/orchestration-build/upbound-environments" \
    --set "ingress.host=${SPACES_ROUTER_HOST}" \
    --set "clusterType=${SPACES_CLUSTER_TYPE}" \
    --set "account=${UPBOUND_ACCOUNT}" \
    --set "authentication.hubIdentities=true" \
    --set "authorization.hubRBAC=true"
  

  Note: This will be the last version that supports the old registry. We will discontinue publishing updates to it after Spaces 1.8.0.

We appreciate your cooperation and understanding during this transition. Should you have any questions or require further assistance, please reach out to your Upbound account representative.

• Simplified Installation Requirements: This release simplifies the installation process for the Spaces Helm Chart. You no longer need to have Crossplane installed with the provider-helm and provider-kubernetes on your HostCluster. If you were only using this Crossplane setup for Spaces, you can safely remove the remaining artifacts by running the following commands:

  kubectl delete xhostclusters.internal.spaces.upbound.io space-hub
  kubectl patch xhostclusters.internal.spaces.upbound.io space-hub --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'
  

  Once these steps are completed, you may proceed to uninstall Crossplane, provider-kubernetes, and provider-helm according to your original installation method.

  Note: The upbound-system namespace must not be removed, as it is still required for Spaces operations.

Other Improvements

• ControlPlane viewers can now list events.events.k8s.io resources and can get secrets.
• ControlPlane editors can no longer write ESO and kyverno resources.
• ControlPlane admins can now write kyverno cleanuppolicies, clustercleanuppolicies, policyexceptions and events.events.k8s.io.
• ControlPlane Admins can now also update CRDs.

Fixed Bugs

• Add priorityClass for telemetry-spaces-logs daemonset
• Cleanup control plane resources out of the system namespace when a control plane is deleted.
• Fix Backup’s expired TTL resulting in deadlock.
• Fixed a bug preventing scraping control plane etcd metrics
• Fixed duplicate port warning printed during installation of the Spaces helm chart.
• Observability: fixed an issue where network policies didnt allow the OTEL Collector’s Prometheus to scrape some pods for metrics.
• We have optimized our controllers and tested hosting up to 500 control planes with a single Spaces installation.

What’s Changed

• We fixed a bug with SharedTelemetryConfig that caused panics in the Spaces controller.
• We fixed Backup’s expired TTL resulting in deadlock.
• We fixed a bug preventing scraping of metrics from the control plane etcd pods.
• We’ve added a configuration option to enable Crossplane SSA Claims alpha feature in managed control planes.

API Changes

• The alpha spec.source ControlPlane field has been removed. It’s no longer supported.

Highlights

• It is now possible to pause and resume control planes through the spec.crossplane.state field.
• We optimized control plane provisioning, reducing time to readiness significantly and supporting up to 500 control planes with a single Spaces installation
• We’ve added alpha support for Query API, allowing performant querying of resources across multiple control planes.
• We’ve added a new feature that allows you to configure the OpenTelemetry Collector to collect logs from control planes.

What’s Changed

• We upgraded Kubernetes used internally in spaces components to v1.30. v1beta1 of Structured Authentication Configuration can now be used.
• Starting from Spaces v1.6, users must upgrade sequentially all the minor versions. A pre-upgrade job is added to enforce this requirement.
• Various bug fixes and performance improvements.

Highlights

• We’ve expanded the observability feature by adding Spaces-level logs collection configurable through helm values. We also added the healthcheck extension and liveness probe to the OpenTelemetry Collector.
• We have a new feature enabled by default that updates the ConfigMap crossplane-versions-public in the upbound-system namespace. Whenever a new security or fix release is published, the ConfigMap will be updated. You can disable this feature with controller.crossplane.versionsController.enabled.false when running in disconnected self-hosted Spaces.

What’s Changed

• We now expose a metrics port on vcluster-etcd containers.
• We removed network policies that block egress from a control plane’s functions.
• We removed the legacy OIDC flags authenticator deprecated in Spaces v1.3.0.

What’s Changed

• We updated the configuration of memory limits on a Space core component to avoid OOMs.
• We updated Kubernetes API, Controller and Manager to v1.28.6.

What’s Changed

• We added missing RBAC permissions for up migration import to work against a Spaces MCP.
• We improved the reliability of group creation in a connected self-hosted space.
• We fixed a bug impacting hub authorization when a Space is deployed on a Google Kubernetes Engine (GKE) cluster.

Highlights

• We’ve introduced a new alpha feature of Upbound IAM: Upbound RBAC. Upbound RBAC allows for a unified authentication and authorization model across Upbound. Users who operate single-tenant Cloud or Disconnected Spaces can continue to use the Kubernetes-native RBAC. Upbound RBAC allows users to control access in the Upbound Console down to the local Space. The new ObjectRoleBinding API type represents these Upbound RBAC role bindings in the Space locally.
• We’ve extended the alpha observability feature which shipped in Spaces v1.3.0. Observability is now also available at the Space level, which lets users observe Spaces machinery. To enable this feature, set the features.alpha.observability.enabled flag to true when installing Spaces.

What’s Changed

• We enabled the Crossplane Usages alpha feature in managed control planes.
• Space admins can now pass custom service account annotations to Crossplane service account.
• We fixed some bugs related to authentication and single-tenant Spaces when in Disconnected mode.
• We now allow scaling up core control plane components via helm values.
• The latest supported Crossplane minor version in Spaces was bumped to 1.16.
• Spaces prereq providers have had version bumps to allow for incorporating new metrics emissions from these providers. Provider-kubernetes is bumped to v0.14.0 and provider-helm to v0.19.0.
• Kube-native Hub authentication and authorization has been enabled by default.

Highlights

• This release fixes some Identity and Access Management related issues.

Highlights

• Control Plane Groups: Introducing Control Plane groups and Shared APIs for managing multiple control planes and related resource types, streamlining operations across environments.
• Automated Crossplane Upgrades: Implementing release channels for automated upgrades of Crossplane versions, ensuring Control planes remains up-to-date by getting latest patches automatically.
• Unified IAM: Unified identity and access management experience to manage access controls to everything within Spaces.
• Performance and Stability Improvements: Enhancements to system performance and stability to ensure a smoother and more reliable experience.

Alpha Features

• Aggregate Query API: Enchanced experience for querying one or more Control Planes with Aggregate Query API.
• External Secret Stores: Introducing the SharedSecretStore API, supporting external secret management.
• Upbound Policy: Introducing SharedUpboundPolicy API for centralized policy management across control planes.
• Observability: ShareTelemetryConfig API enabling exporting one or more telemetry for one or more control planes to the desired observability backends.
• Backup and Restore: Implementing SharedBackup and SharedBackupSchedule APIs to provide robust backup and restore functionality control planes.
• Importing/Exporting Control Planes: Enabling migrating in or out from Spaces control planes.

What’s Changed

• Tweaked the control plane API autoscaler configuration per recent performance testing.
• Fixed an issue causing the kube-state-metrics pods being restarted per CRD deployed in the control plane.
• Optimized the control plane deletion process to reduce the time it takes to delete a control plane.
• Fixed an issue breaking kubectl logs command against the control plane API.

What’s Changed

• This release contains several improvements to improve control plane orchestration performance. It addresses an issue where control plane provisioning time degraded when multiple control planes were provisioned in parallel.
• The latest supported minor Crossplane version is now v1.15.

What’s Changed

• This release fixes an issue that affected control planes’ ability to provision in non-kind cluster environments.

What’s Changed

• Fixed an issue causing the controlplane resources having a benign crossplane.io/external-create-failed annotation.
• Fixed an issue causing hotlooping version controller when a controlplane is deleted.
• Other stability and performance improvements.

What’s Changed

• We introduced a new concept called control plane groups within a Space. Technically, all kind: controlplane resources are now namespace-scoped objects (as opposed to previously being cluster-scoped).
• Control planes now offer auto-upgrade channels (rapid, stable, patch, and none), giving users control over what pace their control plane’s Crossplane version automatically upgrades to. None gives users total control over when to upgrade the Crossplane version in a managed control plane.
• Alpha suppport for a new aggregate query API that can be used to query state across one or more control planes in a group.
• Alpha support for built-in multi-control plane secrets management. Define new SharedSecrets and SharedSecretStores within a control plane group to selectively provision secrets from an external store–such as Vault–into the control planes in the group.
• Support for OIDC auhentication flows when interacting directly with a managed control plane in a Space.
• new up CLI commands to migrate open soure Crossplane or UXP instances into a managed control plane in a Space.

What’s Changed

• Alpha support for enabling External Secrets Operator in a control plane.
• Control plane api-server autoscaling based on CRD count.
• Universal Crossplane was bumped from v1.13.2-up.1 to v.1.13.2-up.2 for all control planes.
• new up CLI commands to interact with managed control planes in a Space.

What’s Changed

• Export mxp-gateway metrics via otlp-collector

What’s Changed

• controllers: patch against original object
• Promote APIs to v1beta1
• apis: clarify resource descriptions
• apis/mxp: minimize unused xr fields
• Stop routing internal traffic in the hub hostcluster through the ingress-controller
• XManagedControlPlane and hub XHostCluster XRs to v1beta1
• Introduce CRD lifecycle management through mxe-apis
• Add external-name to xhostclusterservices composed resource
• Stop logging bearer token at debug
• Clean up misc items
• Update XHostClusterServices resource
• Minor adjustments to destroy process
• Enable git source by default and keep it optional
• kube-control-plane: bump to kube 1.28
• ArgoCD controller to register ControlPlane as target
• vcluster: bump memory limit to 400Mi after seeing 270Mi in reality
• mxp-gateway: use client-go’s transport cache
• Bring vcluster-k8s in tree
• Return error instead of panic if ctp connection secret ref is unset
• vcluster: disable its liveness probe pointing to kube-apiserver
• Observability networkpolicy fixes
• Fix ssh auth with known_hosts and sub-directory discovery
• git: fix commit ref bugs
• git: run through cleanup even if controlplane is not ready
• Fix otlp-collector networkpolicy ports