Metrics
Insights shows you the state of your fleet right now. When something goes wrong, you also want its history: when a resource stopped being ready, how often it flaps, and whether one control plane is struggling or all of them are. Usually that means a Grafana for every cluster. Metrics collects Crossplane's metrics from every control plane you connect to Hub and lets you query them together, with PromQL, in one place.
What you can do
- Find where errors come from. Rank control planes by their reconcile errors, then narrow to the kind of resource that's failing.
- See when a resource stopped being ready. Follow a resource's conditions over time, not only their current state.
- Spot flapping. Find resources whose conditions keep changing.
- Tell one control plane's problem from a fleet-wide one. Compare the same metric across every control plane you can see.
- Watch your providers' cloud API usage. See how many calls each provider makes to the cloud APIs it manages.
How Metrics works
A collector on each connected control plane scrapes Crossplane, its providers, and its functions, and keeps only an allowlisted set of metrics. The connector sends them to Hub over the same authenticated channel it uses for resources. Hub labels each series with the control plane it came from, checks the allowlist again, and stores the series in a Prometheus-compatible backend that your platform team runs.
You query Metrics through the Insights API, with PromQL. Hub runs your query across every control plane you have permission to read, and returns one result covering all of them.
What Hub collects
| Metric | What it tells you |
|---|---|
controller_runtime_reconcile_total | How many reconciles each controller runs, and with what result. |
controller_runtime_reconcile_errors_total | How many reconciles fail, by controller. |
controller_runtime_reconcile_time_seconds | How long reconciles take, as a histogram. |
upjet_resource_external_api_calls_total | The calls Upjet-based providers make to the cloud APIs they manage. |
function_run_function_seconds | How long Crossplane's calls to composition functions take, as a histogram. Crossplane v2.1 and later. |
kube_customresource_resource_condition | The status conditions of resources on the control plane, when your platform team turns on resource state metrics. |
modelplane_* | Metrics from Modelplane. |
Your platform team can allow more metrics. See Add metrics to the allowlist.
How series identify their control plane
Hub labels every series with two labels, which you use to filter and group queries:
| Label | Value |
|---|---|
control_plane_name | The control plane's name. For a control plane in a Space, the name is <space>.<control plane>. |
realm | The control plane's realm. |
Hub sets these labels itself, from the connector that sent the series, so a control plane can't report metrics as another one.
Metrics follow the control plane, not its name. When you delete a control plane, its series drop out of your results. When you create a new control plane with the same name, it starts with no history.
Requirements and limits
Metrics is an alpha feature. Its API may change in incompatible ways between releases. See the feature lifecycle. It's off by default. Turn it on with the Metrics feature flag.
- Your platform team also sets up the metrics pipeline and a backend. See Metrics pipeline.
- Every query covers a time range. Hub doesn't run instant queries.
- Metrics carries metrics only, not logs or traces. It doesn't collect CPU or memory usage.
- How long Hub keeps metrics depends on your platform team's backend.