Observability

Hub's components emit their own Prometheus metrics: request rates and latency, Go runtime health, and Hub-specific counters for ingestion, queries, and internal work. This page covers what Hub exposes, a turnkey chart to view it, and how to wire it into your own monitoring backend for production.

This page isn't the Metrics pipeline

This page is about the health of the Hub software itself. It's separate from the Metrics pipeline, which collects control-plane metrics from connected control planes and serves them through the Metrics Query API. The two are independent: you can run either, both, or neither, and they never share storage.

What the Hub exposes today

hub-core serves a Prometheus metrics endpoint on port 8085. It carries two kinds of metrics:

• Standard framework metrics its runtime emits automatically. The OpenTelemetry HTTP server metrics for every API route (the RED signals: request rate, errors, and latency) and the usual Go runtime metrics (go_*: goroutines, heap, and GC). Consult those conventions for the full field list.
• Hub instruments (prefixed hub_core_) for the ingest and query paths and the internal subsystems, cataloged below.

All Hub metric names below are the Prometheus-exposition form. Counters end in _total; histograms expose _bucket, _sum, and _count series.

Resource ingestion

Metric	Type	Key labels	Measures
hub_core_ingest_resource_total	counter	control_plane, realm, op	Committed resource ingests. The foundation volume metric.
hub_core_ingest_resource_by_type_total	counter	group, kind, crossplane_type, op	The same ingests broken down by resource type.
hub_core_ingest_rejected_total	counter	reason	Ingest requests rejected before commit.
hub_core_ingest_stale_write_dropped_total	counter	control_plane, realm	Upserts dropped because the incoming resource version wasn't newer than the stored one. Spikes indicate out-of-order delivery or a connector resync storm.
hub_core_ingest_event_lag_seconds	histogram	-	Lag between event generation at the source and receipt by Hub.
hub_core_reconcile_pruned_total	counter	control_plane, realm, group, resource	Resources soft-deleted because a connector's reconcile snapshot no longer contained them.

Metrics ingestion

The ingest side of the Metrics pipeline, measured as Hub's own operational counters.

Metric	Type	Key labels	Measures
hub_core_metrics_ingest_forward_total	counter	control_plane, http_response_status_code	OTLP metrics forwarded to the gateway.
hub_core_metrics_ingest_rejected_total	counter	reason	Ingest requests rejected before forwarding.
hub_core_metrics_ingest_body_bytes	histogram	-	Size of accepted OTLP request bodies.

Query APIs

Metric	Type	Key labels	Measures
hub_core_resource_query_view_total	counter	view	Resource list queries by projection view (full vs. summary).
hub_core_resource_query_result_count	histogram	view	Result count returned per list query.
hub_core_metrics_query_total	counter	outcome	Metric query requests by terminal outcome.
hub_core_metrics_query_duration_seconds	histogram	status	Prometheus backend range-query latency.
hub_core_metrics_query_result_series	histogram	-	Series returned by a range query.
hub_core_metrics_query_points_estimate	histogram	-	Estimated sample points a query materializes .(end − start) / step. An expensive-query detector.
hub_core_metrics_query_span_seconds	histogram	-	Requested query time-range width.
hub_core_metrics_query_scope_rejections_total	counter	reason	PromQL scope-injection rejections. Attempted tenant-boundary violations.
hub_core_metrics_query_scope_size	histogram	-	Authorized control-plane matchers injected into a scoped query.

Internals

Hub extracts relationship edges (owner references, Crossplane composite → claim → managed-resource links) at ingest. When an edge points at a resource Hub hasn't ingested yet, Hub holds the edge in a backlog, and a background sweeper promotes it once the target arrives.

Metric	Type	Measures
hub_core_pending_edges_count	gauge	Edges waiting for their target resource. The backlog. A steady rise means many references point at resources not yet ingested.
hub_core_edge_sweeper_promoted_total	counter	Pending edges resolved into real edges.
hub_core_edge_sweeper_cycle_capped_total	counter	Sweeps that hit the per-cycle batch cap with a backlog remaining. The sweeper can't clear the backlog at its current rate.

Ingest inbox

Hub queues incoming resource events in a PostgreSQL-backed inbox before committing them. The inbox metrics carry the hub_core_inbox_ prefix. The gauges refresh every 30 seconds on each replica.

Metric	Type	Measures
hub_core_inbox_messages_pending	gauge	Messages waiting to be processed. A steady rise means processing can't keep up with ingest.
hub_core_inbox_messages_processing	gauge	Messages currently being processed.
hub_core_inbox_messages_failed	gauge	Messages in the failed state.
hub_core_inbox_dlq_messages	gauge	Messages in the dead-letter queue.
hub_core_inbox_messages_stored_total	counter	Messages written to the inbox.
hub_core_inbox_messages_processed_total	counter	Messages processed successfully.
hub_core_inbox_messages_failed_total	counter	Message processing failures.
hub_core_inbox_messages_retried_total	counter	Message processing retries.
hub_core_inbox_dlq_entered_total	counter	Messages moved to the dead-letter queue after exhausting their retries.
hub_core_inbox_partitions_overdue	gauge	Partitions past the retention window that cleanup couldn't remove. A sustained non-zero value means cleanup is falling behind. Use the maximum across replicas.
hub_core_inbox_store_duration_seconds	histogram	Time to write messages to the inbox.
hub_core_inbox_process_duration_seconds	histogram	Time to process a message.

Start with the hub-observability chart

The quickest way to see these metrics is the hub-observability chart. It installs a single grafana/otel-lgtm pod:

• an OpenTelemetry Collector
• Prometheus
• Grafana

The pod scrapes hub-core:8085 into its own Prometheus. It comes with three preloaded dashboards: Hub Service Health, Hub Ingest, and Hub Query.

The chart discovers Hub pods in its own release namespace, so install it in the same namespace as the Hub:

helm install hub-observability oci://xpkg.upbound.io/upbound/hub-observability \
--version 1.2.0 \
--namespace hub
kubectl -n hub port-forward svc/hub-observability-hub-observability 3000:3000
# → http://localhost:3000

To run it in a separate namespace instead, point it at the namespace where Hub runs with scrape.namespace:

helm install hub-observability oci://xpkg.upbound.io/upbound/hub-observability \
--version 1.2.0 \
--namespace observability --create-namespace \
--set scrape.namespace=hub

The chart is demo and evaluation-grade. It runs as one all-in-one pod with a single PVC on a single node. Use it to explore the metrics and dashboards. Back production alerting with your own monitoring stack instead.

Export to your own backend

For production, scrape hub-core:8085 with your own agent and send the metrics to the observability backend of your choice.

The pipeline below is only an example. An OpenTelemetry Collector that scrapes the Hub pods and remote-writes to a Prometheus-compatible endpoint. Swap the exporter for one that matches your backend (OTLP, Google Cloud, Datadog, and so on):

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: hub-core
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [hub]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
              action: keep
              regex: hub-core
            - source_labels: [__meta_kubernetes_pod_ip]
              target_label: __address__
              # $$ escapes the Collector's env-var expansion so the relabel
              # engine receives a literal ${1} capture group.
              replacement: $${1}:8085
exporters:
  prometheusremotewrite:
    endpoint: https://<your-backend>/api/v1/write
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheusremotewrite]

Hub Service Health, Hub Ingest, and Hub Query are plain Grafana JSON built on PromQL. If your backend serves the Prometheus query API, import them into your own Grafana for the same views.

Roadmap

Instrumentation today covers the ingest and query paths. Planned additions, subject to change:

• Authentication and tokens: OIDC login, token exchange, and device-flow funnels, currently visible only in logs.
• Authorization: the per-request filter cost that dominates list latency for large organizations, and denial breakdowns.
• Database: connection-pool saturation and per-store query latency.
• Fail-closed alerts: cheap error counters for feature-flag evaluation and authorization role checks, so a silent dependency outage becomes an alert.
• Distributed tracing: connector ↔ API request-path traces through the same collector, with an opt-in trace backend in the chart.

Next step

If you haven't yet, work through Production hardening for sizing, high availability, autoscaling, RBAC, and upgrades. Once you've prepared your Hub install for production, come back and configure watching for your hardened install.