forked from continuwuation/continuwuity
100 lines
6.2 KiB
Text
100 lines
6.2 KiB
Text
# Metrics
|
|
|
|
Continuwuity records metrics with OpenTelemetry and can export them in two ways, both disabled by default:
|
|
|
|
- **Prometheus scraping** from `/_continuwuity/metrics`
|
|
- **OTLP push**, alongside traces
|
|
|
|
## Prometheus
|
|
|
|
```toml
|
|
[global]
|
|
allow_prometheus_metrics = true
|
|
prometheus_metrics_token = "a-long-random-secret"
|
|
```
|
|
|
|
The endpoint is served on the same listener as the rest of the server. Either set `prometheus_metrics_token` or block `/_continuwuity/metrics` at your reverse proxy so it is not publicly readable.
|
|
|
|
```yaml
|
|
scrape_configs:
|
|
- job_name: continuwuity
|
|
scheme: https
|
|
metrics_path: /_continuwuity/metrics
|
|
authorization:
|
|
credentials: a-long-random-secret
|
|
static_configs:
|
|
- targets: ["matrix.example.com"]
|
|
```
|
|
|
|
## OTLP
|
|
|
|
Requires the `otlp_telemetry` feature, enabled in default builds.
|
|
|
|
```toml
|
|
[global]
|
|
allow_otlp_metrics = true
|
|
```
|
|
|
|
Configure the exporter with standard OpenTelemetry environment variables:
|
|
|
|
```sh
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
|
|
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf # or "grpc"
|
|
OTEL_METRIC_EXPORT_INTERVAL=15000
|
|
```
|
|
|
|
`service.instance.id` defaults to the server name, and `OTEL_RESOURCE_ATTRIBUTES` or `OTEL_SERVICE_NAME` override the defaults.
|
|
|
|
Prometheus can receive OTLP directly when started with `--web.enable-otlp-receiver`, using `http://prometheus:9090/api/v1/otlp` as the endpoint.
|
|
|
|
## Exported metrics
|
|
|
|
Names below are as they appear in Prometheus. OTLP uses the dotted OpenTelemetry names, such as `continuwuity.sender.events`. HTTP and process metrics follow the OpenTelemetry semantic conventions.
|
|
|
|
| Metric | Type | Labels | Description |
|
|
| --- | --- | --- | --- |
|
|
| `target_info` | gauge | `service_version`, `service_instance_id` | Running server version and name |
|
|
| `http_server_request_duration_seconds` | histogram | `http_request_method`, `http_route`, `http_response_status_code`, `network_protocol_name`, `network_protocol_version`, `url_scheme` | Handled HTTP requests, by route template |
|
|
| `http_server_active_requests` | gauge | `http_request_method`, `url_scheme` | Requests currently being handled |
|
|
| `http_server_request_body_size_bytes` | histogram | HTTP request duration labels | Request body sizes, when `Content-Length` is supplied |
|
|
| `http_server_response_body_size_bytes` | histogram | HTTP request duration labels | Response body sizes, when known |
|
|
| `continuwuity_federation_incoming_transaction_duration_seconds` | histogram | `outcome` | Processing time of incoming transactions |
|
|
| `continuwuity_federation_incoming_transactions_active` | gauge | | Incoming transactions currently being processed |
|
|
| `continuwuity_federation_incoming_pdus_total` | counter | `type`, `outcome` | Incoming PDUs by event type and outcome: `accepted`, `forbidden`, `invalid`, `error` or `unparsable` |
|
|
| `continuwuity_federation_incoming_pdu_duration_seconds` | histogram | `outcome` | Time to handle an incoming PDU |
|
|
| `continuwuity_federation_incoming_pdu_lag_seconds` | histogram | | Time between an accepted PDU's `origin_server_ts` and its arrival |
|
|
| `continuwuity_federation_incoming_edus_total` | counter | `type` | Incoming EDUs by type |
|
|
| `continuwuity_federation_incoming_edu_duration_seconds` | histogram | `type` | Time to handle an incoming EDU |
|
|
| `continuwuity_federation_outgoing_request_duration_seconds` | histogram | `outcome` | Outgoing federation requests |
|
|
| `continuwuity_federation_destinations` | gauge | `state` | Remote servers in retry `backoff` or with a `stale` destination cache entry |
|
|
| `continuwuity_sender_transaction_duration_seconds` | histogram | `destination`, `outcome` | Outgoing federation, appservice and push transactions |
|
|
| `continuwuity_sender_events_total` | counter | `destination`, `kind`, `outcome` | PDUs and EDUs sent in outgoing transactions |
|
|
| `continuwuity_sender_events_queued_total` | counter | `destination`, `kind` | PDUs and EDUs queued for sending |
|
|
| `continuwuity_sender_queue_depth` | gauge | | Messages waiting for the sender workers |
|
|
| `continuwuity_users` | gauge | `origin` | Known `local` users, including deactivated, and `remote` users |
|
|
| `continuwuity_invites_total` | counter | `direction`, `destination`, `outcome` | Invites sent by local users or received over federation |
|
|
| `continuwuity_media_write_size_bytes` | histogram | `origin` | Media files written, from local uploads or cached remote media |
|
|
| `continuwuity_media_read_size_bytes` | histogram | `origin` | Media files served |
|
|
| `continuwuity_media_files` | gauge | | Stored media files, including thumbnails |
|
|
| `continuwuity_antispam_checks_total` | counter | `check`, `outcome` | Antispam service checks; denials include the service being unreachable |
|
|
| `continuwuity_database_size_bytes` | gauge | | Total size of RocksDB SST files |
|
|
| `continuwuity_database_memory_bytes` | gauge | `name` | RocksDB memtable, table reader and row cache usage |
|
|
| `continuwuity_database_block_cache_bytes` | gauge | `column` | RocksDB block cache usage |
|
|
| `continuwuity_cache_entries` | gauge | `name` | Entries in the auth chain and state info caches |
|
|
| `continuwuity_jemalloc_memory_bytes` | gauge | `kind` | jemalloc statistics, only with the `jemalloc_stats` feature |
|
|
| `process_uptime_seconds` | gauge | | Time since the server started |
|
|
| `process_cpu_time_seconds_total` | counter | `cpu_mode` | CPU time used by the process |
|
|
| `process_memory_usage_bytes` | gauge | | Resident memory (Linux) |
|
|
| `process_memory_virtual_bytes` | gauge | | Virtual memory (Linux) |
|
|
| `process_unix_file_descriptor_count` | gauge | | Open file descriptors (Linux) |
|
|
| `continuwuity_tokio_workers` | gauge | | Tokio runtime worker threads |
|
|
| `continuwuity_tokio_alive_tasks` | gauge | | Alive tokio tasks |
|
|
| `continuwuity_tokio_global_queue_depth` | gauge | | Tasks in the tokio global queue |
|
|
| `continuwuity_tokio_worker_busy_time_seconds_total` | counter | | Time worker threads spent busy |
|
|
| `continuwuity_tokio_worker_parks_total` | counter | | Number of times worker threads parked |
|
|
|
|
User and media file counts are estimates taken from RocksDB (`rocksdb.estimate-num-keys`), so they may lag slightly or drift after deletions.
|
|
|
|
## Grafana dashboard
|
|
|
|
Import [grafana-dashboard.json](/guides/grafana-dashboard.json) under **Dashboards → New → Import**, or provision it from a file.
|