feat: Add Prometheus metrics endpoint and Grafana dashboard #2282

Merged
Jade merged 12 commits from eleboucher/continuwuity:feat/prometheus-metrics into main 2026-10-12 09:05:23 +00:00
Contributor

Adds an opt-in, token-protected /_continuwuity/metrics endpoint covering HTTP, federation, the sender, RocksDB, caches and the tokio runtime, plus a ready-to-import Grafana dashboard.

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
<!-- In order to help reviewers know what your pull request does at a glance, you should ensure that 1. Your PR title is a short, single sentence describing what you changed 2. You have described in more detail what you have changed, why you have changed it, what the intended effect is, and why you think this will be beneficial to the project. If you have made any potentially strange/questionable design choices, but didn't feel they'd benefit from code comments, please don't mention them here - after opening your pull request, go to "files changed", and click on the "+" symbol in the line number gutter, and attach comments to the lines that you think would benefit from some clarification. --> Adds an opt-in, token-protected /_continuwuity/metrics endpoint covering HTTP, federation, the sender, RocksDB, caches and the tokio runtime, plus a ready-to-import Grafana dashboard. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> <!-- Fixes: #... --> <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [x] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [x] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages
Member

Happy to see folks working on metrics!

That said, could you please take a look at #811, particularly this comment? There are some metrics requested there, including by maintainers, that don't seem to be covered here (yet), and I still think it would be a good idea to support OpenTelemetry for metrics since we already do for traces.

Happy to see folks working on metrics! That said, could you please take a look at #811, particularly [this comment](https://forgejo.ellis.link/continuwuation/continuwuity/issues/811#issuecomment-30023)? There are some metrics requested there, including by maintainers, that don't seem to be covered here (yet), and I still think it would be a good idea to support OpenTelemetry for metrics since we already do for traces.
Author
Contributor

@julian45 wrote in #2282 (comment):

Happy to see folks working on metrics!

That said, could you please take a look at #811, particularly this comment? There are some metrics requested there, including by maintainers, that don't seem to be covered here (yet), and I still think it would be a good idea to support OpenTelemetry for metrics since we already do for traces.

Done we should be good

@julian45 wrote in https://forgejo.ellis.link/continuwuation/continuwuity/pulls/2282#issuecomment-36113: > Happy to see folks working on metrics! > > That said, could you please take a look at #811, particularly [this comment](https://forgejo.ellis.link/continuwuation/continuwuity/issues/811#issuecomment-30023)? There are some metrics requested there, including by maintainers, that don't seem to be covered here (yet), and I still think it would be a good idea to support OpenTelemetry for metrics since we already do for traces. Done we should be good
eleboucher force-pushed feat/prometheus-metrics from 563b13a176
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m22s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m34s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m58s
to 6a80d3b69b
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m44s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m20s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m51s
2026-09-24 14:10:43 +00:00
Compare
eleboucher force-pushed feat/prometheus-metrics from 6a80d3b69b
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m44s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m20s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m51s
to d59b503eec
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m19s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m57s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 2m49s
2026-09-24 15:48:00 +00:00
Compare
Member

Thanks! Better, but I think there are some further improvements you might want to look at:

One bit of existing work that involves a some of both is opentelemetry-instrumentation-tower; see e.g. their axum worked example. This crate alone may give you a bunch of the Tower/Axum-based stuff you're looking for right out of the gate.

Thanks! Better, but I think there are some further improvements you might want to look at: - Leveraging [OTel Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) as made available in the [opentelemetry-semantic-conventions](https://github.com/open-telemetry/opentelemetry-rust/tree/main/opentelemetry-semantic-conventions) crate - Leveraging existing tools to reduce the need for hand-built instrumentation One bit of existing work that involves a some of both is [opentelemetry-instrumentation-tower](https://github.com/open-telemetry/opentelemetry-rust-contrib/tree/main/opentelemetry-instrumentation-tower); see e.g. their [axum worked example](https://github.com/open-telemetry/opentelemetry-rust-contrib/blob/main/opentelemetry-instrumentation-tower/examples/axum-http-service/src/main.rs). This crate alone may give you a bunch of the Tower/Axum-based stuff you're looking for right out of the gate.
eleboucher force-pushed feat/prometheus-metrics from a71f8d1606
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m8s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m59s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 2m16s
to 42583b9473
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Checks / Prek / Check changed files (pull_request) Successful in 10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m18s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m6s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m38s
2026-09-25 15:48:34 +00:00
Compare
eleboucher force-pushed feat/prometheus-metrics from 42583b9473
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Checks / Prek / Check changed files (pull_request) Successful in 10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m18s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m6s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m38s
to 66b595e330
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m11s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m53s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m46s
2026-09-26 09:39:50 +00:00
Compare
eleboucher force-pushed feat/prometheus-metrics from 66b595e330
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m11s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m53s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m46s
to b81fbb1649
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Checks / Prek / Check changed files (pull_request) Successful in 5s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m28s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m48s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m3s
2026-09-28 06:24:08 +00:00
Compare
nex requested review from Jade 2026-09-28 21:13:23 +00:00
eleboucher force-pushed feat/prometheus-metrics from b81fbb1649
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 6s
Checks / Prek / Check changed files (pull_request) Successful in 5s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m28s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m48s
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 5m3s
to b19e4b384a
All checks were successful
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 12s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m17s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m54s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 20m48s
2026-10-01 15:12:34 +00:00
Compare
Jade force-pushed feat/prometheus-metrics from b19e4b384a
All checks were successful
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 12s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m17s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m54s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 20m48s
to 0a36795479
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 18s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m48s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m57s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
2026-10-04 23:01:24 +00:00
Compare
Jade force-pushed feat/prometheus-metrics from 0a36795479
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 18s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 7s
Checks / Prek / Pre-commit & Formatting (pull_request) Failing after 1m48s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m57s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
to 19a942d8ac
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m20s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m3s
2026-10-04 23:28:54 +00:00
Compare
Jade requested changes 2026-10-06 19:28:59 +00:00
Dismissed
Jade left a comment

Mostly OK, some small issues

Mostly OK, some small issues
@ -78,0 +96,4 @@
scheme.eq_ignore_ascii_case("bearer") && constant_time_eq(provided, token)
});
if !authorized {
Owner

401 Unauthorized is probably the correct code

`401 Unauthorized` is probably the correct code
@ -154,0 +161,4 @@
.metrics
.instruments
.federation_incoming_pdus
.add(1, &[KeyValue::new("outcome", "unparsable")]);
Owner

Might wanna also set the type

Might wanna also set the type
@ -0,0 +133,4 @@
fn size_buckets() -> Vec<f64> { (0..11).map(|i| 1024.0 * f64::from(4_u32.pow(i))).collect() }
fn register_process(meter: &Meter) {
let started = SystemTime::now();
Owner

Instant::now()?

`Instant::now()`?
@ -0,0 +245,4 @@
let busy_metrics = metrics.clone();
meter
.f64_observable_counter("continuwuity.tokio.worker.busy_time")
Owner

This should only be enabled when compiled with tokio_unstable

This should only be enabled when compiled with `tokio_unstable`
@ -0,0 +35,4 @@
#[cfg(target_os = "linux")]
pub(super) fn open_fds() -> Option<u64> {
let count = std::fs::read_dir("/proc/self/fd").ok()?.count();
Owner

This probably counts itself :3

This probably counts itself :3
@ -90,3 +89,3 @@
.build()
.expect("Failed to create OTLP gRPC exporter"),
| "http" => opentelemetry_otlp::SpanExporter::builder()
| _ => opentelemetry_otlp::SpanExporter::builder()
Owner

Why remove the warning?

Why remove the warning?
@ -0,0 +103,4 @@
"RocksDB block cache usage by column family",
"By",
|services, observer| {
let Ok(usage) = services.db.db.memory_usage_stats() else {
Owner

Might wanna only call this once across both continuwuity.database gauges

Might wanna only call this once across both `continuwuity.database` gauges
@ -0,0 +153,4 @@
self.server.runtime().spawn(async move {
let mut interval = tokio::time::interval(COUNT_INTERVAL);
loop {
interval.tick().await;
Owner

Set missed_tick_behavior to something other than Burst

Set [`missed_tick_behavior`](https://docs.rs/tokio/latest/tokio/time/struct.Interval.html#method.set_missed_tick_behavior) to something other than `Burst`
Jade marked this conversation as resolved
@ -0,0 +162,4 @@
}
let local = services.users.count().await;
let remote = services.users.stream_remote_users().count().await;
Owner

I don't think reading every single remote user in the database, unconditionally, is a great idea for performance. Same for other places you have used .count on a database stream.

I don't think reading every single remote user in the database, unconditionally, is a great idea for performance. Same for other places you have used .count on a database stream.
Jade marked this conversation as resolved
fix(metrics): Address review feedback
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 5s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m12s
Update flake hashes / update-flake-hashes (pull_request) Successful in 2m57s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 7m41s
612c963025
Jade requested changes 2026-10-06 22:14:32 +00:00
Dismissed
Lines 179-192
@ -213,1 +177,16 @@
media_files: AtomicU64,
struct MemoryUsageCache(Mutex<Option<(Instant, Arc<MemoryUsage>)>>);
impl MemoryUsageCache {
fn get(&self, services: &Services) -> Option<Arc<MemoryUsage>> {
let mut cached = self.0.lock().expect("locked");
if let Some((at, usage)) = &*cached {
if at.elapsed() < MEMORY_USAGE_TTL {
return Some(usage.clone());
}
}
let usage = Arc::new(services.db.db.memory_usage_stats().ok()?);
*cached = Some((Instant::now(), usage.clone()));
Some(usage)
}
}
Owner

This is... significantly worse. When responding to review feedback, either push back when it's a bad idea, or look for a better solution. In this case, you could have gone to definition to see that you can get the mem_table & row_cache metrics separately from the col_caches metrics

This is... significantly worse. When responding to review feedback, either push back when it's a bad idea, or look for a better solution. In this case, you could have gone to definition to see that you can get the mem_table & row_cache metrics separately from the col_caches metrics
fix(metrics): Read database memory gauges without a cache
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 2m25s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 7m39s
f5996eb0b8
Jade force-pushed feat/prometheus-metrics from f5996eb0b8
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m10s
Update flake hashes / update-flake-hashes (pull_request) Successful in 2m25s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 7m39s
to 87404f5420
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m13s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m48s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
2026-10-07 14:29:49 +00:00
Compare
Jade force-pushed feat/prometheus-metrics from 87404f5420
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 9s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m13s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m48s
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
to 53aa7f10ba
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 11s
Checks / Prek / Check changed files (pull_request) Successful in 8s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m25s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m2s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m32s
2026-10-07 14:32:06 +00:00
Compare
Owner

Have you tested this on your server?

Testing locally spews a bunch of errors:

Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588030Z  INFO opentelemetry: name="MeterProvider.GlobalSet" "Global meter provider is set. Meters can now be created using global::meter() or global::meter_with_scope()."
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588067Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.uptime", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588104Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.cpu.time", Callbacks for this ObservableCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588129Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.memory.usage", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588155Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.memory.virtual", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588197Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.unix.file_descriptor.count", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588241Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.jemalloc.memory", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588273Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.workers", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588307Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.alive_tasks", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588337Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.global_queue_depth", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588371Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.worker.busy_time", Callbacks for this ObservableCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]:   2026-10-07T16:31:07.588404Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.worker.parks", Callbacks for this ObservableCounter will not be invoked. Check View Configuration.
Oct 07 16:31:07 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1)
Oct 07 16:31:07 personal2 continuwuation[2285871]:
Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588447Z  INFO conduwuit::server: 26.9.1 (53aa7f10b) server_name=ellis.link database_path="/var/lib/continuwuation" log_levels=info,hickory_proto::xfer::dns_exchange=error,conduit_service::pusher=error
Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.074596Z  INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=115 sequence=4652292224 time=457.434797ms
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111385Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.sender.queue.depth", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111437Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.federation.incoming.transactions.active", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111460Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.users", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111484Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.media.files", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111511Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.federation.destinations", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111534Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.memory", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111559Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.block_cache", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111591Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.size", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:
Oct 07 16:31:08 personal2 continuwuation[2285871]:   2026-10-07T16:31:08.111619Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.cache.entries", Callbacks for this ObservableGauge will not be invoked. Check View Configuration.
Oct 07 16:31:08 personal2 continuwuation[2285871]:     at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1)
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit_router::run::start
Oct 07 16:31:08 personal2 continuwuation[2285871]:     in conduwuit::main
Oct 07 16:31:08 personal2 continuwuation[2285871]:

Plus debug info at a log level that's too high

INFO stop: conduwuit::runtime: Final runtime metrics runtime_metrics=RuntimeMetrics { workers_count: 8, live_tasks_count: 0, total_park_count: 169290, max_park_count: 25502, min_park_count: 17732, total_busy_duration: 19.787454712s, max_busy_duration: 3.352558762s, min_busy_duration: 1.57758088s, global_queue_depth: 0, elapsed: 80.468234909s, mean_poll_duration: 130.401µs, mean_poll_duration_worker_min: 58.69µs, mean_poll_duration_worker_max: 223.972µs, poll_time_histogram: PollTimeHistogram { buckets: [HistogramBucket { range_start: 0ns, range_end: 25µs, count: 61001 }, HistogramBucket { range_start: 25µs, range_end: 50µs, count: 26739 }, HistogramBucket { range_start: 50µs, range_end: 75µs, count: 27091 }, HistogramBucket { range_start: 75µs, range_end: 100µs, count: 14618 }, HistogramBucket { range_start: 100µs, range_end: 125µs, count: 7227 }, HistogramBucket { range_start: 125µs, range_end: 150µs, count: 4467 }, HistogramBucket { range_start: 150µs, range_end: 175µs, count: 3402 }, HistogramBucket { range_start: 175µs, range_end: 200µs, count: 2449 }, HistogramBucket { range_start: 200µs, range_end: 225µs, count: 1557 }, HistogramBucket { range_start: 225µs, range_end: 250µs, count: 1040 }, HistogramBucket { range_start: 250µs, range_end: 275µs, count: 731 }, HistogramBucket { range_start: 275µs, range_end: 300µs, count: 500 }, HistogramBucket { range_start: 300µs, range_end: 325µs, count: 363 }, HistogramBucket { range_start: 325µs, range_end: 350µs, count: 302 }, HistogramBucket { range_start: 350µs, range_end: 375µs, count: 218 }, HistogramBucket { range_start: 375µs, range_end: 400µs, count: 191 }, HistogramBucket { range_start: 400µs, range_end: 425µs, count: 158 }, HistogramBucket { range_start: 425µs, range_end: 450µs, count: 120 }, HistogramBucket { range_start: 450µs, range_end: 475µs, count: 105 }, HistogramBucket { range_start: 475µs, range_end: 18446744073.709551615s, count: 5304 }] }, total_noop_count: 118500, max_noop_count: 18205, min_noop_count: 12325, total_steal_count: 19726, max_steal_count: 2755, min_steal_count: 2217, total_steal_operations: 18283, max_steal_operations: 2543, min_steal_operations: 0, num_remote_schedules: 668, total_local_schedule_count: 194176, max_local_schedule_count: 27512, min_local_schedule_count: 20703, total_overflow_count: 0, max_overflow_count: 0, min_overflow_count: 0, total_polls_count: 157582, max_polls_count: 22122, min_polls_count: 16915, total_local_queue_depth: 0, max_local_queue_depth: 0, min_local_queue_depth: 0, blocking_queue_depth: 0, blocking_threads_count: 0, idle_blocking_threads_count: 0, budget_forced_yield_count: 3512, io_driver_ready_count: 126438 }
Have you tested this on your server? Testing locally spews a bunch of errors: ``` Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588030Z INFO opentelemetry: name="MeterProvider.GlobalSet" "Global meter provider is set. Meters can now be created using global::meter() or global::meter_with_scope()." Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588067Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.uptime", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588104Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.cpu.time", Callbacks for this ObservableCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588129Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.memory.usage", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588155Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.memory.virtual", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588197Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "process.unix.file_descriptor.count", Callbacks for this ObservableUpDownCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:203 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588241Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.jemalloc.memory", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588273Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.workers", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588307Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.alive_tasks", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588337Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.global_queue_depth", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588371Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.worker.busy_time", Callbacks for this ObservableCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588404Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.tokio.worker.parks", Callbacks for this ObservableCounter will not be invoked. Check View Configuration. Oct 07 16:31:07 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:144 on main ThreadId(1) Oct 07 16:31:07 personal2 continuwuation[2285871]: Oct 07 16:31:07 personal2 continuwuation[2285871]: 2026-10-07T16:31:07.588447Z INFO conduwuit::server: 26.9.1 (53aa7f10b) server_name=ellis.link database_path="/var/lib/continuwuation" log_levels=info,hickory_proto::xfer::dns_exchange=error,conduit_service::pusher=error Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.074596Z INFO main:start:open: conduwuit_database::engine::open: Opened database. columns=115 sequence=4652292224 time=457.434797ms Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111385Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.sender.queue.depth", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111437Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.federation.incoming.transactions.active", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111460Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.users", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111484Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.media.files", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111511Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.federation.destinations", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111534Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.memory", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111559Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.block_cache", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111591Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.database.size", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: Oct 07 16:31:08 personal2 continuwuation[2285871]: 2026-10-07T16:31:08.111619Z ERROR opentelemetry_sdk: name: "InstrumentCreationFailed", meter_name: "continuwuity", instrument_name: "continuwuity.cache.entries", Callbacks for this ObservableGauge will not be invoked. Check View Configuration. Oct 07 16:31:08 personal2 continuwuation[2285871]: at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opentelemetry_sdk-0.33.0/src/metrics/meter.rs:262 on main ThreadId(1) Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit_router::run::start Oct 07 16:31:08 personal2 continuwuation[2285871]: in conduwuit::main Oct 07 16:31:08 personal2 continuwuation[2285871]: ``` Plus debug info at a log level that's too high ``` INFO stop: conduwuit::runtime: Final runtime metrics runtime_metrics=RuntimeMetrics { workers_count: 8, live_tasks_count: 0, total_park_count: 169290, max_park_count: 25502, min_park_count: 17732, total_busy_duration: 19.787454712s, max_busy_duration: 3.352558762s, min_busy_duration: 1.57758088s, global_queue_depth: 0, elapsed: 80.468234909s, mean_poll_duration: 130.401µs, mean_poll_duration_worker_min: 58.69µs, mean_poll_duration_worker_max: 223.972µs, poll_time_histogram: PollTimeHistogram { buckets: [HistogramBucket { range_start: 0ns, range_end: 25µs, count: 61001 }, HistogramBucket { range_start: 25µs, range_end: 50µs, count: 26739 }, HistogramBucket { range_start: 50µs, range_end: 75µs, count: 27091 }, HistogramBucket { range_start: 75µs, range_end: 100µs, count: 14618 }, HistogramBucket { range_start: 100µs, range_end: 125µs, count: 7227 }, HistogramBucket { range_start: 125µs, range_end: 150µs, count: 4467 }, HistogramBucket { range_start: 150µs, range_end: 175µs, count: 3402 }, HistogramBucket { range_start: 175µs, range_end: 200µs, count: 2449 }, HistogramBucket { range_start: 200µs, range_end: 225µs, count: 1557 }, HistogramBucket { range_start: 225µs, range_end: 250µs, count: 1040 }, HistogramBucket { range_start: 250µs, range_end: 275µs, count: 731 }, HistogramBucket { range_start: 275µs, range_end: 300µs, count: 500 }, HistogramBucket { range_start: 300µs, range_end: 325µs, count: 363 }, HistogramBucket { range_start: 325µs, range_end: 350µs, count: 302 }, HistogramBucket { range_start: 350µs, range_end: 375µs, count: 218 }, HistogramBucket { range_start: 375µs, range_end: 400µs, count: 191 }, HistogramBucket { range_start: 400µs, range_end: 425µs, count: 158 }, HistogramBucket { range_start: 425µs, range_end: 450µs, count: 120 }, HistogramBucket { range_start: 450µs, range_end: 475µs, count: 105 }, HistogramBucket { range_start: 475µs, range_end: 18446744073.709551615s, count: 5304 }] }, total_noop_count: 118500, max_noop_count: 18205, min_noop_count: 12325, total_steal_count: 19726, max_steal_count: 2755, min_steal_count: 2217, total_steal_operations: 18283, max_steal_operations: 2543, min_steal_operations: 0, num_remote_schedules: 668, total_local_schedule_count: 194176, max_local_schedule_count: 27512, min_local_schedule_count: 20703, total_overflow_count: 0, max_overflow_count: 0, min_overflow_count: 0, total_polls_count: 157582, max_polls_count: 22122, min_polls_count: 16915, total_local_queue_depth: 0, max_local_queue_depth: 0, min_local_queue_depth: 0, blocking_queue_depth: 0, blocking_threads_count: 0, idle_blocking_threads_count: 0, budget_forced_yield_count: 3512, io_driver_ready_count: 126438 } ```
Owner

That seems to be an issue solely where metrics are disabled entirely btw

That seems to be an issue solely where metrics are disabled entirely btw
fix(runtime): Log final runtime metrics at debug
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m15s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m49s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m25s
fb2ad46334
refactor: Delegate otlp_protocol to standard OTLP env variables
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 7s
Checks / Prek / Check changed files (pull_request) Successful in 7s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m20s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m46s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 11m30s
e931c09573
Jade approved these changes 2026-10-07 19:32:15 +00:00
Jade left a comment

Working in my setup!

Working in my setup!
fix(metrics): Scope the dashboard instance variable to continuwuity
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 10s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m12s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m52s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m41s
3695f90fc6
@ -0,0 +33,4 @@
```toml
[global]
allow_otlp_metrics = true
otlp_protocol = "http" # or "grpc"
Owner

I'll update this to match the last commit when I open my IDE again

I'll update this to match the last commit when I open my IDE again
Owner
https://github.com/open-telemetry/opentelemetry-rust-contrib/issues/839
Jade force-pushed feat/prometheus-metrics from 3695f90fc6
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Checks / Prek / Check changed files (pull_request) Successful in 6s
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 10s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m12s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m52s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 8m41s
to 3975d31b91
Some checks failed
Checks / Changelog / Check changelog is added (pull_request_target) Successful in 8s
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Documentation / Build and Deploy Documentation (push) Has been cancelled
Checks / Prek / Pre-commit & Formatting (push) Has been cancelled
Checks / Prek / Check changed files (push) Has been cancelled
Checks / Prek / Clippy and Cargo Tests (push) Has been cancelled
Release Docker Image / Build linux-amd64 (release) (push) Has been cancelled
Release Docker Image / Build linux-arm64 (release) (push) Has been cancelled
Release Docker Image / Create Multi-arch Release Manifest (push) Has been cancelled
Release Docker Image / Build linux-amd64 (max-perf) (push) Has been cancelled
Release Docker Image / Build linux-arm64 (max-perf) (push) Has been cancelled
Release Docker Image / Create Max-Perf Manifest (push) Has been cancelled
Release Docker Image / Release Binaries (push) Has been cancelled
Release Docker Image / Mirror Images (push) Has been cancelled
Checks / Prek / Check changed files (pull_request) Successful in 6s
Update flake hashes / update-flake-hashes (pull_request) Successful in 1m34s
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 20m52s
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 9m30s
2026-10-12 09:04:37 +00:00
Compare
Jade merged commit 3975d31b91 into main 2026-10-12 09:05:23 +00:00
Jade deleted branch feat/prometheus-metrics 2026-10-12 09:05:23 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
continuwuation/continuwuity!2282
No description provided.