tower_http 500 errors when using x_envoy_external_address #1986

New issue

Closed

opened 2026-07-14 16:28:41 +00:00 by kiaraly · 0 comments

kiaraly commented

2026-07-14 16:28:41 +00:00

I'm getting the following errors in my logs on version 26.6.2 when setting request_ip_source = "x_envoy_external_address". I've changed it to request_ip_source = "x_forwarded_for" and the errors went away.

I'm using envoy-gateway as my reverse proxy and have configured a corresponding ClientTrafficPolicy with

  headers:
    enableEnvoyHeaders: true

so the headers should be present.

Continuwuity logs:

2026-07-14T16:19:52.853155Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)

  2026-07-14T16:19:52.999422Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m
atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)

  2026-07-14T16:19:52.999498Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)

  2026-07-14T16:19:59.695350Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m
atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)

  2026-07-14T16:19:59.695416Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:06.771775Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m
atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:06.771848Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:09.908789Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m
atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:09.908875Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:13.254832Z ERROR conduwuit_router::request: 500 Internal Server Error, method: PUT, uri: /_matrix/federation/v1/send/1783479526451
    at src/router/request.rs:105 on conduwuit:worker ThreadId(9)

  2026-07-14T16:20:13.254911Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(9)

  2026-07-14T16:20:14.336343Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m
atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126
    at src/router/request.rs:105 on conduwuit:worker ThreadId(4)

  2026-07-14T16:20:14.336414Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms
    at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4)
    ```

I'm getting the following errors in my logs on version 26.6.2 when setting `request_ip_source = "x_envoy_external_address"`. I've changed it to `request_ip_source = "x_forwarded_for"` and the errors went away. I'm using envoy-gateway as my reverse proxy and have configured a corresponding ClientTrafficPolicy with ```yaml headers: enableEnvoyHeaders: true ``` so the headers should be present. Continuwuity logs: ``` 2026-07-14T16:19:52.853155Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) 2026-07-14T16:19:52.999422Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) 2026-07-14T16:19:52.999498Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) 2026-07-14T16:19:59.695350Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) 2026-07-14T16:19:59.695416Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:06.771775Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:06.771848Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:09.908789Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:09.908875Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:13.254832Z ERROR conduwuit_router::request: 500 Internal Server Error, method: PUT, uri: /_matrix/federation/v1/send/1783479526451 at src/router/request.rs:105 on conduwuit:worker ThreadId(9) 2026-07-14T16:20:13.254911Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(9) 2026-07-14T16:20:14.336343Z ERROR conduwuit_router::request: 500 Internal Server Error, method: GET, uri: /_matrix/client/v3/sync?filter=q9Kw&timeout=0&org.m atrix.msc4222.use_state_after=true&set_presence=unavailable&since=1974126 at src/router/request.rs:105 on conduwuit:worker ThreadId(4) 2026-07-14T16:20:14.336414Z ERROR tower_http::trace::on_failure: response failed, classification: Status code: 500 Internal Server Error, latency: 0 ms at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-http-0.7.0/src/trace/on_failure.rs:93 on conduwuit:worker ThreadId(4) ```