continuwuation/continuwuity

Fork 64

feature request: Configurable URL preview scraper user-agent #1364

New issue

Closed

opened 2026-02-12 19:48:44 +00:00 by gan · 4 comments

gan commented

2026-02-12 19:48:44 +00:00

(Conversation reference), a person experienced an issue with generating a preview/embed off xwitter embed generator service (i.e., FxEmbed) because Continuwuity's request isn't treated as a bot request thus causing the preview generation to fail because of a redirect, this is not the case for Synapse, examples:

curl -iA "continuwuity/0.5.4 (1914616)" "https://fxtwitter.com/scarytoilet/status/1140686354403213314" (returns 302 Found);
curl -iA "Synapse (bot; +https://github.com/matrix-org/synapse)" "https://fxtwitter.com/scarytoilet/status/1140686354403213314" (returns the content necessary for an embed).

it's properly returned for synapse because it contains "bot" in its user-agent (the regex used by FxEmbed).

I believe lack of the "bot" declaration within the user-agent may cause some other preview generations off other services to fail as well, so, there might be a desire to allow control of a user agent for preview generation or globally.

[(Conversation reference)](matrix:roomid/da26JtAjE6APGLnX8ncWsvc-skF2KQZ9Nw_MbNpYD2k/e/Os6xEWjNqpu4JjOrVRjD2gZRRJZui5kiw8b7VAipTN8?via=explodie.org&via=explodie.org&via=continuwuity.org), a person experienced an issue with generating a preview/embed off xwitter embed generator service (i.e., FxEmbed) because Continuwuity's request isn't treated as a bot request thus causing the preview generation to fail because of a redirect, this is not the case for Synapse, examples: * `curl -iA "continuwuity/0.5.4 (1914616)" "https://fxtwitter.com/scarytoilet/status/1140686354403213314"` (returns `302 Found`); * `curl -iA "Synapse (bot; +https://github.com/matrix-org/synapse)" "https://fxtwitter.com/scarytoilet/status/1140686354403213314"` (returns the content necessary for an embed). it's properly returned for synapse because it contains "bot" in its user-agent (the regex used by [FxEmbed](https://github.com/FxEmbed/FxEmbed/blob/main/src/constants.ts#L31)). I believe lack of the "bot" declaration within the user-agent may cause some other preview generations off other services to fail as well, so, there might be a desire to allow control of a user agent for preview generation or globally.

nex added the

Enhancement

Matrix/Administration

Matrix/Client

Matrix/Media

labels

2026-02-12 19:59:18 +00:00

nex commented

2026-02-12 19:59:29 +00:00

Owner

Having a configurable user agent in the config file probably wouldn't be too hard

nex added the

Good first issue

label

2026-02-12 19:59:33 +00:00

nex changed title from ~~An ablity to set user-agent globally or for the preview generator~~ to feature request: Configurable URL preview scraper user-agent

2026-02-12 22:06:10 +00:00

gamesguru commented

2026-02-14 03:02:13 +00:00

Contributor

matrix-media-repo is also in the "approved" FxEmbed regexes

would having a configuration value with that as the DEFAULT be potentially harmful in anyway? Seems relatively benign, but I'm also new here, so the others can speak more to the project's inner workings.

It would definitely be good to see previews better supported!

# This also works
curl -iA "continuwuity/0.5.4 (1914616) [UA=matrix-media-repo]" "https://fxtwitter.com/scarytoilet/status/1140686354403213314"

`matrix-media-repo` is also in the "approved" FxEmbed regexes would having a configuration value with that as the DEFAULT be potentially harmful in anyway? Seems relatively benign, but I'm also new here, so the others can speak more to the project's inner workings. It would definitely be good to see previews better supported! ```shell # This also works curl -iA "continuwuity/0.5.4 (1914616) [UA=matrix-media-repo]" "https://fxtwitter.com/scarytoilet/status/1140686354403213314" ```

trashpanda referenced this issue from a pull request that will close it,

2026-02-14 03:18:34 +00:00

feat: Customize url preview user agent #1372

Jade commented

2026-02-14 11:29:06 +00:00

Owner

bot and embed are in there and would be better

[bot and embed are in there](https://github.com/FxEmbed/FxEmbed/blob/03b3481054c6ef662eb8dda986039f2d37dd525c/src/constants.ts#L32) and would be better

Jade commented

2026-02-14 11:31:20 +00:00

Owner

for URL previews something like continuwuity/0.5.4+1914616 (embedbot; +https://continuwuity.org) would probably be best

for URL previews something like `continuwuity/0.5.4+1914616 (embedbot; +https://continuwuity.org)` would probably be best

trashpanda referenced this issue

2026-02-14 19:50:04 +00:00

feat: Customize url preview user agent #1372