continuwuation/continuwuity
continuwuation/continuwuity
16
60
Fork 64
Code Issues 99 Pull requests 31 Releases 26 Packages 3 Activity Actions 10

feat: Customize url preview user agent #1372

Merged
Jade merged 4 commits from trashpanda/continuwuity:trashpanda/url-preview-ua into main 2026-02-15 15:23:35 +00:00
Conversation 8 Commits 4 Files changed 5 +26 -1
trashpanda commented 2026-02-14 03:18:34 +00:00
Contributor
Copy link

This pull request implements a config option to set the user agent of the URL preview reqwest client, which can help with embed generator services like FxEmbed which expect certain strings in user agents to determine that the requestor is not a user attempting to navigate to the requested website. If those certain strings aren't found, they usually issue a redirect to the requested page.

Fixes: #1364

Pull request checklist:

  • This pull request targets the main branch, and the branch is named something other than
    main.
  • I have written an appropriate pull request title and my description is clear.
  • I understand I am responsible for the contents of this pull request.
  • I have followed the contributing guidelines:
<!-- In order to help reviewers know what your pull request does at a glance, you should ensure that 1. Your PR title is a short, single sentence describing what you changed 2. You have described in more detail what you have changed, why you have changed it, what the intended effect is, and why you think this will be beneficial to the project. If you have made any potentially strange/questionable design choices, but didn't feel they'd benefit from code comments, please don't mention them here - after opening your pull request, go to "files changed", and click on the "+" symbol in the line number gutter, and attach comments to the lines that you think would benefit from some clarification. --> This pull request implements a config option to set the user agent of the URL preview reqwest client, which can help with embed generator services like FxEmbed which expect certain strings in user agents to determine that the requestor is not a user attempting to navigate to the requested website. If those certain strings aren't found, they usually issue a redirect to the requested page. <!-- Example: This pull request allows us to warp through time and space ten times faster than before by double-inverting the warp drive with hyperheated jump fluid, both making the drive faster and more efficient. This resolves the common issue where we have to wait more than 10 milliseconds to engage, use, and disengage the warp drive when travelling between galaxies. --> <!-- Closes: #... --> Fixes: #1364 <!-- Uncomment the above line(s) if your pull request fixes an issue or closes another pull request by superseding it. Replace `#...` with the issue/pr number, such as `#123`. --> **Pull request checklist:** <!-- You need to complete these before your PR can be considered. If you aren't sure about some, feel free to ask for clarification in #dev:continuwuity.org. --> - [x] This pull request targets the `main` branch, and the branch is named something other than `main`. - [x] I have written an appropriate pull request title and my description is clear. - [x] I understand I am responsible for the contents of this pull request. - I have followed the [contributing guidelines][c1]: - [x] My contribution follows the [code style][c2], if applicable. - [ ] I ran [pre-commit checks][c1pc] before opening/drafting this pull request. - [ ] I have [tested my contribution][c1t] (or proof-read it for documentation-only changes) myself, if applicable. This includes ensuring code compiles. - [x] My commit messages follow the [commit message format][c1cm] and are descriptive. - [ ] I have written a [news fragment][n1] for this PR, if applicable<!--(can be done after hitting open!)-->. <!-- Notes on these requirements: - While not required, we encourage you to sign your commits with GPG or SSH to attest the authenticity of your changes. - While we allow LLM-assisted contributions, we do not appreciate contributions that are low quality, which is typical of machine-generated contributions that have not had a lot of love and care from a human. Please do not open a PR if all you have done is asked ChatGPT to tidy up the codebase with a +-100,000 diff. - In the case of code style violations, reviewers may leave review comments/change requests indicating what the ideal change would look like. For example, a reviewer may suggest you lower a log level, or use `match` instead of `if/else` etc. - In the case of code style violations, pre-commit check failures, minor things like typos/spelling errors, and in some cases commit format violations, reviewers may modify your branch directly, typically by making changes and adding a commit. Particularly in the latter case, a reviewer may rebase your commits to squash "spammy" ones (like "fix", "fix", "actually fix"), and reword commit messages that don't satisfy the format. - Pull requests MUST pass the `Checks` CI workflows to be capable of being merged. This can only be bypassed in exceptional circumstances. If your CI flakes, let us know in matrix:r/dev:continuwuity.org. - Pull requests have to be based on the latest `main` commit before being merged. If the main branch changes while you're making your changes, you should make sure you rebase on main before opening a PR. Your branch will be rebased on main before it is merged if it has fallen behind. - We typically only do fast-forward merges, so your entire commit log will be included. Once in main, it's difficult to get out cleanly, so put on your best dress, smile for the cameras! --> [c1]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md [c2]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/docs/development/code_style.mdx [c1pc]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#pre-commit-checks [c1t]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#running-tests-locally [c1cm]: https://forgejo.ellis.link/continuwuation/continuwuity/src/branch/main/CONTRIBUTING.md#commit-messages [n1]: https://towncrier.readthedocs.io/en/stable/tutorial.html#creating-news-fragments
feat(service): apply url preview user agent from config
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 22s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m25s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 18m52s
Details
e8f5c49341
trashpanda reviewed 2026-02-14 03:26:05 +00:00
src/service/client/mod.rs Outdated
@ -52,0 +53,4 @@
config
.url_preview_user_agent
.as_ref()
.map_or(conduwuit::version::user_agent(), |ua| ua.as_str()),
trashpanda commented 2026-02-14 03:26:05 +00:00
Author
Contributor
Copy link

I considered making this into a block expression to only set the user agent when the config option is set, but it felt less concise. Duplicating the user agent set in the base() function feels weird, though.

url_preview: {
	let url_preview = base(config)
		.and_then(|builder| {
			builder_interface(builder, url_preview_bind_iface.as_deref())
		})?
		.local_address(url_preview_bind_addr)
		.dns_resolver(resolver.resolver.clone())
		.timeout(Duration::from_secs(config.url_preview_timeout))
		.redirect(redirect::Policy::limited(3))
		.user_agent(
			config
				.url_preview_user_agent
				.as_ref()
				.map_or(conduwuit::version::user_agent(), |ua| ua.as_str()),
		);
	match &config.url_preview_user_agent {
		| Some(ua) => url_preview.user_agent(ua),
		| None => url_preview,
	}
}
.build()?,
I considered making this into a block expression to only set the user agent when the config option is set, but it felt less concise. Duplicating the user agent set in the `base()` function feels weird, though. ```rust url_preview: { let url_preview = base(config) .and_then(|builder| { builder_interface(builder, url_preview_bind_iface.as_deref()) })? .local_address(url_preview_bind_addr) .dns_resolver(resolver.resolver.clone()) .timeout(Duration::from_secs(config.url_preview_timeout)) .redirect(redirect::Policy::limited(3)) .user_agent( config .url_preview_user_agent .as_ref() .map_or(conduwuit::version::user_agent(), |ua| ua.as_str()), ); match &config.url_preview_user_agent { | Some(ua) => url_preview.user_agent(ua), | None => url_preview, } } .build()?, ```
trashpanda marked this conversation as resolved
fix: Clippy lint
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 30s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 3m29s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 31m41s
Details
9d0b5e7819
ginger approved these changes 2026-02-14 19:42:04 +00:00
Dismissed
ginger left a comment
Owner
Copy link

Code looks good to me. Perhaps the default user agent could be updated to include the magic bot substring mentioned in !1364?

Code looks good to me. Perhaps the default user agent could be updated to include the magic `bot` substring mentioned in !1364?
👍 1
trashpanda commented 2026-02-14 19:50:04 +00:00
Author
Contributor
Copy link

Won't be able to come back to this until tonight, but I'll get that change in! I'll implement the one Jade suggested here.

Won't be able to come back to this until tonight, but I'll get that change in! I'll implement the one Jade suggested [here](https://forgejo.ellis.link/continuwuation/continuwuity/issues/1364#issuecomment-24048).
👍 1
gamesguru commented 2026-02-15 02:44:00 +00:00
Contributor
Copy link

This is an important fix i'd like to get in! Have you added the .toml section yet?

I honestly think making it configurable could be a bit overkill.

If you guys are happy to fix the thumbnail, preview, or whichever relevant endpoints with the embedbot suffix section, that would be a simple compromise perhaps

This is an important fix i'd like to get in! Have you added the `.toml` section yet? I honestly think making it configurable could be a bit overkill. If you guys are happy to fix the thumbnail, preview, or whichever relevant endpoints with the `embedbot` suffix section, that would be a simple compromise perhaps
trashpanda changed title from WIP: feat: customize url preview user agent to WIP: feat: Customize url preview user agent 2026-02-15 02:45:07 +00:00
feat(core): Change default user agent
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 20s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 4m13s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 21m38s
Details
c348096c4c
trashpanda changed title from WIP: feat: Customize url preview user agent to feat: Customize url preview user agent 2026-02-15 02:48:58 +00:00
fix: Clippy lints
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 27s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m10s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
5564865a00
trashpanda force-pushed trashpanda/url-preview-ua from 5564865a00
Some checks failed
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Update flake hashes / update-flake-hashes (pull_request) Successful in 27s
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m10s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled
Details
to 975d33bddf
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m58s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 45m3s
Details
2026-02-15 03:32:00 +00:00 Compare
Jade force-pushed trashpanda/url-preview-ua from 975d33bddf
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m58s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 45m3s
Details
to 564edcb1f4
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m31s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 36m10s
Details
2026-02-15 14:33:35 +00:00 Compare
Jade approved these changes 2026-02-15 14:34:59 +00:00
Jade left a comment
Owner
Copy link

Should be good to merge now. Don't forget to build your changes, both to test it locally and to generate the config file.

Should be good to merge now. Don't forget to build your changes, both to test it locally and to generate the config file.
ginger requested changes 2026-02-15 14:43:58 +00:00
Dismissed
src/core/config/mod.rs Outdated
@ -1699,0 +1699,4 @@
/// User agent that is used specifically when fetching url previews.
///
/// default: "continuwuity/<version> (bot; +https://continuwuity.org)"
pub url_preview_user_agent: Option<String>,
ginger commented 2026-02-15 14:43:53 +00:00
Owner
Copy link

Does this need to be an Option<String> if it has a default?

Does this need to be an `Option<String>` if it has a default?
trashpanda commented 2026-02-15 14:52:12 +00:00
Author
Contributor
Copy link

Currently, yes. I imagine the default updating with the actual version is preferred, as long as people don't have it explicitly set? Unless we want to do a little templating and replace any instances of <version> with the version_ua() string?

Currently, yes. I imagine the default updating with the actual version is preferred, as long as people don't have it explicitly set? Unless we want to do a little templating and replace any instances of `<version>` with the `version_ua()` string?
Jade commented 2026-02-15 15:13:09 +00:00
Owner
Copy link

It's generated outside of the config. Could have probably been done the other way (moving the init into the config default function), but doesn't matter now

It's generated outside of the config. Could have probably been done the other way (moving the init into the config default function), but doesn't matter now
ginger commented 2026-02-15 15:13:53 +00:00
Owner
Copy link

@trashpanda wrote in #1372 (comment):

Currently, yes. I imagine the default updating with the actual version is preferred, as long as people don't have it explicitly set? Unless we want to do a little templating and replace any instances of <version> with the version_ua() string?

Ah, that makes sense.

@trashpanda wrote in https://forgejo.ellis.link/continuwuation/continuwuity/pulls/1372#issuecomment-24160: > Currently, yes. I imagine the default updating with the actual version is preferred, as long as people don't have it explicitly set? Unless we want to do a little templating and replace any instances of `<version>` with the `version_ua()` string? Ah, that makes sense.
ginger marked this conversation as resolved
ginger approved these changes 2026-02-15 15:14:05 +00:00
ginger commented 2026-02-15 15:14:54 +00:00
Owner
Copy link

oh, this needs a new fragment

oh, this needs a new fragment
Jade force-pushed trashpanda/url-preview-ua from 564edcb1f4
All checks were successful
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 2m31s
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Successful in 36m10s
Details
to 26aac1408e
Some checks are pending
Documentation / Build and Deploy Documentation (pull_request) Waiting to run
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Waiting to run
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Waiting to run
Details
2026-02-15 15:21:18 +00:00 Compare
docs: Changelog
Some checks failed
Release Docker Image / Build linux-amd64 (release) (push) Waiting to run
Details
Release Docker Image / Build linux-arm64 (release) (push) Waiting to run
Details
Release Docker Image / Create Multi-arch Release Manifest (push) Blocked by required conditions
Details
Release Docker Image / Build linux-amd64 (max-perf) (push) Blocked by required conditions
Details
Release Docker Image / Build linux-arm64 (max-perf) (push) Blocked by required conditions
Details
Release Docker Image / Create Max-Perf Manifest (push) Blocked by required conditions
Details
Documentation / Build and Deploy Documentation (pull_request) Has been skipped
Details
Checks / Prek / Pre-commit & Formatting (pull_request) Successful in 1m54s
Details
Documentation / Build and Deploy Documentation (push) Has been cancelled
Details
Checks / Prek / Pre-commit & Formatting (push) Has been cancelled
Details
Checks / Prek / Clippy and Cargo Tests (push) Has been cancelled
Details
Checks / Prek / Clippy and Cargo Tests (pull_request) Failing after 1h0m0s
Details
f3f82831b4
Jade merged commit f3f82831b4 into main 2026-02-15 15:23:35 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Jade
ginger
Labels
Clear labels   
Blocked

This pull request or issue is currently blocked from being merged/closed

  
Bug

Something isn't working as intended

  
Cherry-picking

Commits picked from other conduit projects

  
Database

This requires or includes changes to the database

  
Dependencies

Something dependency related

  
Dependencies/Renovate

Automatic dependency upgrades by Renovate

  
Difficulty
Easy
Low difficulty to implement - touches few parts of the codebase, low complexity

  
Difficulty
Hard
High difficulty to implement - touches many parts of the codebase, high complexity

  
Difficulty
Medium
Medium difficulty to implement - touches more parts of the codebase, higher complexity

  
Documentation

Improvements or additions to documentation

  
Enhancement

New feature or request

  
Good first issue

Good for newcomers

  
Help wanted

Additional eyes and keyboards are required for this one

  
Inherited

Issues that have been inhereted from the project pre-fork

  
Matrix/Administration

Features pertaining to homeserver administration

  
Matrix/Appservices

Features pertaining to the appservice API

  
Matrix/Auth

Features pertaining to authentication

  
Matrix/Client

Features pertaining to client-to-server interactions

  
Matrix/Core

Issues relating to core matrix functionality, such as state resolution and PDU formats

  
Matrix/Federation

Features pertaining to server-to-server interactions

  
Matrix/Hydra

Issues related to room version 12 and related changes (temporary label)

  
Matrix/MSC

Features pertaining to unstable matrix features

  
Matrix/Media

Features pertaining to media interactions

  
Meta

Related to housekeeping, maintenance, or other repo-meta.

  
Meta/CI

Issues related to CI changes

  
Meta/Packaging

Packaging

  
Priority
Blocking
This issue is blocking the next release

  
Priority
High
This issue is very important

  
Priority
Low
This issue is of a rather low priority

  
Security

This item is related to general security

  
Status
Confirmed
This issue has enough information and is confirmed

  
Status
Duplicate
This issue or pull request already exists

  
Status
Invalid
This issue doesn't seem right

  
Status
Needs Investigation
This issue needs further investigation

  
Support

Questions or support requests

  
To-Merge

   
Wont fix

This will not be worked on

  
old/ci/cd

Ci/CD

Archived

  
old/rust

Pull requests that update Rust code

Archived

No labels
Blocked
Bug
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
To-Merge
Wont fix
old/ci/cd
old/rust
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!1372
No description provided.