perf(build): Option to use precompiled RocksDB. Isolate spurious debug crates from release #1509
Open
gamesguru
wants to merge 48 commits from
gamesguru/continuwuity:guru/perf/build-optimizations-dep-tree into main
pull from: gamesguru/continuwuity:guru/perf/build-optimizations-dep-tree
merge into: continuwuation:main
continuwuation:main
continuwuation:renovate/lock-file-maintenance
continuwuation:aranje/illegal-car-mods
continuwuation:renovate/recaptcha-verify-0.x
continuwuation:jade/community-guidelines
continuwuation:nex/experiment/push-gateway-logs
continuwuation:ginger/msc3575-obliteration
continuwuation:ginger/password-reset
continuwuation:renovate/docker-setup-qemu-action-4.x
continuwuation:renovate/rand_core-0.x
continuwuation:renovate/serde_html_form-0.x
continuwuation:renovate/reqwest-0.x
continuwuation:renovate/github-actions-non-major
continuwuation:nex/feat/block-busted-rooms
continuwuation:nex/fix/informative-startup-errs
continuwuation:ginger/no-left-room-initial-sync
continuwuation:nex/feat/policy-servers-2-electric-boogaloo
continuwuation:jade/docker-entrypoint
continuwuation:jade/dehydrated-devices
continuwuation:ginger/complement-fixes
continuwuation:nex/fix/stale-destination-cache
continuwuation:nex/experiment/sync-mutex
continuwuation:tcpipuk/docker-docs
continuwuation:jade/snafu
continuwuation:jade/rand-update
continuwuation:nex/stateres-refactor
continuwuation:ginger/779-in-troubleshooting
continuwuation:jade/liveit-guide
continuwuation:jade/http3
continuwuation:nex/feat/admin-hide-empty-rooms
continuwuation:ginger/oobe
continuwuation:nex/fix/debian-thingy
continuwuation:jade/ldap-admin-check
continuwuation:nex/fix/remote-restricted-joins
continuwuation:nex/feat/msc4406-sender-ignored
continuwuation:jade/deadlock-detection
continuwuation:nex/feat/room-shutdown
continuwuation:jade/get-started
continuwuation:jade/docs-guide
continuwuation:ginger/fix-local-invites
continuwuation:nex/fix/tpi
continuwuation:nex/feat/room-deletion
continuwuation:nex/feat/msc4322-media-redaction
continuwuation:ginger/stitched-order
continuwuation:jade/build-info
continuwuation:ginger/deps/update-rspress
continuwuation:jade/admin-announce-improvements
continuwuation:ginger/xtask-improvements
continuwuation:jade/improve-admin-config-display
continuwuation:nex/fix/better-stateres-error-logs
continuwuation:jade/sender-timeouts
continuwuation:nex/feat/custom-v12-room-ids
continuwuation:ginger/update-metadata
continuwuation:nex/feat/admin-force-logout
continuwuation:tom/max-perf-docs
continuwuation:nex/fix/invalid-appservice-reg
continuwuation:nex/feat/antispam
continuwuation:nex/feat/account-locking
continuwuation:jade/logging-cleanup
continuwuation:jade/remove-legacy-appservice-auth
continuwuation:nex/fix/key-query
continuwuation:jade/update-prek
continuwuation:nex/fix/room-summaries
continuwuation:ginger/restrict-admin-commands
continuwuation:ginger/enable-console-by-default
continuwuation:jade/tag-fixes
continuwuation:jade/otlp
continuwuation:nex/meta/pull-req-template
continuwuation:nex/fix/fed-invite-compliance
continuwuation:nex/feat/build-commit
continuwuation:nex/feat/join-logging
continuwuation:jade/mailmap-updates
continuwuation:jade/hack-ci-tmp
continuwuation:jade/v12-stable
continuwuation:jade/relations
continuwuation:ginger/database-refactor
continuwuation:jade/fix-ldap-uiaa
continuwuation:nex/fix/validation
continuwuation:ginger/nuke-invalid-msc4133-fields-in-migration
continuwuation:ginger/downgrade-artifact-actions
continuwuation:oddlid/reload-fix
continuwuation:jade/fix-assert
continuwuation:ginger/sync-v3-cleanup
continuwuation:ginger/remove-absolute-action-urls
continuwuation:jade/website
continuwuation:nex/fix/backoff
continuwuation:ginger/fix-mdbook-for-0.5
continuwuation:ginger/no-docker-on-prs
continuwuation:backport/v0.5.0-rc.8-1
continuwuation:nex/fed-improvements
continuwuation:jade/rust-1.90
continuwuation:jade/mirror-dockerhub
continuwuation:jade/clippy-fixes
continuwuation:jade/fix-support
continuwuation:jade/clean-images
continuwuation:jade/wal-compression-type
continuwuation:jade/flake-clone
continuwuation:ginger/upload-rpms-on-schedule
continuwuation:nex/fix/incoming-fetch
continuwuation:nex/fix/upgrade
continuwuation:tom/ci-fedora-rpm
continuwuation:jade/ci-release-fix
continuwuation:jade/rocksdb-10-5
continuwuation:ginger/fix-msc4133-migration
continuwuation:ginger/migrate-busted-tz
continuwuation:hydra/public
continuwuation:nex/feat/manual-extremities
continuwuation:nex/feat/async-media
continuwuation:nex/feat/fast-joins-hack-do-not-use-DO-NOT-USE
continuwuation:nex/feat/better-logging
continuwuation:trigger-ci-so-latest-isnt-on-illegal-car-mods
continuwuation:nex/feat/pins-backfill
continuwuation:jade/tuwunel-2025-06-old
continuwuation:jade/ai-slop-db-docs
continuwuation:nex/fix-create-auth
continuwuation:jade/version-stats
continuwuation:jade/read-receipts
continuwuation:jade/rust-toolchain-no-targets
continuwuation:jade/logging-features
continuwuation:jade/syncv5-typing
continuwuation:jade/msc2815
continuwuation:jade/purge-sync-tokens
continuwuation:morguldir/see-eye
continuwuation:jade/css-small-screen
continuwuation:nex/wip-751
continuwuation:tuwunel-rebase
continuwuation:test
continuwuation:oddlid/rename-admin-room-bot
continuwuation:strawberry/nix-ci-stuff
continuwuation:strawberry/valgrind
continuwuation:phonemain
continuwuation:strawberry/morgs-snake-sync-jason-main
continuwuation:newer-media-endpoints
continuwuation:folly-coroutines-async-io
continuwuation:federation-retry-timer-port
continuwuation:bad-attempt-at-extracting-homeserver-signing-key
continuwuation:room-deletion-attempt-do-not-use
No reviewers
Labels
Clear labels
This pull request or issue is currently blocked from being merged/closed
Something isn't working as intended
Commits picked from other conduit projects
This requires or includes changes to the database
Something dependency related
Automatic dependency upgrades by Renovate
Low difficulty to implement - touches few parts of the codebase, low complexity
High difficulty to implement - touches many parts of the codebase, high complexity
Medium difficulty to implement - touches more parts of the codebase, higher complexity
Improvements or additions to documentation
New feature or request
Good for newcomers
Additional eyes and keyboards are required for this one
Issues that have been inhereted from the project pre-fork
Features pertaining to homeserver administration
Features pertaining to the appservice API
Features pertaining to authentication
Features pertaining to client-to-server interactions
Issues relating to core matrix functionality, such as state resolution and PDU formats
Features pertaining to server-to-server interactions
Issues related to room version 12 and related changes (temporary label)
Features pertaining to unstable matrix features
Features pertaining to media interactions
Changes or issues related to trust & safety tooling
Related to housekeeping, maintenance, or other repo-meta.
Issues related to CI changes
Packaging
This issue is blocking the next release
This issue is very important
This issue is of a rather low priority
This item is related to general security
This issue has enough information and is confirmed
This issue or pull request already exists
This issue doesn't seem right
This issue needs further investigation
Questions or support requests
This will not be worked on
Ci/CD
Pull requests that update Rust code
Blocked
This pull request or issue is currently blocked from being merged/closed
Bug
Something isn't working as intended
Cherry-picking
Commits picked from other conduit projects
Database
This requires or includes changes to the database
Dependencies
Something dependency related
Dependencies/Renovate
Automatic dependency upgrades by Renovate
Difficulty
Easy
Low difficulty to implement - touches few parts of the codebase, low complexity
Difficulty
Hard
High difficulty to implement - touches many parts of the codebase, high complexity
Difficulty
Medium
Medium difficulty to implement - touches more parts of the codebase, higher complexity
Documentation
Improvements or additions to documentation
Enhancement
New feature or request
Good first issue
Good for newcomers
Help wanted
Additional eyes and keyboards are required for this one
Inherited
Issues that have been inhereted from the project pre-fork
Matrix/Administration
Features pertaining to homeserver administration
Matrix/Appservices
Features pertaining to the appservice API
Matrix/Auth
Features pertaining to authentication
Matrix/Client
Features pertaining to client-to-server interactions
Matrix/Core
Issues relating to core matrix functionality, such as state resolution and PDU formats
Matrix/E2EE
Matrix/Federation
Features pertaining to server-to-server interactions
Matrix/Hydra
Issues related to room version 12 and related changes (temporary label)
Matrix/MSC
Features pertaining to unstable matrix features
Matrix/Media
Features pertaining to media interactions
Matrix/T&S
Changes or issues related to trust & safety tooling
Meta
Related to housekeeping, maintenance, or other repo-meta.
Meta/CI
Issues related to CI changes
Meta/Packaging
Packaging
Priority
Blocking
This issue is blocking the next release
Priority
High
This issue is very important
Priority
Low
This issue is of a rather low priority
Security
This item is related to general security
Status
Confirmed
This issue has enough information and is confirmed
Status
Duplicate
This issue or pull request already exists
Status
Invalid
This issue doesn't seem right
Status
Needs Investigation
This issue needs further investigation
Support
Questions or support requests
To-Merge
Wont fix
This will not be worked on
old/ci/cd
Ci/CD
Archived
old/rust
Pull requests that update Rust code
Archived
No labels
Blocked
Bug
Cherry-picking
Database
Dependencies
Dependencies/Renovate
Difficulty
Easy
Difficulty
Hard
Difficulty
Medium
Documentation
Enhancement
Good first issue
Help wanted
Inherited
Matrix/Administration
Matrix/Appservices
Matrix/Auth
Matrix/Client
Matrix/Core
Matrix/E2EE
Matrix/Federation
Matrix/Hydra
Matrix/MSC
Matrix/Media
Matrix/T&S
Meta
Meta/CI
Meta/Packaging
Priority
Blocking
Priority
High
Priority
Low
Security
Status
Confirmed
Status
Duplicate
Status
Invalid
Status
Needs Investigation
Support
To-Merge
Wont fix
old/ci/cd
old/rust
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
continuwuation/continuwuity!1509
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "gamesguru/continuwuity:guru/perf/build-optimizations-dep-tree"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This PR places heavy debug libraries like
tracing-flameBEHIND debug feature flags.I have also documented the benefits of pre-compiling RocksDB (see below), which requires exposing the
ROCKSDB_LIB_DIRenv var.Net result is a beefed up build process, or perhaps leaned down?
It won't cure the deadlocks in our runtime, but at least now we can compile in under 3 minutes
andor on 512 MB of ram (realistically probably not both at once!).This doesn't really help docker-in-docker, since you'd need to compile RustDB every CI run. But I think we may be able to find a docker image which supports the same major and minor version of the RocksDB headers, which would replace building a million lines of code with downloading 150 MB of binaries over enterprise-grade Ethernet.
Pull request checklist:
mainbranch, and the branch is named something other thanmain.myself, if applicable. This includes ensuring code compiles.
@ -59,16 +59,18 @@ standard = ["systemd","url_preview","zstd_compression","sentry_telemetry",Why disable sentry and opentelemetry?
(I use both of these in prod, as do others)
typically telemetry is opt-in
you guys will have to figure out what this means for your release process
This is a misunderstanding of what these options do.
@gamesguru
The telemetry here is opt-in via configuration. Asking end users to recompile continuwuity themselves for observability features seems a little odd, when these features are documented. I'd expect to have to compile a version without them, if I particularly wanted them gone.
Unless you're suggesting they add another build to the release pipeline with the telemetry features compiled in, which would double the current builds, this seems odd.
What i mean is that should be configured as part of the release. I am perfectly okay with you passing the telemetry flag in with every action, workflow, runner, or prod build agent you think needs it.
But i don't think it's appropriate that this should be hard-coded in the rust configurations. I don't think someone compiling from source should be forced or automatically opted-into telemetry unless they have the expertise or time to reverse engineer and modify our rust code's build process. We should allow people who wish to compile their own source the option to disable it.
I'm not even sure if rust supports enablement by default for certain flags? I'm even even okay with enabling the flag by default.
What I'm resisting is the idea that the flag shouldn't exist at all, or there shouldn't be an option to disable the feature.
I'm not sure what you mean by there's no option to disable the feature. You can select features when you compile.
--no-default-features --features "x y z"will only include featuresx,y, andz. I feel like that's not what you meant, but I can't tell what else you could be talking about, so do correct me if I'm wrong.Also, "enablement by default for certain flags"? It's done by default right now. That's what the default feature set is. None of this is forced upon people compiling from source or set in stone.
To clarify, the situation is currently that observability features are compiled in by default, but you always have the option to compile continuwuity without them.
I apologise, but I'm very confused.
for example
cargo build --features full,otlp_telemetry --profile releasewill after this PR accomplish exactly the same as the old default behavior that you're used to.i admittedly did not have time to document things in all the relevant places that may be affected. I am going to lean on the team a bit to help fill in those gaps and make sure this gets laid out as effectively and clearly as we can
ok so, even if you use the
--no-default-features(today onmain), it looks like it skips the runtime but still was compiling the crates.and that's really what this PR is about, cutting build times drastically and making caching of our largest dependency, which has stayed at the same version for a while, trivially easy.
if it's too confusing what to do now about telemetry, we need to figure out some ideas there
plus, even if it no-ops the runtime... a open-source project like ours should probably exclude the telemetry SDK from the build by default
I think the confusion here comes from the term "telemetry" when it is actually for monitoring and health checks, which are incredibly useful and normal features for production server software to have. Caddy, for example, comes with similar features built in.
If you want to omit them by default, that feels like it should be a separate PR.
Some of what I said was wrong and you were right that
--no-default-featuresworks the same as this branch. But that seems like a bit obscure of a flag, and by default a LOT more is included, i.e., I'm not sure whattracing-flameis but it's 2nd slowest for me after RocksDB, and this PR cuts it out by default. I just monitor withjournalctlandsar. Feels like we can maybe revisit adding flag(s) for 3rd party library inclusion. I noticed the anti spam is included even with no-default. I don't use any of the "plugins" so I can't say how they're best organized behind feature flags. And I need to look a bit more how they're connected and the binary size and compile times they add.What is the difference then with full and default?
I guess in the interests of getting this thru I will revert that part for now, since you told me about the no default features flag. That should speed my builds up on its own 👍
CXXFLAGS="-Wno-error=unused-parameter"Build(perf): Option to use precompiled RocksDB. Isolate spurious debug crates from releaseto perf(build): Option to use precompiled RocksDB. Isolate spurious debug crates from releasebindgen, bundling the RocksDB assembly into standalone Rust binaries 605a56e8ecmold, faster linker thanldorcc/usr/local/usr/loca/uwufor isolation.gdbor RustRoverb5565021e933d71edec2justfilefor profiling cpu/mem usage and heat mapsrust-gdburl_previewin feature config flag d7b975fb2c9e76153anor evena6d377f8yet again./usr/local/builde0bb2b3f42Guys, I think what i'm gonna do here is open another PR that makes non-breaking, emergency changes that by default will use the following tools (if installed and available). But this new PR will otherwise change nothing about
--featuresstuff and things that might completely break DevOps pipelines and require a bit more collaboration and thinking things through.So my new PR will configure at the project level, i.e., thru
cargonotdirenvor anything else shell-level based that has a tendency to not be picked up by mycronjobs and five other tools on 3 different machines.moldclang(think we do this already)sccacherocksdb, jemalloc, snappy, zstd, lz4, bzip2, and liburingThe feature flag changes are less important to me.
My laptop and poor VPS simply cannot handle building RocksDB concurrently with the rust project, and they benefit too immensely from these speedups as to neglect expediency here on my part.
I'm not able to quickly build my PRs without first merging this branch (which contains other changes in
src/and therefore arguably pollutes my tests of the PRs).View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.