perf(build): Option to use precompiled RocksDB. Isolate spurious debug crates from release #1509

Open

gamesguru wants to merge 48 commits from gamesguru/continuwuity:guru/perf/build-optimizations-dep-tree into main

Author	SHA1	Message	Date
Shane Jaroch	497dbf9164	tidy/arrange justfile fix rockdb glitch Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-14 14:59:06 -04:00
Shane Jaroch	f4407284d2	fix: justfile for snappy cmake error	2026-03-14 12:48:23 -04:00
Shane Jaroch	0721163df0	tweak justfile	2026-03-14 10:56:37 -04:00
Shane Jaroch	a67e86cf73	tab completion?	2026-03-14 10:36:12 -04:00
Shane Jaroch	e0bb2b3f42	simplify just targets, go back to `/usr/local/build`	2026-03-13 23:39:29 -04:00
Shane Jaroch	4a290c9848	simplify just target for jemalloc, fix to v5.3.0	2026-03-13 23:36:20 -04:00
Shane Jaroch	9fdcf9a889	add additional core deps to justfile prebuild	2026-03-13 23:31:57 -04:00
Shane Jaroch	300be31e5f	wip: idk snappy justfile stuff?	2026-03-12 23:40:56 -04:00
Shane Jaroch	38b48d32f0	build: add snappy explicitly under rocksdb	2026-03-12 23:40:56 -04:00
Shane Jaroch	820f1d5d1f	prune makefile; fix my stupid slowdowns/regressions, but not quite as good as `9e76153a` nor even `a6d377f8` yet again. Some checks failed Documentation / Build and Deploy Documentation (pull_request) Has been cancelled Details Checks / Prek / Pre-commit & Formatting (pull_request) Has been cancelled Details Checks / Prek / Clippy and Cargo Tests (pull_request) Has been cancelled Details Update flake hashes / update-flake-hashes (pull_request) Has been cancelled Details	2026-03-11 21:10:39 -04:00
Shane Jaroch	d7b975fb2c	lint: wrap `url_preview` in feature config flag	2026-03-11 21:03:42 -04:00
Shane Jaroch	d85c889b09	fix	2026-03-11 21:00:25 -04:00
Shane Jaroch	2ec113ea99	wip	2026-03-11 20:55:07 -04:00
Shane Jaroch	60b598c620	wip	2026-03-11 20:38:18 -04:00
Shane Jaroch	0ee24c7be4	seems linking is slow since adding mt_static (bad for devs) [empty-commit]	2026-03-11 19:37:26 -04:00
Shane Jaroch	ee8b4a25eb	restore presence/mod.rs to origin/main except comments on this branch	2026-03-11 19:33:49 -04:00
Shane Jaroch	b74600e490	fix bad 3-way merge	2026-03-11 19:29:55 -04:00
Shane Jaroch	5800d3c677	git: 3-way patch from upstream, see: from this branch (origin is Jade's, github is mine), I ran: ```shell FILES=$(gds origin/main --name-only) BASE=$(git merge-base HEAD github/main) git diff $BASE..github/main -- $FILES \| git apply --3way ```	2026-03-11 19:22:20 -04:00
Shane Jaroch	febd2881f1	Merge remote-tracking branch 'origin/main' into guru/perf/build-optimizations-dep-tree	2026-03-11 19:12:21 -04:00
Shane Jaroch	a6d377f8d1	prune makefile from this branch Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 14:34:45 -04:00
Shane Jaroch	bc90eb5e19	update justfile targets (fix). format files/whitespace. Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 13:21:12 -04:00
Shane Jaroch	714fdec98d	add some build/compile profiling targets to justfile Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 08:31:02 -04:00
Shane Jaroch	9422b5f761	gitignore the .tmp/ in case not everyone echoed a gitignore in it with '**' Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 02:41:48 -04:00
Shane Jaroch	40f9517824	TODO: investigate dep. inversion w/ ruma, since it rebuilds on trivial/unrelated changes it shouldn't. [empty-commit] Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 02:26:09 -04:00
Shane Jaroch	2ec6cafaa3	update "debugging over SSH" docs to use `rust-gdb` Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 01:59:45 -04:00
Shane Jaroch	d6ab95705a	cargo configs so RustRover uses prebuilt libraries Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 01:53:47 -04:00
Shane Jaroch	2b66ae9ad0	justfile targets to prebuild jemalloc, lz4, rocksdb, and zstd. down to 553 crates. Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 01:30:59 -04:00
Shane Jaroch	7c61c8e09a	add a proof of concept `justfile` for profiling cpu/mem usage and heat maps Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 00:48:49 -04:00
Shane Jaroch	33d71edec2	add docs on debugging over SSH with `gdb` or RustRover Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 00:36:04 -04:00
Shane Jaroch	78e24ffc48	help RustRover see. You may need below commands. And i may nest things eventually in `/usr/loca/uwu` for isolation. Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details This will tell your linker to look in `/usr/local`, but it may also cause conflicts or unexpected behavior: ```bash echo "/usr/local/lib" \| sudo tee /etc/ld.so.conf.d/usr-local.conf sudo ldconfig ```	2026-03-11 00:17:07 -04:00
Shane Jaroch	f767a1174d	RustRover default to prebuilt RocksDB if found in`/usr/local` Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-11 00:02:08 -04:00
Shane Jaroch	e9ec975f03	add redundant config since RustRover has the IQ of a carrot. TODO: prevent this breaking if sccache or mold aren't found Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-10 23:47:51 -04:00
Shane Jaroch	2fa147c9ec	direnv auto-detect sccache, faster builds from CLI Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-10 23:37:30 -04:00
Shane Jaroch	9f6fd04507	add cargo config to use `mold`, faster linker than `ld` or `cc` Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details this should also apply to RustRover by default, and you will notice less redundant rebuilding of built static imports.	2026-03-10 04:56:15 -04:00
Shane Jaroch	9e76153ac0	leaner runtime 560 crates. take tracers out of default features. give the people choice Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details 42 megabytes. 8 minutes. release-max-perf. we need to improve our documentation, so admins understand how to get non-default features. i suspect nothing additional is needed to run the full Complement Test suite and serve all basic homeserver functions. not even URL preview zero overhead: extreme cases may leak IPs or allow forgery attacks. there's no harm in recompiling once you find you want it. testing: add WIP engage target for complement tests (not everyone has buildkit or buildx)	2026-03-10 03:03:17 -04:00
Shane Jaroch	2a2c513205	document tech debt: wasteful presence DB i/o and a need for LRU caching Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details Update flake hashes / update-flake-hashes (pull_request) Blocked by required conditions Details	2026-03-10 01:42:32 -04:00
Shane Jaroch	b76ae4141a	build: back to mt_static apparently, demands RustRover	2026-03-10 01:37:33 -04:00
Shane Jaroch	110af0f227	more edits, but both are working fine	2026-03-10 01:09:48 -04:00
Shane Jaroch	605a56e8ec	build: prelim support for `bindgen`, bundling the RocksDB assembly into standalone Rust binaries revised commands to build rocksdb static assembly: ```shell DISABLE_JEMALLOC=1 EXTRA_CXXFLAGS="-Wno-error=unused-parameter" make shared_lib static_lib -j$(nproc) sudo make install-shared INSTALL_PATH=/usr/local sudo make install-static INSTALL_PATH=/usr/local sudo ldconfig ```	2026-03-10 00:33:15 -04:00
Shane Jaroch	d991d71299	update	2026-03-10 00:36:21 -04:00
Shane Jaroch	4112661e61	Merge remote-tracking branch 'origin/main' into guru/perf/build-optimizations-dep-tree	2026-03-09 21:57:03 -04:00
Shane Jaroch	594b39080e	update env vars defs for building. see below note on DISABLE_JEMALLOC=1, Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details ```shell git clone https://forgejo.ellis.link/continuwuation/rust-rocksdb-zaidoon1.git cd rust-rocksdb-zaidoon1 git checkout 61d9d23872197e9ace4a477f2617d5c9f50ecb23 git submodule update --init --recursive cd librocksdb-sys/rocksdb PORTABLE=0 DISABLE_JEMALLOC=1 CXXFLAGS="-Wno-error=unused-parameter" make shared_lib -j$(nproc) sudo make install-shared INSTALL_PATH=/usr/local sudo ldconfig ```	2026-03-09 05:50:13 -04:00
Shane Jaroch	1a6e84c0fc	build(docs): old compilers may need: `CXXFLAGS="-Wno-error=unused-parameter"` Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details	2026-03-08 21:11:44 -04:00
Shane Jaroch	a241fde347	build(perf): document benefits of pre-compiling rocksdb: Some checks are pending Documentation / Build and Deploy Documentation (pull_request) Blocked by required conditions Details Checks / Prek / Pre-commit & Formatting (pull_request) Blocked by required conditions Details Checks / Prek / Clippy and Cargo Tests (pull_request) Blocked by required conditions Details git clone https://forgejo.ellis.link/continuwuation/rust-rocksdb-zaidoon1.git cd rust-rocksdb-zaidoon1 git checkout 61d9d23872197e9ace4a477f2617d5c9f50ecb23 git submodule update --init --recursive cd librocksdb-sys/rocksdb make shared_lib -j$(nproc) sudo make install-shared INSTALL_PATH=/usr/local sudo ldconfig	2026-03-08 06:26:00 -04:00
Shane Jaroch	7a47720398	lint/compile errors due to using library system packages	2026-03-08 06:25:24 -04:00
Shane Jaroch	55ffd8bf00	wip: isolate some open telemetry behind feature flags	2026-03-08 06:23:58 -04:00
Shane Jaroch	b0ffc27bfc	fixup makefile (missing ROCKSDB_LIB_DIR=/usr/lib in one spot)	2026-03-08 06:23:57 -04:00
Shane Jaroch	6216f3dc55	perf(build): isolate tracing-flame from release builds, along with other heavy deps	2026-03-08 06:21:58 -04:00