WIP: ci: Tidy CI pipeline for readability and performance #924

Closed

tcpipuk wants to merge 1 commit from tom/fixes into main

Author SHA1 Message Date

Author	SHA1	Message	Date
Tom Foster	b481ff31c0	ci: Consolidate workflows and optimise build pipeline performance Some checks failed Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 31s Details Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 36s Details Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m21s Details Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 3m41s Details Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 4m12s Details Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s Details Merge rust-checks.yml and release-image.yml into unified ci-build.yml workflow that runs faster and more efficiently. The previous setup ran 4+ parallel jobs immediately (format, clippy, test, builds), causing resource contention. The new pipeline runs max 2 jobs in parallel at each stage, catching lint/format issues quickly before attempting expensive compilation. Extract all Rust setup logic from both workflows into reusable rust-with-cache composite action. This replaces 6 separate actions (rust-toolchain, sccache, timelord, plus inline APT/cache steps) with a single action that handles: - Rust toolchain installation with component selection - Cross-compilation configuration (previously scattered across release-image.yml) - System dependency installation with proper error handling - Comprehensive caching (sccache, cargo registry, cargo target, uv tools) - Timeline tracking and performance monitoring The previous release-image.yml had cross-compilation support but it was implemented inline with complex environment variables. The new rust-with-cache action centralises this with proper parameters for pkg-config paths, foreign architecture setup, and toolchain selection. Performance improvements make the pipeline fast enough to consolidate: - Warmed sccache cache shared between check and build stages - Optimised cargo target cache to exclude incremental/ and binaries (was caching entire target/ directory via buildkit-cache-dance) - Add restore-keys fallback for better cache hit rates - Parallel background tasks for Rust setup while APT runs - Fail-fast on format/lint errors before expensive compilation - Enable Haswell CPU optimisations for x86_64 builds (AVX2, FMA, etc.) - Add cross-language LTO (Link-Time Optimisation) for better performance Fix ARM64 cross-compilation reliability issues: - Move APT installations from background to foreground (background processes would hang during package downloads despite DEBIAN_FRONTEND=noninteractive) - Set proper pkg-config environment for cross-compilation - Configure APT sources to ports.ubuntu.com for foreign architectures - Replace hardened_malloc with jemalloc (ARM64 unsupported) Modernisation from previous commit (`b0ebdb59`): - prefligit renamed to prek (avoid typosquatting) - Direct uvx rustup replacing custom rust-toolchain action - Workflow renames: deploy-element, deploy-docs, docker-mirror - Renovate configuration for .forgejo/ workflows - fix-byte-order-marker replacing check-byte-order-marker Docker improvements: - Remove buildkit-cache-dance injection (now handled natively) - Align tag naming between arch-specific and multi-platform builds - Add branch- prefix for non-default branches - Reserve latest-{arch} tags for version releases only - Remove dynamic library extraction logic (ldd doesn't work for cross-compiled binaries; Rust --release produces mostly-static binaries) Additional improvements based on maintainer feedback: - Generate SBOM (Software Bill of Materials) for security compliance - Include SBOM in uploaded build artefacts alongside binary The consolidated pipeline completes in ~10 minutes with better resource utilisation and clearer failure diagnostics. Both x86_64 and ARM64 builds now work reliably with the centralised cross-compilation configuration.	2025-08-15 22:50:41 +01:00

Tom Foster

b481ff31c0

ci: Consolidate workflows and optimise build pipeline performance

Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 31s

Details

Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 36s

Details

Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m21s

Details

Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 3m41s

Details

Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 4m12s

Details

Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s

Details

Merge rust-checks.yml and release-image.yml into unified ci-build.yml
workflow that runs faster and more efficiently. The previous setup ran
4+ parallel jobs immediately (format, clippy, test, builds), causing
resource contention. The new pipeline runs max 2 jobs in parallel at
each stage, catching lint/format issues quickly before attempting
expensive compilation.

Extract all Rust setup logic from both workflows into reusable
rust-with-cache composite action. This replaces 6 separate actions
(rust-toolchain, sccache, timelord, plus inline APT/cache steps) with
a single action that handles:
- Rust toolchain installation with component selection
- Cross-compilation configuration (previously scattered across
  release-image.yml)
- System dependency installation with proper error handling
- Comprehensive caching (sccache, cargo registry, cargo target, uv
  tools)
- Timeline tracking and performance monitoring

The previous release-image.yml had cross-compilation support but it
was implemented inline with complex environment variables. The new
rust-with-cache action centralises this with proper parameters for
pkg-config paths, foreign architecture setup, and toolchain selection.

Performance improvements make the pipeline fast enough to consolidate:
- Warmed sccache cache shared between check and build stages
- Optimised cargo target cache to exclude incremental/ and binaries
  (was caching entire target/ directory via buildkit-cache-dance)
- Add restore-keys fallback for better cache hit rates
- Parallel background tasks for Rust setup while APT runs
- Fail-fast on format/lint errors before expensive compilation
- Enable Haswell CPU optimisations for x86_64 builds (AVX2, FMA, etc.)
- Add cross-language LTO (Link-Time Optimisation) for better performance

Fix ARM64 cross-compilation reliability issues:
- Move APT installations from background to foreground (background
  processes would hang during package downloads despite
  DEBIAN_FRONTEND=noninteractive)
- Set proper pkg-config environment for cross-compilation
- Configure APT sources to ports.ubuntu.com for foreign architectures
- Replace hardened_malloc with jemalloc (ARM64 unsupported)

Modernisation from previous commit (b0ebdb59):
- prefligit renamed to prek (avoid typosquatting)
- Direct uvx rustup replacing custom rust-toolchain action
- Workflow renames: deploy-element, deploy-docs, docker-mirror
- Renovate configuration for .forgejo/ workflows
- fix-byte-order-marker replacing check-byte-order-marker

Docker improvements:
- Remove buildkit-cache-dance injection (now handled natively)
- Align tag naming between arch-specific and multi-platform builds
- Add branch- prefix for non-default branches
- Reserve latest-{arch} tags for version releases only
- Remove dynamic library extraction logic (ldd doesn't work for
  cross-compiled binaries; Rust --release produces mostly-static binaries)

Additional improvements based on maintainer feedback:
- Generate SBOM (Software Bill of Materials) for security compliance
- Include SBOM in uploaded build artefacts alongside binary

The consolidated pipeline completes in ~10 minutes with better
resource utilisation and clearer failure diagnostics. Both x86_64 and
ARM64 builds now work reliably with the centralised cross-compilation
configuration.

2025-08-15 22:50:41 +01:00