WIP: ci: Tidy CI pipeline for readability and performance #924

Closed
tcpipuk wants to merge 1 commit from tom/fixes into main
Owner

While looking at complement, I've prepared some tidy-ups that shave precious time off the CI checks, use caching more, and update prefligit to the new version of prek.

Pipeline Modernisation:

  • Updated to prek (prefligit renamed to avoid typosquatting)
  • Adopted uvx for tool invocation (prek, rustup) without install scripts
  • Consolidated workflow steps to run in parallel where possible
  • Removed redundant custom actions in favour of direct uvx commands
  • Made sccache optional (based on secrets availability) for fork compatibility

Workflow Restructuring:

  • Renamed workflows for clarity and consistency:
    • rust-checks.yml → ci-checks.yml (now includes prek linting)
    • clients-element.yml → deploy-element.yml
    • documentation.yml → deploy-docs.yml
    • release-image.yml → release-builds.yml (builds images AND binaries)
    • mirror-images.yml → docker-mirror.yml
  • Simplified job and step naming throughout for better readability

Performance Improvements:

  • Parallel execution of independent steps (formatting and prek checks)
  • Optimised caching strategy for uv and Rust dependencies
  • Reduced CI execution time through better parallelisation

Dependency Management:

  • Added Renovate configuration for automated dependency updates
  • Configured to monitor both .forgejo/ and .github/ workflows
  • Set up automatic PR creation for non-major GitHub Actions updates
While looking at complement, I've prepared some tidy-ups that shave precious time off the CI checks, use caching more, and update prefligit to the new version of prek. Pipeline Modernisation: - Updated to prek (prefligit renamed to avoid typosquatting) - Adopted uvx for tool invocation (prek, rustup) without install scripts - Consolidated workflow steps to run in parallel where possible - Removed redundant custom actions in favour of direct uvx commands - Made sccache optional (based on secrets availability) for fork compatibility Workflow Restructuring: - Renamed workflows for clarity and consistency: * rust-checks.yml → ci-checks.yml (now includes prek linting) * clients-element.yml → deploy-element.yml * documentation.yml → deploy-docs.yml * release-image.yml → release-builds.yml (builds images AND binaries) * mirror-images.yml → docker-mirror.yml - Simplified job and step naming throughout for better readability Performance Improvements: - Parallel execution of independent steps (formatting and prek checks) - Optimised caching strategy for uv and Rust dependencies - Reduced CI execution time through better parallelisation Dependency Management: - Added Renovate configuration for automated dependency updates - Configured to monitor both .forgejo/ and .github/ workflows - Set up automatic PR creation for non-major GitHub Actions updates
tcpipuk self-assigned this 2025-08-12 11:20:11 +00:00
ci: modernise CI pipeline with improved tooling and workflow structure
All checks were successful
CI Checks / Prek & Format (push) Successful in 42s
CI Checks / Prek & Format (pull_request) Successful in 1m1s
CI Checks / Clippy (push) Successful in 7m43s
CI Checks / Tests (push) Successful in 7m43s
Documentation / Build and Deploy Documentation (pull_request) Successful in 52s
CI Checks / Clippy (pull_request) Successful in 4m22s
CI Checks / Tests (pull_request) Successful in 3m37s
a71e584a25
Pipeline Modernisation:
- Updated to prek (prefligit renamed to avoid typosquatting)
- Adopted uvx for tool invocation (prek, rustup) without install scripts
- Consolidated workflow steps to run in parallel where possible
- Removed redundant custom actions in favour of direct uvx commands
- Made sccache optional (based on secrets availability) for fork compatibility

Workflow Restructuring:
- Renamed workflows for clarity and consistency:
  * rust-checks.yml → ci-checks.yml (now includes prek linting)
  * clients-element.yml → deploy-element.yml
  * documentation.yml → deploy-docs.yml
  * release-image.yml → release-builds.yml (builds images AND binaries)
  * mirror-images.yml → docker-mirror.yml
- Simplified job and step naming throughout for better readability

Performance Improvements:
- Parallel execution of independent steps (formatting and prek checks)
- Optimised caching strategy for uv and Rust dependencies
- Reduced CI execution time through better parallelisation

Dependency Management:
- Added Renovate configuration for automated dependency updates
- Configured to monitor both .forgejo/ and .github/ workflows
- Set up automatic PR creation for non-major GitHub Actions updates
tcpipuk force-pushed tom/fixes from a71e584a25
All checks were successful
CI Checks / Prek & Format (push) Successful in 42s
CI Checks / Prek & Format (pull_request) Successful in 1m1s
CI Checks / Clippy (push) Successful in 7m43s
CI Checks / Tests (push) Successful in 7m43s
Documentation / Build and Deploy Documentation (pull_request) Successful in 52s
CI Checks / Clippy (pull_request) Successful in 4m22s
CI Checks / Tests (pull_request) Successful in 3m37s
to 98632d511b
Some checks failed
CI Checks / Prek & Format (push) Failing after 11s
Release Builds / Prepare Build Matrix (push) Successful in 12s
CI Checks / Prek & Format (pull_request) Failing after 18s
CI Checks / Tests (push) Successful in 8m41s
CI Checks / Clippy (push) Successful in 8m54s
CI Checks / Clippy (pull_request) Successful in 8m15s
Documentation / Build and Deploy Documentation (pull_request) Successful in 44s
CI Checks / Tests (pull_request) Successful in 4m56s
Release Builds / Build Images & Binaries (push) Successful in 18m11s
Release Builds / Publish Multi-platform Manifest (push) Successful in 14s
2025-08-12 11:44:31 +00:00
Compare
Owner

Fwiw sccache only needs the secrets to download files from GitHub without rate limiting, I'm not sure why that action forces them. If you can get the binary in a different way that might be better

Fwiw sccache only needs the secrets to download files from GitHub without rate limiting, I'm not sure why that action forces them. If you can get the binary in a different way that might be better
Owner

Renaming things is redundant by the way, and breaks the branch protection rule (Checks / *)

Renaming things is redundant by the way, and breaks the branch protection rule (`Checks / *`)
@ -0,0 +43,4 @@
// Extract hostname from server URL
const serverUrl = '${{ github.server_url }}'
const registry = serverUrl.replace(/^https?:\/\//, '')
Owner

This is wrong - would extract the registry as github.com rather than ghcr.io on github for example. Perhaps there's an env variable for this? Anyway

This is wrong - would extract the registry as github.com rather than ghcr.io on github for example. Perhaps there's an env variable for this? Anyway
Author
Owner

There isn't, github.server_url is the only one I could find in the docs... we could specify an env var for the preferred Docker repository though, which would make this easier - my problem was that it wasn't at all portable, so when I tried to run CI on my fork, it immediately collapsed when trying to auth with forgejo.ellis.link

There isn't, `github.server_url` is the only one I could find in the docs... we could specify an env var for the preferred Docker repository though, which would make this easier - my problem was that it wasn't at all portable, so when I tried to run CI on my fork, it immediately collapsed when trying to auth with forgejo.ellis.link
Author
Owner

Perhaps env.BUILTIN_REGISTRY could be removed from the top and we just declare it on the repo, then I'll adjust this to skip trying to upload an image if that's missing?

Perhaps `env.BUILTIN_REGISTRY` could be removed from the top and we just declare it on the repo, then I'll adjust this to skip trying to upload an image if that's missing?
Owner

Yeah that works

Yeah that works
Owner

It would be vars.

It would be vars.
Author
Owner

I've set BUILTIN_REGISTRY on my version and this repo, and testing a version that uses vars.BUILTIN_REGISTRY instead, if that's ok?

(Edit: Ah ha, you'd already replied!)

I've set `BUILTIN_REGISTRY` on my version and this repo, and testing a version that uses `vars.BUILTIN_REGISTRY` instead, if that's ok? (Edit: Ah ha, you'd already replied!)
tcpipuk marked this conversation as resolved
@ -0,0 +120,4 @@
env:
DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,index
- name: Get commit SHA
Owner

Short commit sha specified explicitly here for a reason

Short commit sha specified explicitly here for a reason
tcpipuk marked this conversation as resolved
Owner

I can't actually review this with the renames because it breaks the diff.

I can't actually review this with the renames because it breaks the diff.
@ -0,0 +194,4 @@
labels: ${{ steps.meta.outputs.labels }}
annotations: ${{ steps.meta.outputs.annotations }}
cache-from: type=gha
# cache-to: type=gha,mode=max
Owner

Have you tried enabling this again?

Have you tried enabling this again?
tcpipuk marked this conversation as resolved
@ -0,0 +57,4 @@
LOG_LEVEL: ${{ inputs.logLevel || 'info' }}
# Forgejo/Gitea specific
RENOVATE_GIT_AUTHOR: 'Renovate Bot <renovate@noreply.${{ github.server_url }}>'
Owner

Should be configurable so I can get this to actually match up with the account

Should be configurable so I can get this to actually match up with the account
Author
Owner

Sure! Would you prefer it configurable as one line, or individual vars for name and email?

Sure! Would you prefer it configurable as one line, or individual vars for name and email?
Owner

Probably easier as just one line

Probably easier as just one line
Author
Owner

Cool, I've added RENOVATE_AUTHOR with an example value if you want to update it 🙂

Cool, I've added `RENOVATE_AUTHOR` with an example value if you want to update it 🙂
tcpipuk marked this conversation as resolved
@ -0,0 +60,4 @@
RENOVATE_GIT_AUTHOR: 'Renovate Bot <renovate@noreply.${{ github.server_url }}>'
# PR settings
RENOVATE_PR_HOURLY_LIMIT: '2'
Owner

Pretty sure you already set this in the config file

Pretty sure you already set this in the config file
tcpipuk marked this conversation as resolved
tcpipuk force-pushed tom/fixes from 98632d511b
Some checks failed
CI Checks / Prek & Format (push) Failing after 11s
Release Builds / Prepare Build Matrix (push) Successful in 12s
CI Checks / Prek & Format (pull_request) Failing after 18s
CI Checks / Tests (push) Successful in 8m41s
CI Checks / Clippy (push) Successful in 8m54s
CI Checks / Clippy (pull_request) Successful in 8m15s
Documentation / Build and Deploy Documentation (pull_request) Successful in 44s
CI Checks / Tests (pull_request) Successful in 4m56s
Release Builds / Build Images & Binaries (push) Successful in 18m11s
Release Builds / Publish Multi-platform Manifest (push) Successful in 14s
to 667f5f3472
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 48s
Release / Builds / Prepare Build Matrix (push) Successful in 3s
Checks / CI / Prek & Format (pull_request) Successful in 56s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s
Checks / CI / Clippy (pull_request) Successful in 6m26s
Checks / CI / Tests (pull_request) Successful in 8m59s
2025-08-13 20:07:35 +00:00
Compare
tcpipuk force-pushed tom/fixes from 667f5f3472
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 48s
Release / Builds / Prepare Build Matrix (push) Successful in 3s
Checks / CI / Prek & Format (pull_request) Successful in 56s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m17s
Checks / CI / Clippy (pull_request) Successful in 6m26s
Checks / CI / Tests (pull_request) Successful in 8m59s
to 69d8c82a7f
Some checks failed
Checks / CI / Clippy (push) Waiting to run
Checks / CI / Tests (push) Waiting to run
Release / Builds / Prepare Build Matrix (push) Waiting to run
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Has been cancelled
Checks / CI / Prek & Format (pull_request) Successful in 49s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 59s
Checks / CI / Clippy (pull_request) Successful in 8m3s
Checks / CI / Tests (pull_request) Successful in 8m46s
2025-08-13 20:12:23 +00:00
Compare
tcpipuk force-pushed tom/fixes from 69d8c82a7f
Some checks failed
Checks / CI / Clippy (push) Waiting to run
Checks / CI / Tests (push) Waiting to run
Release / Builds / Prepare Build Matrix (push) Waiting to run
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Has been cancelled
Checks / CI / Prek & Format (pull_request) Successful in 49s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 59s
Checks / CI / Clippy (pull_request) Successful in 8m3s
Checks / CI / Tests (pull_request) Successful in 8m46s
to 087beca7fc
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 45s
Release / Builds / Prepare Build Matrix (push) Successful in 6s
Checks / CI / Prek & Format (pull_request) Successful in 58s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m11s
Checks / CI / Clippy (pull_request) Successful in 9m29s
Checks / CI / Tests (pull_request) Successful in 9m3s
2025-08-13 20:15:04 +00:00
Compare
tcpipuk force-pushed tom/fixes from 087beca7fc
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 45s
Release / Builds / Prepare Build Matrix (push) Successful in 6s
Checks / CI / Prek & Format (pull_request) Successful in 58s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m11s
Checks / CI / Clippy (pull_request) Successful in 9m29s
Checks / CI / Tests (pull_request) Successful in 9m3s
to a20fe8fc8e
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 1m12s
Release / Builds / Prepare Build Matrix (push) Successful in 9s
Checks / CI / Clippy (push) Successful in 7m36s
Checks / CI / Prek & Format (pull_request) Successful in 1m3s
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 50s
Checks / CI / Clippy (pull_request) Successful in 4m35s
Checks / CI / Tests (pull_request) Successful in 6m24s
2025-08-13 20:26:23 +00:00
Compare
tcpipuk force-pushed tom/fixes from a20fe8fc8e
Some checks failed
Release / Builds / Build Images & Binaries (push) Blocked by required conditions
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Checks / CI / Prek & Format (push) Successful in 1m12s
Release / Builds / Prepare Build Matrix (push) Successful in 9s
Checks / CI / Clippy (push) Successful in 7m36s
Checks / CI / Prek & Format (pull_request) Successful in 1m3s
Checks / CI / Tests (push) Has been cancelled
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 50s
Checks / CI / Clippy (pull_request) Successful in 4m35s
Checks / CI / Tests (pull_request) Successful in 6m24s
to 741a26a7c5
Some checks failed
Checks / CI / Prek & Format (push) Successful in 39s
Release / Builds / Prepare Build Matrix (push) Successful in 6s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m4s
Checks / CI / Tests (push) Successful in 8m27s
Checks / CI / Clippy (push) Successful in 6m56s
Release / Builds / Build Images & Binaries (push) Failing after 14m56s
Release / Builds / Publish Multi-platform Manifest (push) Has been skipped
2025-08-13 20:39:21 +00:00
Compare
tcpipuk force-pushed tom/fixes from 741a26a7c5
Some checks failed
Checks / CI / Prek & Format (push) Successful in 39s
Release / Builds / Prepare Build Matrix (push) Successful in 6s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m4s
Checks / CI / Tests (push) Successful in 8m27s
Checks / CI / Clippy (push) Successful in 6m56s
Release / Builds / Build Images & Binaries (push) Failing after 14m56s
Release / Builds / Publish Multi-platform Manifest (push) Has been skipped
to 39e5e2b119
Some checks failed
Checks / CI / Prek & Format (push) Successful in 37s
Release / Builds / Prepare Build Matrix (push) Successful in 4s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s
Checks / CI / Clippy (push) Successful in 8m15s
Checks / CI / Tests (push) Successful in 8m16s
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Release / Builds / Build Images & Binaries (push) Has been cancelled
2025-08-14 10:13:06 +00:00
Compare
tcpipuk force-pushed tom/fixes from 39e5e2b119
Some checks failed
Checks / CI / Prek & Format (push) Successful in 37s
Release / Builds / Prepare Build Matrix (push) Successful in 4s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m16s
Checks / CI / Clippy (push) Successful in 8m15s
Checks / CI / Tests (push) Successful in 8m16s
Release / Builds / Publish Multi-platform Manifest (push) Blocked by required conditions
Release / Builds / Build Images & Binaries (push) Has been cancelled
to 8c1b59c684
All checks were successful
Checks / CI / Prek & Format (push) Successful in 35s
Release / Builds / Prepare Build Matrix (push) Successful in 8s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m12s
Checks / CI / Tests (push) Successful in 5m45s
Release / Builds / Build Images & Binaries (push) Successful in 15m44s
Release / Builds / Publish Multi-platform Manifest (push) Successful in 20s
Checks / CI / Clippy (push) Successful in 2m52s
2025-08-14 10:53:12 +00:00
Compare
tcpipuk force-pushed tom/fixes from 8c1b59c684
All checks were successful
Checks / CI / Prek & Format (push) Successful in 35s
Release / Builds / Prepare Build Matrix (push) Successful in 8s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m12s
Checks / CI / Tests (push) Successful in 5m45s
Release / Builds / Build Images & Binaries (push) Successful in 15m44s
Release / Builds / Publish Multi-platform Manifest (push) Successful in 20s
Checks / CI / Clippy (push) Successful in 2m52s
to 74b0760982
Some checks failed
Checks / CI / Prek & Format (push) Successful in 27s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 56s
Checks / CI / Tests (push) Has been cancelled
Checks / CI / Clippy (push) Has been cancelled
2025-08-14 11:37:50 +00:00
Compare
tcpipuk changed title from ci: tidy CI pipeline for readability and performance to ci: Tidy CI pipeline for readability and performance 2025-08-14 11:38:25 +00:00
tcpipuk force-pushed tom/fixes from 74b0760982
Some checks failed
Checks / CI / Prek & Format (push) Successful in 27s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 56s
Checks / CI / Tests (push) Has been cancelled
Checks / CI / Clippy (push) Has been cancelled
to a08c6aacc5
Some checks failed
Checks / CI / Prek & Format (push) Successful in 38s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 59s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
2025-08-14 11:40:55 +00:00
Compare
tcpipuk force-pushed tom/fixes from a08c6aacc5
Some checks failed
Checks / CI / Prek & Format (push) Successful in 38s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 59s
Checks / CI / Clippy (push) Has been cancelled
Checks / CI / Tests (push) Has been cancelled
to b0ebdb59ed
All checks were successful
Checks / CI / Prek & Format (push) Successful in 39s
Release / Builds / Prepare Build Matrix (push) Successful in 3s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Release / Builds / Build Images & Binaries (push) Successful in 8m12s
Release / Builds / Publish Multi-platform Manifest (push) Successful in 12s
Checks / CI / Clippy (push) Successful in 3m4s
Checks / CI / Tests (push) Successful in 3m0s
2025-08-14 11:47:38 +00:00
Compare
tcpipuk force-pushed tom/fixes from b0ebdb59ed
All checks were successful
Checks / CI / Prek & Format (push) Successful in 39s
Release / Builds / Prepare Build Matrix (push) Successful in 3s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 1m3s
Release / Builds / Build Images & Binaries (push) Successful in 8m12s
Release / Builds / Publish Multi-platform Manifest (push) Successful in 12s
Checks / CI / Clippy (push) Successful in 3m4s
Checks / CI / Tests (push) Successful in 3m0s
to 50250f51fc
All checks were successful
Checks / Build / Publish / Pre-commit & Formatting (push) Successful in 35s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 45s
Checks / Build / Publish / Build linux/amd64 (push) Successful in 2m8s
Checks / Build / Publish / Build linux/arm64 (push) Successful in 2m17s
Checks / Build / Publish / Publish Multi-Platform (push) Successful in 28s
Checks / Build / Publish / Clippy and Cargo Tests (push) Successful in 3m40s
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 26s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m45s
Checks / Build / Publish / Build linux/amd64 (pull_request) Successful in 2m50s
Checks / Build / Publish / Build linux/arm64 (pull_request) Successful in 3m0s
Checks / Build / Publish / Publish Multi-Platform (pull_request) Successful in 23s
2025-08-15 20:15:27 +00:00
Compare
@ -0,0 +225,4 @@
# Stage 2: Final scratch image
FROM scratch
ARG GIT_COMMIT_HASH
Owner

these are all for build time

these are all for build time
Author
Owner

They were being passed in as build args from the workflow, but I'll convert them so we're directly substituting into envs in the workflow.

They were being passed in as build args from the workflow, but I'll convert them so we're directly substituting into envs in the workflow.
@ -0,0 +204,4 @@
# For native builds, extract libraries if dynamically linked
if [ "${{ matrix.platform }}" = "linux/amd64" ]; then
if ldd target/${{ matrix.rust_target }}/release/conduwuit 2>/dev/null | grep -q "=>" ; then
Owner

LDD doesn't work cor cross compile

LDD doesn't work cor cross compile
Owner

(you will end with none of the needed libs on cross-compiles)

(you will end with none of the needed libs on cross-compiles)
Author
Owner

It was only being done for linux/amd64 (which we can do it for) but I'll remove it to make all of the builds identical - when we don't extract them, we just end with a larger static build.

It was only being done for `linux/amd64` (which we can do it for) but I'll remove it to make all of the builds identical - when we don't extract them, we just end with a larger static build.
@ -0,0 +185,4 @@
--release \
--locked \
--features "standard,jemalloc_prof,perf_measurements,tokio_console" \
--target ${{ matrix.rust_target }}
Owner

Missing SBOM, missing ability to build for a specific target CPU (eg haswell), not doing cross-language LTO afaict

Missing SBOM, missing ability to build for a specific target CPU (eg haswell), not doing cross-language LTO afaict
Author
Owner

I've added SBOM and updated the x86_64 arch from "base" to "haswell" and enabled LTO 🙂

I've added SBOM and updated the x86_64 arch from "base" to "haswell" and enabled LTO 🙂
Owner

We still need to build for base because not all CPUs are post haswell

We still need to build for base because not all CPUs are post haswell
Author
Owner

Ok, I'd appreciate if you could be a bit clearer on how you'd prefer this to look then - I couldn't see any Docker tags for haswell, so how would you prefer it be delivered?

Ok, I'd appreciate if you could be a bit clearer on how you'd prefer this to look then - I couldn't see any Docker tags for haswell, so how would you prefer it be delivered?
Owner

The ability to pass in a target CPU that gets set as the needed compiler flags -

ARG TARGET_CPU

- and then to build that as a separate job in the matrix and publish the image with a suffix. This wasn't used previously because of the extra build time. Automatic selection of the correct binary from a single image could be achieved via https://forgejo.ellis.link/continuwuation/hwcaps-image but I never got around to it and I'm not sure about the thing being GPL3

The ability to pass in a target CPU that gets set as the needed compiler flags - https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/583cb924f1510960702f626c7397cb9e87e1722a/docker/Dockerfile#L114 - and then to build that as a separate job in the matrix and publish the image with a suffix. This wasn't used previously because of the extra build time. Automatic selection of the correct binary from a single image could be achieved via https://forgejo.ellis.link/continuwuation/hwcaps-image but I never got around to it and I'm not sure about the thing being GPL3
@ -0,0 +168,4 @@
gcc-package: ${{ matrix.gcc_package }}
gxx-package: ${{ matrix.gxx_package }}
liburing-package: ${{ matrix.liburing_package }}
is-cross-compile: ${{ matrix.is_cross_compile }}
Owner

Cross compiling should be effectively identical to normal builds

Cross compiling should be effectively identical to normal builds
Author
Owner

That would be the dream, but there are differences - in particular, pkg-config needs to be told it's a cross-compile, but we also need to download arch-specific liburing too.

That would be the dream, but there are differences - in particular, `pkg-config` needs to be told it's a cross-compile, but we also need to download arch-specific liburing too.
Owner

See how the Dockerfile handles it. For example, it has two sets of deps - host and target - and always installs both sets.

See how the Dockerfile handles it. For example, it has two sets of deps - host and target - and always installs both sets.
Author
Owner

I believe Docker emulates the architecture outside the container, so inside the container we just install for emulated architecture?

In this case, to use tools like sccache, we're cross-compiling without using Docker (just adding the binary to Docker afterwards) so we need to configure cross-compilation and tell pkg-config we're cross-compiling.

I believe Docker emulates the architecture outside the container, so inside the container we just install for emulated architecture? In this case, to use tools like sccache, we're cross-compiling without using Docker (just adding the binary to Docker afterwards) so we need to configure cross-compilation and tell `pkg-config` we're cross-compiling.
@ -0,0 +123,4 @@
cxx: g++
linker: gcc
dpkg_arch: ""
gcc_package: gcc
Owner

What's going on here? this really shoudn't need to be specified for every platform in the matrix.

What's going on here? this really shoudn't need to be specified for every platform in the matrix.
Author
Owner

The inputs save defining any OS-specific config inside the action, so literally all the compile config is defined in the workflow, then the action just configures the Rust environment to use what it's been told.

The inputs save defining any OS-specific config inside the action, so literally all the compile config is defined in the workflow, then the action just configures the Rust environment to use what it's been told.
Owner

.forgejo/actions/rust-with-cache/action.yml - where all the logic is - is already tied to bash and debian very specifically.

`.forgejo/actions/rust-with-cache/action.yml` - where all the logic is - is already tied to bash and debian very specifically.
Author
Owner

"OS" was intended to mean "architecture" but the runner commands are only written to use Debian-type because the runner is ubuntu-latest - the actual cross-compile config provided in the matrix strategy can target Darwin, FreeBSD, or even Windows if we have the necessary libraries.

"OS" was intended to mean "architecture" but the runner commands are only written to use Debian-type because the runner is `ubuntu-latest` - the actual cross-compile config provided in the matrix strategy can target Darwin, FreeBSD, or even Windows if we have the necessary libraries.
tcpipuk marked this conversation as resolved
@ -1,235 +0,0 @@
ARG RUST_VERSION=1
Owner

Shouldn't be deleting the dockerfile anyway.

Shouldn't be deleting the dockerfile anyway.
Author
Owner

It seemed a risk to leave the Dockerfile laying around if we never actually use it for builds, as it'll quickly fall out of maintenance... I can leave it in to be retired on a future date, whatever works for you 🤷‍♂️

It seemed a risk to leave the Dockerfile laying around if we never actually use it for builds, as it'll quickly fall out of maintenance... I can leave it in to be retired on a future date, whatever works for you 🤷‍♂️
Owner

I'll still be using (a slightly modified version) of it for my prod deploys.

But the main reason was I was still experimenting with getting it and #912 to work

I'll still be using (a slightly modified version) of it for my prod deploys. But the main reason was I was still experimenting with getting it and https://forgejo.ellis.link/continuwuation/continuwuity/issues/912 to work
Author
Owner

I've restored the Dockerfile, I'll look at whether anything needs updating later.

I've restored the Dockerfile, I'll look at whether anything needs updating later.
tcpipuk marked this conversation as resolved
@ -0,0 +1,409 @@
name: Checks / Build / Publish
Owner

Checks / means it blocks merges - should probably be separate (you can add depends across workflows IIRC)

`Checks /` means it blocks merges - should probably be separate (you can add depends across workflows IIRC)
Author
Owner

I'm not sure what you mean here - the current checks and builds run on every branch/PR, so I've organised them into one workflow to stop 6 jobs trying to run at once, to generate better cache, and complete faster. The same jobs will run, it'll just run more efficiently (and fast-fail the other jobs when linting/clippy fails).

I'm not sure what you mean here - the current checks and builds run on every branch/PR, so I've organised them into one workflow to stop 6 jobs trying to run at once, to generate better cache, and complete faster. The same jobs will run, it'll just run more efficiently (and fast-fail the other jobs when linting/clippy fails).
Owner

Everything under Checks / has to finish before you can hit merge - so you have to wait for the images to publish now, when you didn't before. Not as much of an issue because it's faster now though

Everything under `Checks /` has to finish before you can hit merge - so you have to wait for the images to publish now, when you didn't before. Not as much of an issue because it's faster now though
@ -0,0 +626,4 @@
echo "::group::🎯 Cross-compilation target setup"
# Wait for basic Rust setup to complete first
WAIT_START=$(date +%s)
Owner

for al bash scripts, given there are so many, it would probably be better if they were in files where they're syntax highlighted and not indented

for al bash scripts, given there are so many, it would probably be better if they were in files where they're syntax highlighted and not indented
Author
Owner

I'm not sure we can use GHA variables the same way inside Bash scripts in separate files... I could convert it to JavaScript if you'd rather, or just split into more actions so it's easier to audit in chunks?

I'm not sure we can use GHA variables the same way inside Bash scripts in separate files... I could convert it to JavaScript if you'd rather, or just split into more actions so it's easier to audit in chunks?
Owner

GHA vars can still be passed in via env: inputs - it's up to you tbh. It just gets kind of hard to read & edit code embedded into yaml lol

GHA vars can still be passed in via `env: ` inputs - it's up to you tbh. It just gets kind of hard to read & edit code embedded into yaml lol
Author
Owner

It is a big chonk, but it was partly why I tried to shift most of the common "make Rust work please" dependencies into a big action that extrapolates a lot of the mess out of the other workflows.

I'd be up for splitting it into more actions, just think it might be a rough experience to have all of the shell scripts separate in folders. I can see what that'd look like though?

It is a big chonk, but it was partly why I tried to shift most of the common "make Rust work please" dependencies into a big action that extrapolates a lot of the mess out of the other workflows. I'd be up for splitting it into more actions, just think it might be a rough experience to have all of the shell scripts separate in folders. I can see what that'd look like though?
Owner

Nah it's OK if you don't think it's worth it.

Nah it's OK if you don't think it's worth it.
tcpipuk force-pushed tom/fixes from 50250f51fc
All checks were successful
Checks / Build / Publish / Pre-commit & Formatting (push) Successful in 35s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 45s
Checks / Build / Publish / Build linux/amd64 (push) Successful in 2m8s
Checks / Build / Publish / Build linux/arm64 (push) Successful in 2m17s
Checks / Build / Publish / Publish Multi-Platform (push) Successful in 28s
Checks / Build / Publish / Clippy and Cargo Tests (push) Successful in 3m40s
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 26s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m45s
Checks / Build / Publish / Build linux/amd64 (pull_request) Successful in 2m50s
Checks / Build / Publish / Build linux/arm64 (pull_request) Successful in 3m0s
Checks / Build / Publish / Publish Multi-Platform (pull_request) Successful in 23s
to b9b33e3db7
Some checks failed
Checks / Build / Publish / Pre-commit & Formatting (push) Successful in 47s
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 37s
Checks / Build / Publish / Clippy and Cargo Tests (push) Successful in 5m11s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 37s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 5m43s
Checks / Build / Publish / Build linux/arm64 (push) Failing after 7m41s
Checks / Build / Publish / Build linux/amd64 (push) Failing after 6m49s
Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 2m27s
Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 3m0s
Checks / Build / Publish / Publish Multi-Platform (push) Failing after 0s
Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s
2025-08-15 21:41:01 +00:00
Compare
@ -0,0 +216,4 @@
# Copy binary
cp target/${{ matrix.rust_target }}/release/conduwuit docker-context/
# Note: We rely on Rust producing mostly-static binaries with --release
Owner

image

![image](/attachments/1aea912f-fe48-4fcb-aaea-f5ba8e2a706e)
651 KiB
Author
Owner

I'd prefer words as well as screenshots to communicate, but I think what you're saying here is that the current build method (that does everything inside an emulated architecture inside Docker) can use lld to pull out dynamic libraries, but you've also pointed out that we can't just use lld to pull out dynamic libraries on cross-compiled binaries.

I'm currently testing switching from gcc to clang so that lld can support the cross-compilation environment and potentially replicate how this was working in Docker... if that doesn't work then 🤷‍♂️

I'd prefer words as well as screenshots to communicate, but I _think_ what you're saying here is that the current build method (that does everything inside an emulated architecture inside Docker) can use `lld` to pull out dynamic libraries, but [you've also pointed out](https://forgejo.ellis.link/continuwuation/continuwuity/pulls/924#issuecomment-17966) that we can't just use `lld` to pull out dynamic libraries on cross-compiled binaries. I'm currently testing switching from `gcc` to `clang` so that `lld` can support the cross-compilation environment and potentially replicate how this was working in Docker... if that doesn't work then 🤷‍♂️
Owner

This is done in the docker image using lddtree/goblin, which does not need to execute the files and so can work on cross compiles.

This is done [in the docker image](https://forgejo.ellis.link/continuwuation/continuwuity/src/commit/583cb924f1510960702f626c7397cb9e87e1722a/docker/Dockerfile#L207) using [lddtree](https://lib.rs/crates/lddtree)/goblin, which does not need to execute the files and so can work on cross compiles.
Owner

Oh and you're gonna need to switch to clang anyway for the cross-language LTO to work

Oh and you're gonna need to switch to clang anyway for the cross-language LTO to work
tcpipuk force-pushed tom/fixes from b9b33e3db7
Some checks failed
Checks / Build / Publish / Pre-commit & Formatting (push) Successful in 47s
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 37s
Checks / Build / Publish / Clippy and Cargo Tests (push) Successful in 5m11s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 37s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 5m43s
Checks / Build / Publish / Build linux/arm64 (push) Failing after 7m41s
Checks / Build / Publish / Build linux/amd64 (push) Failing after 6m49s
Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 2m27s
Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 3m0s
Checks / Build / Publish / Publish Multi-Platform (push) Failing after 0s
Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s
to b481ff31c0
Some checks failed
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 31s
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 36s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m21s
Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 3m41s
Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 4m12s
Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s
2025-08-15 21:54:53 +00:00
Compare
Jade changed title from ci: Tidy CI pipeline for readability and performance to WIP: ci: Tidy CI pipeline for readability and performance 2025-08-15 22:08:40 +00:00
Owner

Marking this WIP till it produces working images

Marking this WIP till it produces working images
tcpipuk closed this pull request 2025-08-17 16:39:18 +00:00
Some checks failed
Checks / Build / Publish / Pre-commit & Formatting (pull_request) Successful in 31s
Required
Details
Deploy / Documentation / Build and Deploy Documentation (pull_request) Successful in 36s
Checks / Build / Publish / Clippy and Cargo Tests (pull_request) Successful in 3m21s
Required
Details
Checks / Build / Publish / Build linux/amd64 (pull_request) Failing after 3m41s
Required
Details
Checks / Build / Publish / Build linux/arm64 (pull_request) Failing after 4m12s
Required
Details
Checks / Build / Publish / Publish Multi-Platform (pull_request) Failing after 0s
Required
Details

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
continuwuation/continuwuity!924
No description provided.