todo: zap nix out of the build process

nex commented

2025-04-14 20:30:19 +00:00

Owner

Makes things simpler. Jade is apparently knowledgeable in a solution for this.

nex added the

old/ci/cd

Enhancement

Meta/Packaging

labels 2025-04-14 20:30:19 +00:00

Jade was assigned by nex

2025-04-14 20:30:19 +00:00

Jade commented

2025-04-14 20:37:36 +00:00

Owner

My current dockerfile, for reference:

ARG RUST_VERSION=1

FROM --platform=$BUILDPLATFORM docker.io/tonistiigi/xx AS xx
FROM --platform=$BUILDPLATFORM rust:${RUST_VERSION}-slim-bookworm AS base
FROM --platform=$BUILDPLATFORM rust:${RUST_VERSION}-slim-bookworm AS toolchain

# Prevent deletion of apt cache
RUN rm -f /etc/apt/apt.conf.d/docker-clean

# Match Rustc version as close as possible
# rustc -vV
ARG LLVM_VERSION=19
# ENV RUSTUP_TOOLCHAIN=${RUST_VERSION}

# Install repo tools
# Line one: compiler tools
# Line two: curl, for downloading binaries
# Line three: for xx-verify
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get update && apt-get install -y \
    clang-${LLVM_VERSION} lld-${LLVM_VERSION} pkg-config make jq \
    curl git \
    file

# Create symlinks for LLVM tools
RUN <<EOF
    # clang
    ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang
    ln -s "/usr/bin/clang++-${LLVM_VERSION}" "/usr/bin/clang++"
    # lld
    ln -s /usr/bin/ld64.lld-${LLVM_VERSION} /usr/bin/ld64.lld
    ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld
    ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld
    ln -s /usr/bin/lld-link-${LLVM_VERSION} /usr/bin/lld-link
    ln -s /usr/bin/wasm-ld-${LLVM_VERSION} /usr/bin/wasm-ld
EOF

# Developer tool versions
# renovate: datasource=github-releases depName=cargo-bins/cargo-binstall
ENV BINSTALL_VERSION=1.12.3
# renovate: datasource=github-releases depName=psastras/sbom-rs
ENV CARGO_SBOM_VERSION=0.9.1
# renovate: datasource=crate depName=lddtree
ENV LDDTREE_VERSION=0.3.7

# Install unpackaged tools
RUN <<EOF
    curl --retry 5 -L --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash
    cargo binstall --no-confirm cargo-sbom --version $CARGO_SBOM_VERSION
    cargo binstall --no-confirm lddtree --version $LDDTREE_VERSION
EOF

# Set up xx (cross-compilation scripts)
COPY --from=xx / /
ARG TARGETPLATFORM

# Install libraries linked by the binary
# xx-* are xx-specific meta-packages
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
xx-apt-get install -y \
    xx-c-essentials xx-cxx-essentials pkg-config \
    liburing-dev

# Set up Rust toolchain
WORKDIR /app
COPY ./rust-toolchain.toml .
RUN rustc --version \
    && rustup target add $(xx-cargo --print-target-triple)

# Build binary
# We disable incremental compilation to save disk space, as it only produces a minimal speedup for this case.
RUN echo "CARGO_INCREMENTAL=0" >> /etc/environment

# Configure pkg-config
RUN <<EOF
    echo "PKG_CONFIG_LIBDIR=/usr/lib/$(xx-info)/pkgconfig" >> /etc/environment
    echo "PKG_CONFIG=/usr/bin/$(xx-info)-pkg-config" >> /etc/environment
    echo "PKG_CONFIG_ALLOW_CROSS=true" >> /etc/environment
EOF

# Configure cc to use clang version
RUN <<EOF
    echo "CC=clang" >> /etc/environment
    echo "CXX=clang++" >> /etc/environment
EOF

# Cross-language LTO
RUN <<EOF
    echo "CFLAGS=-flto" >> /etc/environment
    echo "CXXFLAGS=-flto" >> /etc/environment
    # Linker is set to target-compatible clang by xx
    echo "RUSTFLAGS='-Clinker-plugin-lto -Clink-arg=-fuse-ld=lld'" >> /etc/environment
EOF

# Apply CPU-specific optimizations if TARGET_CPU is provided
ARG TARGET_CPU=
RUN <<EOF
  set -o allexport
  . /etc/environment
  if [ -n "${TARGET_CPU}" ]; then
    echo "CFLAGS='${CFLAGS} -march=${TARGET_CPU}'" >> /etc/environment
    echo "CXXFLAGS='${CXXFLAGS} -march=${TARGET_CPU}'" >> /etc/environment
    echo "RUSTFLAGS='${RUSTFLAGS} -C target-cpu=${TARGET_CPU}'" >> /etc/environment
  fi
EOF

# Prepare output directories
RUN mkdir /out

FROM toolchain AS builder

# Conduwuit version info
ARG COMMIT_SHA=
ARG CONDUWUIT_VERSION_EXTRA=
ENV CONDUWUIT_VERSION_EXTRA=$CONDUWUIT_VERSION_EXTRA
RUN <<EOF
if [ -z "${CONDUWUIT_VERSION_EXTRA}" ]; then
    echo "CONDUWUIT_VERSION_EXTRA='$(set -e; git rev-parse --short ${COMMIT_SHA:-HEAD} || echo unknown) Jade Build'" >> /etc/environment
fi
EOF

ARG TARGETPLATFORM

# Verify environment configuration
RUN cat /etc/environment
RUN xx-cargo --print-target-triple

# Get source
COPY . .

# Build the binary
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git/db \
    --mount=type=cache,target=/app/target \
    bash <<'EOF'
    set -o allexport
    . /etc/environment
    TARGET_DIR=($(cargo metadata --no-deps --format-version 1 | \
            jq -r ".target_directory"))
    mkdir /out/sbin
    PACKAGE=conduwuit
    xx-cargo build --locked --release \
        -p $PACKAGE;
    BINARIES=($(cargo metadata --no-deps --format-version 1 | \
        jq -r ".packages[] | select(.name == \"$PACKAGE\") | .targets[] | select( .kind | map(. == \"bin\") | any ) | .name"))
    for BINARY in "${BINARIES[@]}"; do
        echo $BINARY
        xx-verify $TARGET_DIR/$(xx-cargo   --print-target-triple)/release/$BINARY
        cp $TARGET_DIR/$(xx-cargo --print-target-triple)/release/$BINARY /out/sbin/$BINARY
    done
EOF

# Generate Software Bill of Materials (SBOM)
RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git/db \
    bash <<'EOF'
    mkdir /out/sbom
    typeset -A PACKAGES
    for BINARY in /out/sbin/*; do
        BINARY_BASE=$(basename ${BINARY})
        package=$(cargo metadata --no-deps --format-version 1 | jq -r ".packages[] | select(.targets[] | select( .kind | map(. == \"bin\") | any ) | .name == \"$BINARY_BASE\") | .name")
        if [ -z "$package" ]; then
            continue
        fi
        PACKAGES[$package]=1
    done
    for PACKAGE in $(echo ${!PACKAGES[@]}); do
        echo $PACKAGE
        cargo sbom --cargo-package $PACKAGE > /out/sbom/$PACKAGE.spdx.json
    done
EOF

# Extract dynamically linked dependencies
RUN <<EOF
    mkdir /out/libs
    mkdir /out/libs-root
    for BINARY in /out/sbin/*; do
        lddtree "$BINARY" | awk '{print $(NF-0) " " $1}' | sort -u -k 1,1 | awk '{print "install", "-D", $1, (($2 ~ /^\//) ? "/out/libs-root" $2 : "/out/libs/" $2)}' | xargs -I {} sh -c {}
    done
EOF

FROM scratch

WORKDIR /

# Copy root certs for tls into image
# You can also mount the certs from the host
# --volume /etc/ssl/certs:/etc/ssl/certs:ro
COPY --from=base /etc/ssl/certs /etc/ssl/certs

# Copy our build
COPY --from=builder /out/sbin/ /sbin/
# Copy SBOM
COPY --from=builder /out/sbom/ /sbom/

# Copy dynamic libraries to root
COPY --from=builder /out/libs-root/ /
COPY --from=builder /out/libs/ /usr/lib/

# Inform linker where to find libraries
ENV LD_LIBRARY_PATH=/usr/lib

CMD ["/sbin/conduwuit"]

Possibly could do with a couple enhancements, but is fully functional and running on matrix.ellis.link right now.

Statically linked binaries for standalone distribution require a bit more thought.

Packaging for RPMs & debs is also an option

My current dockerfile, for reference: ```Dockerfile ARG RUST_VERSION=1 FROM --platform=$BUILDPLATFORM docker.io/tonistiigi/xx AS xx FROM --platform=$BUILDPLATFORM rust:${RUST_VERSION}-slim-bookworm AS base FROM --platform=$BUILDPLATFORM rust:${RUST_VERSION}-slim-bookworm AS toolchain # Prevent deletion of apt cache RUN rm -f /etc/apt/apt.conf.d/docker-clean # Match Rustc version as close as possible # rustc -vV ARG LLVM_VERSION=19 # ENV RUSTUP_TOOLCHAIN=${RUST_VERSION} # Install repo tools # Line one: compiler tools # Line two: curl, for downloading binaries # Line three: for xx-verify RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y \ clang-${LLVM_VERSION} lld-${LLVM_VERSION} pkg-config make jq \ curl git \ file # Create symlinks for LLVM tools RUN <<EOF # clang ln -s /usr/bin/clang-${LLVM_VERSION} /usr/bin/clang ln -s "/usr/bin/clang++-${LLVM_VERSION}" "/usr/bin/clang++" # lld ln -s /usr/bin/ld64.lld-${LLVM_VERSION} /usr/bin/ld64.lld ln -s /usr/bin/ld.lld-${LLVM_VERSION} /usr/bin/ld.lld ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/lld ln -s /usr/bin/lld-link-${LLVM_VERSION} /usr/bin/lld-link ln -s /usr/bin/wasm-ld-${LLVM_VERSION} /usr/bin/wasm-ld EOF # Developer tool versions # renovate: datasource=github-releases depName=cargo-bins/cargo-binstall ENV BINSTALL_VERSION=1.12.3 # renovate: datasource=github-releases depName=psastras/sbom-rs ENV CARGO_SBOM_VERSION=0.9.1 # renovate: datasource=crate depName=lddtree ENV LDDTREE_VERSION=0.3.7 # Install unpackaged tools RUN <<EOF curl --retry 5 -L --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash cargo binstall --no-confirm cargo-sbom --version $CARGO_SBOM_VERSION cargo binstall --no-confirm lddtree --version $LDDTREE_VERSION EOF # Set up xx (cross-compilation scripts) COPY --from=xx / / ARG TARGETPLATFORM # Install libraries linked by the binary # xx-* are xx-specific meta-packages RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ --mount=type=cache,target=/var/lib/apt,sharing=locked \ xx-apt-get install -y \ xx-c-essentials xx-cxx-essentials pkg-config \ liburing-dev # Set up Rust toolchain WORKDIR /app COPY ./rust-toolchain.toml . RUN rustc --version \ && rustup target add $(xx-cargo --print-target-triple) # Build binary # We disable incremental compilation to save disk space, as it only produces a minimal speedup for this case. RUN echo "CARGO_INCREMENTAL=0" >> /etc/environment # Configure pkg-config RUN <<EOF echo "PKG_CONFIG_LIBDIR=/usr/lib/$(xx-info)/pkgconfig" >> /etc/environment echo "PKG_CONFIG=/usr/bin/$(xx-info)-pkg-config" >> /etc/environment echo "PKG_CONFIG_ALLOW_CROSS=true" >> /etc/environment EOF # Configure cc to use clang version RUN <<EOF echo "CC=clang" >> /etc/environment echo "CXX=clang++" >> /etc/environment EOF # Cross-language LTO RUN <<EOF echo "CFLAGS=-flto" >> /etc/environment echo "CXXFLAGS=-flto" >> /etc/environment # Linker is set to target-compatible clang by xx echo "RUSTFLAGS='-Clinker-plugin-lto -Clink-arg=-fuse-ld=lld'" >> /etc/environment EOF # Apply CPU-specific optimizations if TARGET_CPU is provided ARG TARGET_CPU= RUN <<EOF set -o allexport . /etc/environment if [ -n "${TARGET_CPU}" ]; then echo "CFLAGS='${CFLAGS} -march=${TARGET_CPU}'" >> /etc/environment echo "CXXFLAGS='${CXXFLAGS} -march=${TARGET_CPU}'" >> /etc/environment echo "RUSTFLAGS='${RUSTFLAGS} -C target-cpu=${TARGET_CPU}'" >> /etc/environment fi EOF # Prepare output directories RUN mkdir /out FROM toolchain AS builder # Conduwuit version info ARG COMMIT_SHA= ARG CONDUWUIT_VERSION_EXTRA= ENV CONDUWUIT_VERSION_EXTRA=$CONDUWUIT_VERSION_EXTRA RUN <<EOF if [ -z "${CONDUWUIT_VERSION_EXTRA}" ]; then echo "CONDUWUIT_VERSION_EXTRA='$(set -e; git rev-parse --short ${COMMIT_SHA:-HEAD} || echo unknown) Jade Build'" >> /etc/environment fi EOF ARG TARGETPLATFORM # Verify environment configuration RUN cat /etc/environment RUN xx-cargo --print-target-triple # Get source COPY . . # Build the binary RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git/db \ --mount=type=cache,target=/app/target \ bash <<'EOF' set -o allexport . /etc/environment TARGET_DIR=($(cargo metadata --no-deps --format-version 1 | \ jq -r ".target_directory")) mkdir /out/sbin PACKAGE=conduwuit xx-cargo build --locked --release \ -p $PACKAGE; BINARIES=($(cargo metadata --no-deps --format-version 1 | \ jq -r ".packages[] | select(.name == \"$PACKAGE\") | .targets[] | select( .kind | map(. == \"bin\") | any ) | .name")) for BINARY in "${BINARIES[@]}"; do echo $BINARY xx-verify $TARGET_DIR/$(xx-cargo --print-target-triple)/release/$BINARY cp $TARGET_DIR/$(xx-cargo --print-target-triple)/release/$BINARY /out/sbin/$BINARY done EOF # Generate Software Bill of Materials (SBOM) RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/usr/local/cargo/git/db \ bash <<'EOF' mkdir /out/sbom typeset -A PACKAGES for BINARY in /out/sbin/*; do BINARY_BASE=$(basename ${BINARY}) package=$(cargo metadata --no-deps --format-version 1 | jq -r ".packages[] | select(.targets[] | select( .kind | map(. == \"bin\") | any ) | .name == \"$BINARY_BASE\") | .name") if [ -z "$package" ]; then continue fi PACKAGES[$package]=1 done for PACKAGE in $(echo ${!PACKAGES[@]}); do echo $PACKAGE cargo sbom --cargo-package $PACKAGE > /out/sbom/$PACKAGE.spdx.json done EOF # Extract dynamically linked dependencies RUN <<EOF mkdir /out/libs mkdir /out/libs-root for BINARY in /out/sbin/*; do lddtree "$BINARY" | awk '{print $(NF-0) " " $1}' | sort -u -k 1,1 | awk '{print "install", "-D", $1, (($2 ~ /^\//) ? "/out/libs-root" $2 : "/out/libs/" $2)}' | xargs -I {} sh -c {} done EOF FROM scratch WORKDIR / # Copy root certs for tls into image # You can also mount the certs from the host # --volume /etc/ssl/certs:/etc/ssl/certs:ro COPY --from=base /etc/ssl/certs /etc/ssl/certs # Copy our build COPY --from=builder /out/sbin/ /sbin/ # Copy SBOM COPY --from=builder /out/sbom/ /sbom/ # Copy dynamic libraries to root COPY --from=builder /out/libs-root/ / COPY --from=builder /out/libs/ /usr/lib/ # Inform linker where to find libraries ENV LD_LIBRARY_PATH=/usr/lib CMD ["/sbin/conduwuit"] ``` Possibly could do with a couple enhancements, but is fully functional and running on matrix.ellis.link right now. Statically linked binaries for standalone distribution require a bit more thought. Packaging for RPMs & debs is also an option

~~Jade referenced this issue 2025-04-14 20:59:05 +00:00~~

meta(bug): Packaging meta issue #747

nex commented

2025-04-14 23:44:16 +00:00

Author

Owner

RPMs, debs, and standalone binaries are definitely something we'll need. Probably a PKGBUILD for arch, whatever it is nix uses this week, but RPMs/Debs/Standalone covers most of the prod server systems

nex commented

2025-04-15 15:12:23 +00:00

Author

Owner

Noting that it's been mentioned that nix is very helpful for static & cross-platform builds

Jade commented

2025-04-15 15:21:44 +00:00

Owner

At the moment we only support linux. We currently can't build for windows afaict, and don't support running on MacOS. We at least can do cross-architecture builds, though.
@Aranjedeath builds & runs on FreeBSD, and has mentioned that he can't use nix for that due to some of the native deps (it was either jemalloc or rocksdb)

At the moment we only support linux. We currently can't build for windows afaict, and don't support running on MacOS. We at least can do cross-architecture builds, though. @Aranjedeath builds & runs on FreeBSD, and has mentioned that he can't use nix for that due to some of the native deps (it was either jemalloc or rocksdb)

Aranjedeath commented

2025-04-15 15:55:30 +00:00

Owner

I don't use and have never touched nix in my life; I solely use cargo build. If we put this into FreeBSD ports at some point, that is what it would use there.

👍 2

nex commented

2025-04-21 15:27:27 +00:00

Author

Owner

I'll say, nix might be unwieldy, but it definitely helped with creating the static builds for the 0.5.0rc5 release, once I'd gotten over the bumps

Jade commented

2025-04-21 15:30:31 +00:00

Owner

Yeah, the Nix setup here includes a lot of automation for building binaries and installing relevant dependencies that would need to be replicated

June commented

2025-04-21 19:43:53 +00:00

Nix does not support *BSDs

nex commented

2025-04-21 19:55:29 +00:00

Author

Owner

We could have a separate CI flow for building BSD-specific binaries - I know the ones that we currently generate are also a bit of a pain on BSD sometimes

June commented

2025-04-21 21:07:14 +00:00

You will most definitely need to use emulation/virtualisation like
https://github.com/vmactions/freebsd-vm and it will be very slow. This
is what conduwuit planned to try using.

~ June 🏳️‍⚧️

You will most definitely need to use emulation/virtualisation like https://github.com/vmactions/freebsd-vm and it will be very slow. This is what conduwuit planned to try using. ~ June 🏳️‍⚧️

Jade commented

2025-04-22 00:14:42 +00:00

Owner

Ideally, we wouldn't have to do that, but the issue is likely to be linked in system libraries. If we can get around that or bodge it somehow, we could get cross-compiles working. It's likely a similar story to what I did to get cross-arch Linux builds working in Docker, though.

Jade commented

2025-04-22 00:16:28 +00:00

Owner

That image is already using clang for building C deps, which supports cross-compilation to freebsd x86_64-pc-freebsd<ver>. It could be a matter of disabling some deps and vendoring in the rest.

That image is already using clang for building C deps, which supports cross-compilation to freebsd `x86_64-pc-freebsd<ver>`. It could be a matter of disabling some deps and vendoring in the rest.

Rows
Columns

todo: zap nix out of the build process #740