Conversation
|
As mentioned in #1215 before considering adding these targets I would like to see benchmarks which should that there is a benefit to building this microarchitecture specific distributions. Additional the change to bookworm likely raises the minimum glibc version required to consume these distributions from 2.24 (Debian 9 Stretch) to 2.36 (Debian 12 Bookworm). A better approach that is more maintainable in the long term to to use the LLVM toolchain to cross-compile these targets using a sysroot. This was mentioned in #1215. |
a61a7be to
cbcc593
Compare
Hardware validation + apples-to-apples benchmark — Python 3.13.15 on real P9/P10/P11 hardwareAll three binaries were cross-compiled from x86 (commit cbcc593) and tested on RHEL 9.8 machines inside Smoke tests — ALL PASSEDAll three binaries passed: Python version check, Performance benchmark — generic vs tuned, same machineTo isolate the tuning benefit from hardware differences, the same generic Workload: Pi to 10,000 digits via Python's POWER10
→ 38.7% faster on the same P10 machine with POWER11
→ 38.3% faster on the same P11 machine with glibc requirementPower9 requires glibc ≥ 2.17 (same as the generic baseline). Power10 and Power11 require glibc ≥ 2.36 due to the Bookworm build base. This is enforced in |
| # Stretch stopped publishing snapshots in April 2023. Last snapshot | ||
| # is 20230423T032533Z. But there are package authentication issues | ||
| # with this snapshot. | ||
| RUN for s in debian_stretch debian_stretch-updates debian-security_stretch/updates; do \ | ||
| echo "deb http://snapshot.debian.org/archive/${s%_*}/20221105T150728Z/ ${s#*_} main"; \ | ||
| done > /etc/apt/sources.list && \ |
There was a problem hiding this comment.
The snapshot release should not be removed. We want reproducible images when possible
|
A good amount of work is done to keep the x86-64 microarchitecture variants based on the same glibc version. The same should be done here. There is limited functionality in uv to select a build based on the host glibc version so supporting older versions is important. I would appreciate more details benchmarks, at minimum using fastmark but if possible pyperformance. These would give a more complete picture. The benchmarks should be based on the same compiler version as these difference can be substantial. If upgrading the compiler (and glibc version) give the bulk of the performance gain there is no reason to expand the build matrix. Is there a reason why cross-compiling using LLVM was not been tried discussed in #1215 and my earlier comment? |
| @@ -0,0 +1,54 @@ | |||
| # Debian Bookworm. | |||
| FROM docker.io/library/debian@sha256:6bc30d909583f38600edd6609e29eb3fb284ab8affce8d0389f332fc91c2dd91 | |||
| LABEL org.opencontainers.image.authors="Gregory Szorc <gregory.szorc@gmail.com>" | |||
There was a problem hiding this comment.
This would need to have a different author
|
@veenious sorry I missed that author fix when I sent over my initial suggestions. lmk if you want me to re-push anything or if your'e fine with fixing my commits. @jjhelmus thanks for the reviews again. for the sake of this PR -- we'd assumed that since the project hadn't gone that route yet we could add these in the way they're already built and continue to discuss in #1100 . No other reason. |
Adds three new ppc64le architecture variant targets, mirroring the existing x86_64_v2/v3/v4 pattern: - ppc64le_power9-unknown-linux-gnu — compiled with -mcpu=power9 -mtune=power9 - ppc64le_power10-unknown-linux-gnu — compiled with -mcpu=power10 -mtune=power10 - ppc64le_power11-unknown-linux-gnu — compiled with -mcpu=power11 -mtune=power11 The existing generic ppc64le-unknown-linux-gnu artifact is untouched. Changes: - ci-targets.yaml: add all three power variant CI targets - cpython-unix/targets.yml: add build definitions with correct compiler paths; power9 reuses build.cross.Dockerfile (Debian Stretch + GCC 6), power10 uses new Bookworm + GCC 12 image, power11 uses new Bookworm + GCC 14 from Trixie image (GCC 12 does not support -mcpu=power11) - cpython-unix/build.cross.Dockerfile: switch apt sources to archive.debian.org over HTTPS (port 80 blocked in build containers) - cpython-unix/build.cross-ppc64le_power10.Dockerfile: new image - cpython-unix/build.cross-ppc64le_power11.Dockerfile: new image - cpython-unix/build.py: strip ppc64le_powerN- variant prefix in triple normalization, matching the x86_64_vN pattern - pythonbuild/downloads.json: fix x.org and sourceforge download URLs that were unreachable (redirect to blocked port 80) - src/release.rs: register all three triples in RELEASE_TRIPLES - src/validation.rs: add RECOGNIZED_TRIPLES, GLIBC_MAX_VERSION, ELF_ALLOWED_LIBRARIES, PLATFORM_TAG, and ELF machine entries Power9 support is based on Christy Norman's work (clnperez/ppc64le-currency). Power11 requires GCC 14 because GCC 12 does not support -mcpu=power11. Tested by cross-compiling on x86 and running on physical POWER9, POWER10, and POWER11 hardware (RHEL 9.8, glibc 2.34, UBI 10 container glibc 2.39). Closes astral-sh#1215
cbcc593 to
739a932
Compare
|
Hi @jjhelmus — updated the PR to address all review feedback. Summary below. Changes from previous revisionApproach: LLVM 23 + Debian Stretch sysroot (cross-compiled on x86) The LLVM cross-compilation approach in this PR follows #1259 (riscv64 LLVM PR) and #1241 (aarch64 Stretch sysroot PR) as the reference patterns.
Shell script changes — each one fixed a specific bug required for LLVM cross-compilation:
All changes are guarded to clang or ppc64le targets and do not affect any existing x86/aarch64/musl builds. glibcAll three new variants now match the existing
pyperformance benchmarksRan pyperformance 1.14.0 (
Selected results:
The gains are from POWER10/11 ISA features (MMA, new vector/crypto instructions) enabled by |
|
Thanks for the update. I'm out this week but will review this when I am back. |
Adds three new ppc64le architecture variant targets, mirroring the existing
x86_64_v2/v3/v4 pattern:
ppc64le_power9-unknown-linux-gnu— compiled with-mcpu=power9 -mtune=power9ppc64le_power10-unknown-linux-gnu— compiled with-mcpu=power10 -mtune=power10ppc64le_power11-unknown-linux-gnu— compiled with-mcpu=power11 -mtune=power11The existing generic
ppc64le-unknown-linux-gnuartifact is untouched.Power9 support is based on Christy Norman's work (clnperez#1).
Power10 uses Debian Bookworm's GCC 12. Power11 uses GCC 14 from Debian Trixie because
GCC 12 does not support
-mcpu=power11.Closes #1215