Sporadic SIGSEGV in macOS [scans] workflow in CI

[mbutrovich]

I've seen this a few different times now in the macos [scans] workflow in CI. Also reported here: #5014 (comment).

Looks like:

ParquetReadFromFakeHadoopFsSuite:
- native scan on fake fs (530 milliseconds)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=22862, tid=59915
#
# JRE version: OpenJDK Runtime Environment Zulu17.68+17-CA (17.0.20+8) (build 17.0.20+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.68+17-CA (17.0.20+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/runner/work/datafusion-comet/datafusion-comet/spark/hs_err_pid22862.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
ParquetTimestampLtzAsNtzSuite:

Recent failure: https://github.com/apache/datafusion-comet/actions/runs/30050199441/job/89355580948?pr=5014

[peterxcli]

Issue #5023 is an upstream libhdfs JNIEnv use-after-free, not a Parquet failure or a regression from #5014.

Evidence

• I downloaded the failing job’s hs_err_pid22862.log and exact native-lib-macos artifact, then symbolized it:
  • pc = 0
  • lr = hdfsThreadDestructor + 80
  • x8 = 0
• Disassembly shows hdfsThreadDestructor loads the JNI GetJavaVM function pointer into x8, then executes blr x8. The stale JNIEnv contains a null function entry, producing the jump to address zero.
• The vulnerable call is exactly (*env)->GetJavaVM. Its null checks cannot detect a freed-but-non-null JNIEnv.
• This precisely matches Apache HDFS-16021: heap-use-after-free in hdfsThreadDestructorand the earlier HDFS-14488. HDFS-16021 remains open without a fix version, and even current Hadoop trunk still contains the vulnerable destructor implementation.

The Comet path is:

fake:// → libhdfs scheme → OpenDAL HDFS → hdrs → hdfs-sys 0.3.0 → vendored libhdfs

The test explicitly registers fake as a libhdfs scheme in ParquetReadFromFakeHadoopFsSuite.scala. Native routing occurs in is_hdfs_scheme and create_hdfs_object_store. macOS explicitly enables vendored hdrs in Cargo.toml, while Cargo.lock pins hdfs-sys 0.3.0.

Recommended fix

Carry or upstream the HDFS-16021 patch: record whether libhdfs itself attached the thread, and only call GetJavaVM/DetachCurrentThread when libhdfs owns that attachment. No released hdfs-sys upgrade currently includes it.

Assist by LLM

[comphead]

Adding a Linux/x86-64/JDK 11 occurrence. This issue is titled and evidenced as macOS, but the same crash reproduces on ubuntu-24.04, which I think widens the scope rather than adding a duplicate.

Run: 34404348583, job PR Build (Linux) / Spark 3.4, JDK 11, Scala 2.12 [scans], on an unrelated CI-config PR (#5782).

Same signature

SIGSEGV (0xb) at pc=0x0000000000000000, pid=1317, tid=4739
JRE: Zulu11.90+19-CA (11.0.32+9), linux-amd64   (vs Zulu17 / bsd-aarch64 above)
Problematic frame:  C  0x0000000000000000
Current thread is native thread
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Consistent with @peterxcli's lr = hdfsThreadDestructor+80, x8 = 0 on aarch64:

• RIP = 0x0, si_addr = 0x0 — jump to address zero.
• RDI == R15 == 0x00007f6d4003c350. On the SysV x86-64 ABI RDI is the first argument, which is the JNIEnv* position for (*env)->GetJavaVM(env, ...).
• Stack used is ~4.8 KB of a 4 MB thread stack, i.e. a handful of frames from thread entry. That is what a pthread TLS destructor at thread exit looks like.
• The return address on top of stack, 0x00007f6ccbc978df, and a deeper entry, 0x00007f6cd0b6f47e, both resolve into the r-xp text segment of /spark/target/tmp/libcomet-4125068328974464746.so (base 0x7f6cca000000, so file offsets 0x1c978df and 0x6b6f47e).

One caveat: I could not symbolize those offsets, so the hdfsThreadDestructor attribution here is by signature and register shape, not by symbol. The dump is also missing its thread list entirely, because error reporting itself SIGSEGV'd twice inside libjvm.so:

[error occurred during error reporting (printing register info), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238]
[error occurred during error reporting (inspecting top of stack), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238]
[thread 17129 also had an error]

Not resource exhaustion: G1 heap was 727 MB used of 1432 MB committed against -Xmx4g, host had 10.1 GB MemAvailable and completely untouched swap.

Why it is not macOS-specific

native/core/Cargo.toml:

default = ["hdfs-opendal"]
hdfs-opendal = ["opendal", "object_store_opendal", "hdfs-sys"]

hdfs-sys (with hdfs_3_3, vendoring the libhdfs C that contains the destructor) is a default feature on every platform. Only hdrs = { features = ["vendored"] } is macOS-gated. So every default libcomet build carries the vulnerable hdfsThreadDestructor, and ParquetReadFromFakeHadoopFsSuite is in the [scans] bucket of both pr_build_linux.yml and pr_build_macos.yml.

The destructor fires in a later suite than the one that arms it

This may be the more actionable part. In the macOS reports the crash lands right at the ParquetReadFromFakeHadoopFsSuite to ParquetTimestampLtzAsNtzSuite boundary. Here it landed 4m03s into the fork, during CometIcebergNativeSuite, long after the fake-fs suite finished.

The hs_err command line confirms both suites were in the same JVM, and SparkTestSuite.txt shows the order:

CometParquetWriterSuite: → ParquetReadV1Suite: → ParquetReadFromFakeHadoopFsSuite:
→ ParquetTimestampLtzAsNtzSuite: → CometNativeReaderSuite: → CometIcebergNativeSuite:   ← crash here

Last test to complete was runtime filtering - join with dynamic partition pruning.

So the failure is not bounded to the suite that registers the fake libhdfs scheme. That suite attaches libhdfs to whatever pooled threads it touches, and the TLS destructor only runs when one of those threads eventually exits, which can be minutes later in an unrelated suite. Any suite sharing the fork is a candidate crash site, which is probably why this reads as a random [scans] flake rather than something pinned to one test.

Incidental, and I do not think it is causal, but recording it: the final 9 seconds before the crash show 1269 CodecPool: Got brand-new compressor [.zstd] at ~100/s on a single executor task, meaning compressors are never being returned to the pool. Possibly a separate minor leak.

Repro artifacts

The run's crash-logs-*, unit-tests-* and java-test-reports-* artifacts for that job carry hs_err_pid1317.log, unit-tests.log and the surefire output, and native-lib-linux from the same run is the exact binary that crashed, so the offsets above should symbolize directly:

gh run download 34404348583 --repo apache/datafusion-comet -p '*3.4*scans*'
gh run download 34404348583 --repo apache/datafusion-comet -n native-lib-linux
addr2line -f -C -e libcomet.so 0x1c978df 0x6b6f47e

Nothing here changes the recommended fix; it just argues the title should lose "macOS" and that the priority is a bit higher than a single-platform flake.

Analysis assisted by LLM.