`RUST_BACKTRACE=full` loop with `-Cpanic=abort` on `aarch64-unknown-linux-gnu`

[cuviper]

When compiled with -Cpanic=abort, a program as simple as fn main() { panic!(); } gets into a loop with RUST_BACKTRACE=full on aarch64-unknown-linux-gnu:

$ echo 'fn main() { panic!() }' | rustc - -Cpanic=abort && RUST_BACKTRACE=full ./rust_out
thread 'main' panicked at <anon>:1:13:
explicit panic
stack backtrace:
   0:     0xaaaad44ee198 - std::backtrace_rs::backtrace::libunwind::trace::hfa92c98074c33c6e
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0xaaaad44ee198 - std::backtrace_rs::backtrace::trace_unsynchronized::h4fca41224fd18d13
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xaaaad44ee198 - std::sys_common::backtrace::_print_fmt::hbb1869018263cc3e
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xaaaad44ee198 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hfa121140e91bdb74
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xaaaad45074b8 - core::fmt::rt::Argument::fmt::h0cc54cf87d70c86b
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/core/src/fmt/rt.rs:142:9
   5:     0xaaaad45074b8 - core::fmt::write::hb74970315812417b
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/core/src/fmt/mod.rs:1153:17
   6:     0xaaaad44ec398 - std::io::Write::write_fmt::h4028ee7f77ba6e46
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/io/mod.rs:1843:15
   7:     0xaaaad44edfe0 - std::sys_common::backtrace::_print::ha41833e087a85e12
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xaaaad44edfe0 - std::sys_common::backtrace::print::h32ae40bc3201cfc9
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xaaaad44ef1e8 - std::panicking::default_hook::{{closure}}::h46d22ee496490b96
  10:     0xaaaad44eee50 - std::panicking::default_hook::h8a05bff9b9ebb786
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/panicking.rs:291:9
  11:     0xaaaad44ef628 - std::panicking::rust_panic_with_hook::h72b4980bd319f2cc
                               at /rustc/8b2459c1f21187f9792d99310171a15e64feb9cf/library/std/src/panicking.rs:788:13
  12:     0xaaaad44d6bb4 - std::panicking::begin_panic::{{closure}}::h6b7a2e6dc54500da
  13:     0xaaaad44d6bb4 - std::panicking::begin_panic::{{closure}}::h6b7a2e6dc54500da
  14:     0xaaaad44d6bb4 - std::panicking::begin_panic::{{closure}}::h6b7a2e6dc54500da
  15:     0xaaaad44d6bb4 - std::panicking::begin_panic::{{closure}}::h6b7a2e6dc54500da
...

With RUST_BACKTRACE=1, it has no backtrace at all:

thread 'main' panicked at <anon>:1:13:
explicit panic
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aborted (core dumped)

Current toolchain:

rustc 1.79.0-nightly (8b2459c1f 2024-04-09)
binary: rustc
commit-hash: 8b2459c1f21187f9792d99310171a15e64feb9cf
commit-date: 2024-04-09
host: aarch64-unknown-linux-gnu
release: 1.79.0-nightly
LLVM version: 18.1.3

Using cargo bisect-rustc, this problem seems to be quite old:

********************************************************************************
Regression in nightly-2022-07-03
********************************************************************************

fetching https://static.rust-lang.org/dist/2022-07-02/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2022-07-02: 40 B / 40 B [============================================================] 100.00 % 229.04 KB/s converted 2022-07-02 to 46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f
fetching https://static.rust-lang.org/dist/2022-07-03/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2022-07-03: 40 B / 40 B [==============================================================] 100.00 % 1.88 MB/s converted 2022-07-03 to f2d93935ffba3ab9d7ccb5300771a2d29b4c8bf3
looking for regression commit between 2022-07-02 and 2022-07-03
fetching (via remote github) commits from max(46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f, 2022-06-30) to f2d93935ffba3ab9d7ccb5300771a2d29b4c8bf3
ending github query because we found starting sha: 46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f
get_commits_between returning commits, len: 8
  commit[0] 2022-07-01: Auto merge of #93967 - cjgillot:short-struct-span, r=petrochenkov
  commit[1] 2022-07-01: Auto merge of #98781 - GuillaumeGomez:rollup-798kb8u, r=GuillaumeGomez
  commit[2] 2022-07-02: Auto merge of #98791 - cuviper:rogue-binary, r=compiler-errors
  commit[3] 2022-07-02: Auto merge of #98802 - Dylan-DPC:rollup-u6mwx27, r=Dylan-DPC
  commit[4] 2022-07-02: Auto merge of #91743 - cjgillot:enable_mir_inlining_inline_all, r=oli-obk
  commit[5] 2022-07-02: Auto merge of #97235 - nbdd0121:unwind, r=Amanieu
  commit[6] 2022-07-02: Auto merge of #97585 - lqd:const-alloc-intern, r=RalfJung
  commit[7] 2022-07-02: Auto merge of #98820 - RalfJung:rollup-i3mip9a, r=RalfJung
ERROR: no CI builds available between 46b8c23f3eb5e4d0e0aa27eb3f20d5b8fc3ed51f and f2d93935ffba3ab9d7ccb5300771a2d29b4c8bf3 within last 167 days

Comparison: 46b8c23...f2d9393

#97235 looks like the most obvious candidate, cc @nbdd0121 @Amanieu

The reason I'm noticing this now is because there's a new cargo test panic_abort_doc_tests from rust-lang/cargo#13388, which I found hanging on Fedora aarch64 in a scratch build of 1.78-beta. After about 2 hours, rustdoc got a SIGKILL, which I assume is an OOM from capturing all that backtrace output. When reproducing this hang, I found that running its RUST_BACKTRACE=full /tmp/rustdoctestGHjZyP/rust_out myself showed the backtrace loop.

---

This is similar to #123686, but that proposed fix only applies to Windows.

[saethlin]

I think this was already reported? #123686

[cuviper]

@saethlin I mentioned that one at the end:

This is similar to #123686, but that proposed fix only applies to Windows.

and AFAICS that report doesn't depend on -Cpanic=abort either.

[cuviper]

It might be a duplicate of #121817, cc @wesleywiser

[nbdd0121]

#97235 adds a lint (which isn't even executed with panic=abort), and should have no codegen changes.

The current behaviour doesn't surprise me since we neither force frame pointers, nor force unwind tables. To generate the stack trace usually what happens is that eh_frame to unwind the frames is used if it exists, otherwise frame pointers are used to continue unwind the frames. We build the standard library with -Cpanic=unwind so the eh_frame is available, but past begin_panic nothing is available anymore.

Adding -Cforce-frame-pointers=yes allows termination (still no traces beyond begin_panic, and adding -Cforce-unwind-tables=yes produces a property stack trace.

[cuviper]

#97235 adds a lint (which isn't even executed with panic=abort), and should have no codegen changes.

OK, I don't know. The bisected nightly works fine when I build it myself, which implies that it's sensitive to the particular configuration, but that makes it hard to try and narrow this further. Or if the bug only arose there because of happenstance code changes, the root cause might not be in that range at all.

[saethlin]

I can reproduce the infinite backtrace with 2020-03-01 and the target is gone in 2020-01-01 so I suspect that this bug has been in the target since we started distributing artifacts for it.

I cannot reproduce the infinite backtrace printing with -Zbuild-std. I suspect the problem has something to do with the fact that the standard library is precompiled with -Cpanic=unwind then we're linking to it with -Cpanic=abort, and I don't know how to imitate that with -Zbuild-std.

The behavior of the MIR inliner is sensitive to the stable crate hash, so if the behavior of the MIR inliner can impact whether the fatal optimization happens, it may not be possible to bisect this.

[saethlin]

The current behaviour doesn't surprise me since we neither force frame pointers, nor force unwind tables. To generate the stack trace usually what happens is that eh_frame to unwind the frames is used if it exists, otherwise frame pointers are used to continue unwind the frames. We build the standard library with -Cpanic=unwind so the eh_frame is available, but past begin_panic nothing is available anymore.

To be clear, the behavior seems wrong to me even using -Zbuild-std:

ubuntu@ip-172-31-13-98:~/demo$ RUSTFLAGS=-Cpanic=abort RUST_BACKTRACE=full cargo +nightly r -Zbuild-std=std,panic_abort --target=aarch64-unknown-linux-gnu
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/aarch64-unknown-linux-gnu/debug/demo`
thread 'main' panicked at src/main.rs:1:13:
explicit panic
stack backtrace:
   0:     0xaaaabefe12a8 - std::backtrace_rs::backtrace::libunwind::trace::h3eb3677c648e214e
                               at /home/ubuntu/.rustup/toolchains/nightly-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0xaaaabefe12a8 - std::backtrace_rs::backtrace::trace_unsynchronized::h6b663bf10ab98a13
                               at /home/ubuntu/.rustup/toolchains/nightly-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
Aborted (core dumped)

Buggy in a different way, but still buggy, so maybe it's not worth splitting hairs.

[Amanieu]

The current behaviour doesn't surprise me since we neither force frame pointers, nor force unwind tables. To generate the stack trace usually what happens is that eh_frame to unwind the frames is used if it exists, otherwise frame pointers are used to continue unwind the frames. We build the standard library with -Cpanic=unwind so the eh_frame is available, but past begin_panic nothing is available anymore.

Shouldn't the unwinder just give up if it reaches a frame with no unwinding information? This seems to be the issue here since the loop start when the unwinder reaches the first frame compiled with panic=abort.

This target uses libgcc as the unwinder, this might be worth looking more deeply into.

A truncated backtrace is normal if everything is compiled with panic=abort since we don't emit eh_frame by default so the unwinder doesn't know how to unwind the stack. What shouldn't be happening is that the unwinder ends up in an infinite loop since that indicates invalid unwind information.

Also note that the call into panic is a non-returning tail call. This has caused issues in the past on the ARM target (#69231) since LLVM would assume the LR value is dead and leave it dangling. If this happens to hold a function pointer to std::panicking::begin_panic then this could lead to the infinite loop we are seeing.