Fatal error: manager->state == AWS_HCMST_READY

[tdelaney-leadiro]

Describe the bug

Intermittent fatal error connection_manager.c:1318: manager->state == AWS_HCMST_READY which crashes the JVM. Only seems to occur when running unit tests using localstack which performs many async SQS requests very quickly (response time usually <1ms). The tests also create and destroy a lot of clients.

I think it first started happening when we upgraded to SDK version 2.41.19 (which coincided with upgrading to Java 25).

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No crash in AWS CRT.

Current Behavior

Mac OS

Fatal error condition occurred in /Users/ec2-user/.jenkins/workspace/aws-crt-java-osx-arm64/aws-crt-java/crt/aws-c-http/source/connection_manager.c:1318: manager->state == AWS_HCMST_READY
Exiting Application
################################################################################
Stack trace:
################################################################################
1   AWSCRT_5045658165822875893libaws-cr 0x000000010669d40c aws_fatal_assert + 80
2   AWSCRT_5045658165822875893libaws-cr 0x0000000106488a3c aws_http_connection_manager_release_connection + 0
3   ???                                 0x000000011833df30 0x0 + 4701019952
4   ???                                 0x0000000117d14a90 0x0 + 4694559376

(with varying numbers of ??? lines).

Linux

[2026-09-17T03:49:07.650Z] Fatal error condition occurred in /codebuild/output/src2730530307/src/aws-crt-java/crt/aws-c-http/source/connection_manager.c:1318: manager->state == AWS_HCMST_READY
[2026-09-17T03:49:07.650Z] Exiting Application
[2026-09-17T03:49:07.650Z] ################################################################################
[2026-09-17T03:49:07.650Z] Stack trace:
[2026-09-17T03:49:07.650Z] ################################################################################
[2026-09-17T03:49:07.650Z] /tmp/AWSCRT_16907766413893129720libaws-crt-jni.so(+0x3b6054) [0x7faece8c5054]
[2026-09-17T03:49:07.650Z] /tmp/AWSCRT_16907766413893129720libaws-crt-jni.so(+0xff1c8) [0x7faece60e1c8]
[2026-09-17T03:49:07.650Z] [0x7faf3469db0f]
[2026-09-17T03:49:08.206Z] Aborted (core dumped)

Reproduction Steps

The following code frequently triggers the issue (JUnit5 - standalone application much less likely to trigger). Test runs sometimes hits connection exhaustion.

package awscrt;

import java.net.URI;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.*;
import org.opentest4j.TestAbortedException;
import software.amazon.awssdk.crt.http.HttpException;
import software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sqs.SqsAsyncClient;
import software.amazon.awssdk.services.sqs.model.*;

class AwsCrtIssue1013Test {

    private static final String REGION = "us-west-2";
    private static final String LOCALSTACK_URL = "http://localhost:4566";
    private static final AtomicInteger NEXT_QUEUE_ID = new AtomicInteger(0);

    @Disabled("https://github.com/awslabs/aws-crt-java/issues/1013")
    @SuppressWarnings("java:S2699")
    @RepeatedTest(100)
    void testStressAwsCrt() throws Throwable {
        try {
            stressAwsCrt();
        } catch (ExecutionException x) {
            throwOrAbort(x);
        }
    }

    void stressAwsCrt() throws Exception {
        var builder = SqsAsyncClient.builder()
            .region(Region.of(REGION))
            .endpointOverride(URI.create(LOCALSTACK_URL))
            .httpClientBuilder(AwsCrtAsyncHttpClient.builder());
        var testRunId = NEXT_QUEUE_ID.incrementAndGet();
        var queueCreateRequest = CreateQueueRequest.builder().queueName("awscrt-test-" + testRunId);
        var threadFactory = Thread.ofVirtual()
            .name("sqs-" + testRunId + ":", 1)
            .factory();

        try (var executor = Executors.newThreadPerTaskExecutor(threadFactory); var client = builder.build()) {
            String queueUrl = null;

            try {
                var response = client.createQueue(queueCreateRequest.build()).get();
                queueUrl = response.queueUrl();
                var receive = ReceiveMessageRequest.builder()
                    .queueUrl(queueUrl)
                    .maxNumberOfMessages(10)
                    .waitTimeSeconds(0)
                    .build();

                // Attempting to clean up in any way seems to reduce the chance of the issue occurring.
                for (int i = 0; i < 1000; ++i) {
                    executor.execute(() -> client.receiveMessage(receive));
                }
            } finally {
                if (queueUrl != null) {
                    client.deleteQueue(DeleteQueueRequest.builder().queueUrl(queueUrl).build()).get();
                }

                executor.shutdownNow();
            }
        }
    }

    private static void throwOrAbort(ExecutionException x) throws Throwable {
        var cx = x.getCause();
        assert cx != null;
        var ccx = cx;

        while (ccx != null) {
            if (ccx instanceof HttpException) {
                throw new TestAbortedException(cx.getMessage(), cx);
            }

            ccx = ccx.getCause();
        }

        throw cx;
    }
}

Possible Solution

No response

Additional Information/Context

No response

aws-crt-java version used

0.48.4, likely at least as far back as 0.40.3 or earlier

Java version used

25.0.4 (OpenJDK, Amazon Corretto)

Operating System and version

MacOS Tahoe 26.7, Ubuntu 20.04.6 LTS

[TingDaoK]

if you can provide some reproducible code (even if it's rare like once in couple thousands) that will help a lot to better identify the issue.
Or a bit more descriptive on how you use java-sdk (code snaps) that triggers this issue?

[tdelaney-leadiro]

I'll try to come up with something, earliest would be next week. Unfortunately it's a rare case that occurs somewhere in a set of spring-boot unit tests, with multiple clients (some active, some not), where some of those test cases are for exceptional conditions (API timeouts, etc) so reducing it down has proven difficult.

[tdelaney-leadiro]

See updates to first post.