[#1180] Report the fail-over failure of the Grizzly proxy listener tests instead of timing out

[vharseko]

Fixes #1180

Problem

GrizzlyLDAPListenerTestCase#testLDAPListenerProxyDuringHandleAccept timed out on windows-latest / JDK 26 with a bare TimeoutException at the first wait (line 605).

The overridden handleAccept connects to an offline port and then to the online listener, and resolves the proxy's context only after both succeed. It runs synchronously on a selector thread of the server transport. Any exception before the last line therefore left that context unresolved. The causes could be: the offline connection unexpectedly succeeding, a TimeoutResultException (which is not a ConnectionException), a failed online connection, or an unchecked exception such as a failed loopback bind in findFreeSocketAddress(). In each case the listener only closed the client connection, and the test waited out its 10 seconds without a cause. The class runs with logging at SEVERE, so the job log shows nothing either. Every way the fail-over can fail looks the same as the CI failure, so the run cannot tell us which one happened.

testLDAPListenerProxyDuringHandleBind had the same blind spot in a different form. It failed the bind with a plain Result, which the listener cannot encode as a bind response (ClassCastException: ResultImpl cannot be cast to BindResult inside Grizzly). The connection was closed and the client saw only Server Connection Closed, or the test hung until its timeOut.

Fix (test only)

• The overridden handleAccept resolves the proxy's context with the failure before rethrowing it, unchecked exceptions included, so the assert fails at once with the actual cause.
• The first wait is 30 s instead of 10 s. handleAccept makes two nested connections before it resolves the context, and each may take up to the default connect timeout (10 s). A shorter wait would again expire before a failed attempt can report itself. Because a failure is now reported as soon as it happens, the longer window does not hide anything.
• The bind variant fails the bind with a BindResult, for unchecked exceptions too. Only the diagnostic message of that result reaches the client, never its cause, so the message carries the nested cause.
• Both tests share one failOverToOnlineServer helper. It closes the offline and online connection factories (previously leaked). It connects to the numeric address of the offline port: the address of the reserved socket keeps the host name localhost, so neither getHostName() nor getHostString() gives a numeric host.
• The listener addresses (the online server and both proxies) have no host name, so getHostName() reverse-resolved them, for the online server inside handleAccept, outside the connect timeout. They are now used through getHostString().
• The client connection factories are closed in finally. In ...DuringHandleAccept, connection.close() moved into finally, the isClosed wait is bounded, and a @Test(timeOut = 60000) was added. The timeOut of ...DuringHandleBind goes from 10 s to 60 s, for the same reason as the longer wait.

The root cause on Windows is still unknown: it cannot be reproduced locally. This change makes the next occurrence name it. I also tried reserving the offline port with a socket that is bound but not listening, to rule out the port being reused. On macOS such a port does not refuse connections: the connect times out instead. So the offline port is still picked with findFreeSocketAddress().

Verification

Each mutant was run against both tests, before (BASE of the PR) and after:

Mutant	Before	After
offline address = the listening online one	accept: bare TimeoutException after 11.2 s; bind: Server Connection Closed, ClassCastException in the log	both fail in about 1 s with Connection to offline server succeeded unexpectedly
online port = a free port (connect refused)	—	accept fails in 1 s, with Connection refused in the cause chain; bind fails in 0.4 s with Unexpected exception when connecting to online server: ...ConnectionException: Connect Error: Connection refused
RuntimeException at the start of the helper	—	accept fails in 1 s with mutant: loopback bind failed; bind fails in 0.5 s with java.lang.RuntimeException: mutant: loopback bind failed

Against the first commit of this PR, the last two mutants gave: a bind message without the cause (Unexpected exception when connecting to online server); and for the RuntimeException, a bare TimeoutException after 31 s in the accept test and Server Connection Closed in the bind test.

Without a mutant: all of opendj-grizzly passes, 1040 tests in 9 classes with 0 failures (mvn verify, checkstyle included).

[maximthomas]

praise: The change sits exactly where the CI failure lost its cause.

• proxyServerConnection.context.handleException(e) in the handleAccept override (:594) turns the bare TimeoutException of the CI run into the actual fail-over failure.
• handleBind now fails with Responses.newBindResult(ResultCode.OTHER) (:666), which removes the ResultImpl → BindResult ClassCastException that left the bind unanswered.
• failOverToOnlineServer closes offlineFactory and onlineFactory in finally (:229, :235), which BASE leaked.

---

issue (non-blocking): The bind variant still does not get the nested cause to the client.

opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java:666-667

The diagnostic message is only the wrapper text ("Unexpected exception when connecting to online server"). setCause(e) stays on the server side, because LDAPWriter.writeResultHeader encodes the result code, matched DN, diagnostic message and referrals, never the cause, and the class logs at SEVERE. When a nested connect in testLDAPListenerProxyDuringHandleBind fails, the report says which leg failed but not whether it was refused, timed out or unresolved. The "succeeded unexpectedly" arm, the one used in the PR's verification, explains itself.

} catch (final LdapException e) {
    // A bind must fail with a bind result: the listener cannot encode any other
    // result as a bind response, and would then never answer the bind.
    final Throwable cause = e.getCause();
    final String message = cause != null
            ? e.getResult().getDiagnosticMessage() + ": " + cause
            : e.getResult().getDiagnosticMessage();
    resultHandler.handleException(newLdapException(Responses.newBindResult(ResultCode.OTHER)
            .setDiagnosticMessage(message).setCause(e)));
}

---

issue (non-blocking): A RuntimeException from the fail-over helper escapes handleAccept without resolving the context.

opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java:588-596, :213

The override catches only LdapException. findFreeSocketAddress() wraps an IOException in a RuntimeException and is called outside any try in the helper. The factory constructors and the close() calls in its finally blocks are not wrapped either. When one of them throws, proxyServerConnection.context stays unresolved and the test again waits out context.get(30, SECONDS) with a bare TimeoutException. The comment at :591-593 promises a report for any handleAccept failure. The only realistic trigger is a failed loopback port-0 bind.

} catch (final LdapException e) {
    proxyServerConnection.context.handleException(e);
    throw e;
} catch (final RuntimeException e) {
    proxyServerConnection.context.handleException(newLdapException(ResultCode.OTHER, e));
    throw e;
}

---

issue (non-blocking): Switching the offline address to getHostString() changes nothing: the host is still "localhost", not numeric.

opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java:212-215

findFreeSocketAddress() returns ServerSocket.getLocalSocketAddress() of a socket bound to InetAddress.getLoopbackAddress(). That address keeps its preset host name, "localhost". So getHostString() returns the same string BASE's getHostName() did, and neither call ever did a lookup. A probe printed hostString=localhost on JDK 8u362, 11.0.25, 17.0.11, 21.0.2 and 26, and with -Djava.net.preferIPv6Addresses=true too. The offline connect still resolves "localhost" forward. The comment and the PR body ("connects to the numeric loopback address") tell the next reader that road is closed.

// The numeric host avoids a forward lookup of "localhost".
final InetSocketAddress offlineAddress = findFreeSocketAddress();
final LDAPConnectionFactory offlineFactory =
        new LDAPConnectionFactory(offlineAddress.getAddress().getHostAddress(), offlineAddress.getPort());

Or: keep getHostName(), drop the comment, and correct the sentence in the commit body.

---

suggestion (non-blocking): Use getHostString() on the listener addresses. Those are the legs that do a reverse lookup.

opendj-grizzly/src/test/java/org/forgerock/opendj/grizzly/GrizzlyLDAPListenerTestCase.java:221, :610, :679

LDAPListener.firstSocketAddress() is the Grizzly channel's getLocalAddress(), which has no host name (hostString=127.0.0.1 measured with Grizzly 2.3.35 on JDK 8/11/17/26). So onlineAddress.getHostName() reverse-resolves 127.0.0.1 inside handleAccept, before the context resolves. That adds time to the 30 s window, which the new comment budgets as two connect timeouts only. A stalled resolver would bring back the bare timeout. This road predates the PR.

new LDAPConnectionFactory(onlineAddress.getHostString(), onlineAddress.getPort());
// and in both tests
new LDAPConnectionFactory(proxyAddr.getHostString(), proxyAddr.getPort());

[vharseko]

Thanks, all four points hold. Addressed in afbe983.

1. Bind cause. Confirmed: LDAPWriter.writeResultHeader never writes the cause. The diagnostic message now carries it, in your form. With the online port replaced by a free one, the bind test reports Unexpected exception when connecting to online server: org.forgerock.opendj.ldap.ConnectionException: Connect Error: Connection refused (before: only the first part).

2. RuntimeException from the fail-over. Confirmed with a mutant that throws at the start of the helper: the accept test waited out 31 s for a bare TimeoutException. Added your catch (RuntimeException); it now fails in 1 s with the mutant's message. The same mutant showed the bind variant had the same gap: handleBind caught only LdapException, the bind went unanswered and the client saw Server Connection Closed. It now catches RuntimeException too and fails the bind with a BindResult whose message is the exception.

3. Offline host. You are right, and my sentence was wrong: getLocalSocketAddress() keeps localhost, and a local probe printed hostString=localhost hostAddress=127.0.0.1. Took option A: the offline factory connects to getAddress().getHostAddress(), and the comment now says it avoids a forward lookup of localhost. The PR body is corrected. The first commit's body still has the old sentence. The correction is in the body of this round's commit. Please take the squash message from there or from the PR body.

4. Listener addresses. Confirmed: on the channel address hostString=127.0.0.1, and getHostName() did a reverse lookup (13 ms here). getHostString() is now used at all three places. The other tests of the class still use getHostName() on listener addresses. I left them alone, since they are outside this change.

The opendj-grizzly module passes, 1040/1040.

[maximthomas]

praise: All four round-1 points are fixed where they occur, and CI at afbe983 is green on every build-maven cell, windows-latest 11 and 26 included.

• offlineAddress.getAddress().getHostAddress() (:215) makes the "numeric host" comment true, and the online leg and both proxy clients use getHostString() of listener addresses (:221, :614, :691). None of them does a name lookup inside the 30 s window any more.
• catch (final RuntimeException e) in both overrides (:596, :677) means an unchecked failure now resolves the waiting context or answers the bind, instead of ending in a timeout with no cause.
• The bind diagnostic message now appends the nested cause (:672-674). The message is the only part of the result that LDAPWriter encodes.