A directory server that was down does not trust a replication server set up again meanwhile: its replayed instance key has no certificate #1188

Summary

A directory server that is down while a replication server is set up again on an empty volume (new instance key, new certificate) may never trust that replication server once it comes back. Its replay of the new instance key entry in cn=admin data can store the entry without ds-cfg-public-key-certificate. CryptoManagerSync cannot copy such an entry into the ads-truststore, so every replication connection from that directory server to the new replication server fails the TLS handshake. A restart does not help.

Found while testing #1178 (PR #1184) on the Docker image, on master d1a4213ba4 with that branch. The Docker join is not involved past setting the topology up: the servers below use plain dsreplication enable under the hood.

Observed

Topology of #534: directory servers dj-d0 and dj-d1 without a replication server, replication servers dj-r0 and dj-r1 without data. dj-d0 is stopped. dj-r1 and then dj-d1 are removed together with their volumes and set up again; each joins through a replication server. dj-d0 is started again.

• dj-d0 never connects to dj-r1:

  • dj-d0 logs Directory server DS(...) encountered an unexpected error while connecting to replication server dj-r1:8989 for domain "...": SSLHandshakeException((bad_certificate) PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed);
  • dj-r1 logs ... but the SSL handshake failed ... The error was: (bad_certificate) Received fatal alert: bad_certificate.

  While dj-r0 is up, dj-d0 replicates through it. Once dj-r0 stops, dj-d0 has no replication server left and refuses writes (53, "not able to connect to any Replication Server").

• dj-r1 presents the certificate AF:38:87:...:7D:30 (alias eac4a9914321d9c50d4761137d03535b, CN=dj-r1). The ads-truststore of dj-d0 (keytool -list on data/config/ads-truststore) does not hold it, only the certificate dj-r1 had before (9D:58:99:...:A8:E3). It does hold the new certificate of dj-d1, which was set up again after dj-r1.

• ds-cfg-key-id=EAC4A9914321D9C50D4761137D03535B,cn=instance keys,cn=admin data exists on all four servers, with the same ds-sync-hist: dn:000001a12fbd9b4b30c100000008:add. On dj-r0, dj-r1 and dj-d1 it holds ds-cfg-public-key-certificate;binary. On dj-d0 it has no ds-cfg-public-key-certificate at all.

• The original add is the one dsreplication made on dj-r0 (server ID 12481 = 30c1, its cn=admin data domain) when dj-r1 joined. dj-d0 received it by replay after it started: ADD REQ conn=-1 op=65 ... dn="ds-cfg-key-id=EAC4A991...,cn=instance keys,cn=admin data" type=synchronization, result=0. The replay ran concurrently with other replayed operations of cn=admin data: the add of the instance key of the new dj-d1 (43AA..., which did arrive with its certificate), deletes and adds under cn=Servers, and modifies of cn=all-servers and cn=admin data.

• Restarting dj-d0 does not repair it. CryptoManagerSync.searchAdminSuffix() finds the entry, but TrustStoreBackend.addCertificate refuses an entry without the certificate attribute. CryptoManagerSync.addEntry() reports that only with logger.debug(INFO_TRUSTSTORESYNC_ADD_FAILED, ...), so nothing shows in the error log.

Not established

Why the replayed add carries no certificate. Either the AddMsg built on dj-r0 lacks the ;binary attribute, or the attribute is lost on replay. The external changelog does not serve cn=admin data, so the message itself could not be read. In a shorter run of the same steps, without the earlier restarts of dj-r0 and dj-d1 in other roles, the key did arrive with its certificate. There the handshake failures lasted only for the first seconds after dj-d0 started, until the replay brought the certificate.

To reproduce

.github/scripts/docker-test-replication.sh of PR #1184: run the REPLICATION_ROLE block, stopped right after the case "A joiner goes through a replication server while the directory server ahead of it in REPLICATION_PEERS is down". That takes about 75 minutes on a 1-CPU Docker VM. With the Alpine image, all three runs that reached this point left dj-d0 without the new certificate of dj-r1. With the Debian image, one run showed the same handshake failures for at least three minutes before it ended for another reason; in another, dj-d0 did connect to dj-r1. Then:

# the certificate dj-r1 presents, and what the truststore of dj-d0 holds
docker exec dj-d0 keytool -printcert -sslserver dj-r1:8989 | grep SHA1
docker exec dj-d0 sh -c 'keytool -list -v -keystore /opt/opendj/data/config/ads-truststore -storepass "$(cat /opt/opendj/data/config/ads-truststore.pin)"' | grep -E "Alias|Owner|SHA1"
# the instance key of dj-r1 on dj-d0, without its certificate
docker exec dj-d0 /opt/opendj/bin/ldapsearch -h localhost -p 4444 -Z -X -D "cn=Directory Manager" -w <password> \
  -b "cn=instance keys,cn=admin data" -s one "(objectClass=*)" ds-cfg-key-id ds-cfg-public-key-certificate ds-sync-hist

Expected

A directory server that was down learns the certificate of every replication server set up meanwhile, and connects to it once it is back. If an instance key cannot be copied into the trust store, that is reported in the error log, not at debug level only.

Impact

In Kubernetes, a replication server pod rescheduled on a new volume while a directory server pod is down leaves that directory server with one replication server fewer. If it is the only other one, the directory server is cut off from the topology and refuses writes.

PR #1184 runs its fail-over check, each replication server stopped in turn, before this case, where every server trusts every other.