[#1178] Give the background join of the Docker image a server role, keep a failed bootstrap or upgrade unhealthy, and test the upgrade from a released image

[vharseko]

Summary

#1178: a server role for the background join

OPENDJ_REPLICATION_TYPE=simple made every server a directory server and a replication server at once. The one-shot srs and sdsr types, the only way to run a replication server without data or a directory server without one, are deprecated in its favour. REPLICATION_ROLE now sets the part a server of simple takes: combined (the default, as before), directory or replication.

• Each side of dsreplication enable gets its own role. Without the role flags the tool gives both servers a replication server, so a join through a directory server handed it one (ReplicationCliMain.configureServer).
  • The peer's role is read from the peer, under its join lock, in one search of its cn=config. That search returns its replication server, its domain and its backend for BASE_DN, plus the role it publishes next to its state in cn=Docker Join.
  • A peer that publishes no state is combined, as every server was before roles. That covers a peer on an image before roles, or a master that runs no join.
• The role is recorded on the volume and does not change after that. setup.sh checks REPLICATION_ROLE before it sets anything up and writes it to .replication-role in the data directory.
  • A later start with another value only logs that the server keeps its role. So does an unknown value.
  • A volume set up by an image before roles is read once from its replication configuration, then recorded. A replication server without a domain counts as the replication server of srs only when REPLICATION_ROLE=replication says so, or when it has no backend for BASE_DN either, which only this image sets up, for a replication server.
• Membership, the reset of an unregistered server and the peers asked about registration follow the role.
  • A replication server is a member through its replication server, and holds no backend whose writes a reset would hold.
  • A peer whose only replication configuration is a replication server is asked about registration only when it publishes the role replication. On a combined server that state is an enable that stopped half-way.
• A replication server holds no data and is never a seed.
  • It goes the pending road, and nothing initializes it. On a later start it publishes rejoining until a round finds it a member, so that no peer enables through it before a reset takes it out.
  • The first peer of REPLICATION_PEERS that holds data seeds the topology. Replication servers listed ahead of it are passed over, by the role they publish.
• A directory server does not enable with a directory peer that no replication server joined yet. dsreplication enable refuses a topology without one, and the log says why.
• The repair of the replication server lists adds only peers that run a replication server.
  • Each peer a list lacks is asked whether it runs one.
  • A peer that does not answer is left for a later start, whatever the roles: a directory server put into a list by mistake would stay there for good, while a replication server left out is added later, and adds this server to its own lists when it starts.
• setup.sh creates no backend and loads no entries for a replication server of the background join.
• BASE_DN goes into the search filters of the join escaped (RFC 4515).

#1182: a failed bootstrap stays unhealthy across restarts

run.sh marks a volume .bootstrap-pending before its bootstrap. It removes the mark only once the whole bootstrap has succeeded: the bootstrap script, and the replicate.sh of a one-shot type.

A start over a volume that still carries the mark starts the server so it can be looked at. It runs the upgrade (start-ds refuses an instance of another version) but not the join, never writes the health marker, and logs what to do: remove the volume, or finish the setup by hand and remove the mark. This holds for a volume of the background join as well: an initialize from the topology would bring its data, but nothing else the bootstrap was to configure. Once the mark is removed, such a volume joins and initializes from the topology. Volumes of earlier images carry no mark and start as before.

#1185: a failed upgrade stays unhealthy across restarts

The upgrade records the new version in buildinfo before its post-upgrade tasks (the index rebuilds and the verification of the DN equality indexes), so after one of them failed, or was cut off by a stop, the next run of upgrade found nothing to do and the container reported itself healthy.

run.sh marks a volume .upgrade-pending before it upgrades it to another version (major.minor.point of data/config/buildinfo against the image's template/config/buildinfo, as BuildVersion.equals compares them), and removes the mark once upgrade succeeded, post-upgrade tasks included. A mark over a volume whose version is already the image's can only come from an upgrade that recorded the version and did not complete: that start stays unhealthy and logs what to do (rebuild the indexes upgrade.log names and remove the mark, or restore the backup). An upgrade that failed before it recorded the version is run again, as before.

#1183: the upgrade from a released image in CI

Both Docker jobs bootstrap a volume with openidentityplatform/opendj:${release_version} (-alpine on the Alpine job). They then start the build image over it and check three things: the volume was upgraded (no "has already been upgraded"), the container reports healthy, and it serves the entries.

It also covers #1185: a successful upgrade leaves no .upgrade-pending; a mark over an upgraded volume keeps the container unhealthy until it is removed; an upgrade failed by a read-only buildinfo (it fails in changeBuildInfoVersion, after the upgrade tasks) leaves the mark. The step warns and is skipped when the latest release has no image on Docker Hub yet (the pull answers manifest unknown, or, on a daemon with the containerd image store, failed to resolve reference ...: not found): release.yml pushes the images after it publishes the release. Any other failed pull fails the step.

A second new step covers #1182: a restart over a volume whose create-backend failed stays unhealthy, and turns healthy once the setup is finished by hand and the mark removed.

Documentation

README.md:

• the new Server roles section;
• how srs and sdsr map onto simple;
• the Kubernetes layout with separate replication servers;
• REPLICATION_ROLE in the environment table;
• the .bootstrap-pending and .upgrade-pending marks.

Installation Guide (To Run OpenDJ in Docker, To Upgrade a Docker Container, from #1179): a restart after a failed first start or a failed upgrade stays unhealthy, and how to get past it by removing the mark, instead of "remove the volume" and "do not restart". Administration Guide, the replication NOTE: REPLICATION_ROLE, linked to the standalone directory server and replication server sections.

.github/scripts/docker-test-replication.sh gains:

• the Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 topology (two directory and two replication servers): roles held across restarts and when started again in another role, the roles published and recorded, each replication server stopped in turn, with the change made once both directory servers are connected to the other one, and before the joins on empty volumes (A directory server that was down does not trust a replication server set up again meanwhile: its replayed instance key has no certificate #1188);
• the role of a volume set up before roles (no .replication-role) inferred from its replication configuration, not from REPLICATION_ROLE: combined, directory and replication volumes, the directory and replication ones started under combined, a combined server without its domain, and the same volume under REPLICATION_ROLE=replication, as the replication server of srs leaves it before roles;
• the repair of the replication server lists leaving a peer that does not answer out, and adding it on a later start;
• setup.sh refusing an unknown role in a container before it sets anything up;
• the time per case, longest first, at the end of every run, and every wait that takes over 20 s, to find where the script spends its time;
• the failed background-join bootstrap (dj-bf) staying unhealthy until the mark is removed, then initializing from the topology;
• a replication server and a directory server joining through a replication server;
• directory servers with no replication server;
• a replication server listed first;
• an unknown role;
• a replica joining through a master that runs no join, which is given a replication server.

Testing

Run locally on a 1-CPU Docker VM (4 CPUs for the review round 3 checks), with images built from this branch over a 5.2.0-SNAPSHOT build of master, and the timeouts of the script tripled:

• CI run 38311485177 (ee8b068, review round 2) passed the whole script in both Docker jobs.
• Review round 3, on dj-0/dj-1/dj-2 through the domain-less volume, the repair case and the new srs volume case: passed on Debian (1221 s) and Alpine (679 s). With legacy_role no longer reading REPLICATION_ROLE=replication, the new case fails: dj-1 was recorded combined from a replication server and a backend under REPLICATION_ROLE=replication.
• The whole of docker-test-replication.sh on Debian passed, in 13401 s, before the fail-over step moved ahead of the joins on empty volumes and the timings were added.
• On Alpine, the run before that move failed at the fail-over step, which is how A directory server that was down does not trust a replication server set up again meanwhile: its replayed instance key has no certificate #1188 was found. Two runs after it failed earlier, at cases this PR's last round does not change (the parallel start of three fresh servers, the first start of the Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 topology): every join stopped logging after its first rounds, while another test stand ran on the same 1-CPU VM and its disk was 98% full. The fail-over step in its new place has not run locally on Alpine.
• Each case added in review round 2 fails against the change it pins: legacy_role answering $REPLICATION_ROLE, its server) branch always answering replication, the repair adding a peer that does not answer, a setup.sh that refuses an unknown role only after it set the instance up, and the join.sh of round 2 (the replication server without a backend recorded combined).
• Round 1 (before review): the REPLICATION_ROLE block passed, and failed on the scripts of master as expected (the directory server dj-d0 runs a replication server).
• The CI steps failed bootstrap and upgrade from a released image (with the Docker image: a restart after a failed post-upgrade task reports healthy, with the upgrade never completed #1185 checks) were run as GitHub runs a bash step (bash -eo pipefail), with the same text as in the workflow, on Debian and Alpine: they pass, the upgrade from 5.1.2 and 5.1.2-alpine. failed bootstrap on the run.sh of master fails: after the restart the container reports healthy with no userRoot. The pull of the released image, run the same way: a missing tag warns and skips, a missing repository fails the step. The pattern of the skip was checked against what a live pull of a missing tag answers on the classic image store (Docker 29.7.2) and on the containerd one (docker:29-dind, 29.9.0), and against sample messages of a rate limit, a refused repository and a network failure, which still fail the step.
• The functions that tell a role from a replication configuration were checked against 30 combinations of input.
• The changed AsciiDoc chapters render with AsciidoctorJ 2.5.3 (-v) without warnings.

Not run locally: the rest of the workflow, which CI runs.

Notes

• dsreplication enable between two replication servers finds BASE_DN only on a directory server of the topology it can reach. A replication server that joins through another one while every directory server is down waits for one to come back. This is documented in the README.

Fixes #1178
Fixes #1182
Fixes #1183
Fixes #1185

[maximthomas]

praise: The role is settled up front and kept with the volume, so a later environment change cannot quietly turn a server into something else.

• setup.sh:35-46 refuses an unknown REPLICATION_ROLE before anything is set up, and writes the role to .replication-role.
• join.sh:1037-1047 refuses a .replication-role that holds anything else. It infers the role of a pre-roles volume once, with legacy_role, and records it.

---

issue (blocking): The unchanged dj-bf case contradicts the new .bootstrap-pending branch, and both build-docker and build-docker-alpine are red at 23a3a58.

.github/scripts/docker-test-replication.sh:403-419, opendj-packages/opendj-docker/run.sh:149-153

dj-bf is a background-join node whose BOOTSTRAP fails. The case restarts it and expects wait_healthy dj-bf and "initializing from dj-0" (:414-417). With #1182, that restart takes the pending branch: no upgrade, no join.sh, no health marker. dj-bf logs "The bootstrap of this volume did not complete…" and the script fails with timed out waiting until dj-bf is healthy (run 37827037318, step "Docker test replication", both jobs). Every step after it was skipped. That covers the new REPLICATION_ROLE block of the same script, "Docker test failed bootstrap" and "Docker test upgrade from a released image", none of which has run in CI. Which contract do you want? Should a background-join volume whose bootstrap failed still recover on restart by initializing from the topology, as #1115's dj-bf pins? Or should it stay unhealthy until someone fixes it by hand, as #1182 now does? Your answer picks the fix below, and either fix needs a green run of both Docker jobs.

# dj-bf under the #1182 contract
docker restart dj-bf >/dev/null
wait_until 300 "dj-bf reports its pending bootstrap" logs_have dj-bf "The bootstrap of this volume did not complete"
[ "$(docker inspect -f '{{.State.Health.Status}}' dj-bf)" != healthy ] || fail "dj-bf turned healthy over a failed bootstrap"
docker exec dj-bf rm /opt/opendj/data/.bootstrap-pending
docker restart dj-bf >/dev/null
wait_healthy dj-bf
logs_have dj-bf "initializing from dj-0" || fail "dj-bf joined with the data of its failed bootstrap"

Or keep #1115's recovery: let a volume that still carries INITIALIZE_PENDING bypass the pending branch, so the initialize from the topology replaces whatever the failed bootstrap left (if [ -f "$BOOTSTRAP_PENDING" ] && ! { join_requested && [ -f "$INITIALIZE_PENDING" ]; }; then). Then make the README's .bootstrap-pending paragraph say which road a background-join volume takes.

---

issue (non-blocking): A start over a .bootstrap-pending volume also skips ./upgrade, so under a newer image start-ds refuses to start and the container crash-loops.

opendj-packages/opendj-docker/run.sh:149-153

The upgrade sits in the elif. A marked volume started under an image of another version therefore reaches BuildVersion.checkVersionMismatch() (DirectoryServer.java:1354, BuildVersion.java:100-104), which throws when the binary and instance versions differ. start-ds exits, PID 1 exits, and the "finish the setup by hand" road that the log line offers cannot be taken under that image. Same-image restarts, which is what the CI step does, are not affected.

if [ -f "$BOOTSTRAP_PENDING" ]; then
  sh ./upgrade -n --force || true
  echo "The bootstrap of this volume did not complete, this container will not report itself healthy: ..."
elif sh ./upgrade -n --force; then

---

issue (non-blocking): repair_replication_servers adds a directory server that does not answer to every replication server list of a combined server, when that directory server is the only peer the lists lack.

opendj-packages/opendj-docker/bootstrap/join.sh:879-911

mixed turns yes only when a lacking peer answers without server. Peers that every list already holds are never asked. Take combined A and B and directory C, with C down while A restarts. B is skipped, C fails replication_config, mixed stays no, and C:REPLICATION_PORT is added to A's server list and to its BASE_DN, cn=schema and cn=admin data domain lists. On later starts every list holds C, so C is never asked again, and cleanup_departed keeps it. That is the outcome the comment at :891-896 says this arm prevents. Replication keeps working, but A dials a port that nothing listens on for good. A peer replication server left out is added on a later start, as the comment already accepts, so the unknown hosts can be deferred on every role:

for host in $unknown; do
  echo "join: could not ask $host whether it runs a replication server, its place in the replication server lists is checked on a later start"
done

---

issue (non-blocking): The upgrade step pulls openidentityplatform/opendj:${release_version} from Docker Hub with no fallback, so Docker CI goes red on every PR between a GitHub release and its Docker push.

.github/workflows/build.yml:829-833

release_version is the name of releases/latest (:499-501). release.yml publishes that release in release-maven and pushes the Docker tags only in release-docker, which needs release-maven. In that window, and for as long as a failed release-docker stays unfixed, the pull fails in both jobs on every PR, through nothing the PR did.

docker pull "$RELEASED" || { echo "::warning::$RELEASED is not on Docker Hub yet, the upgrade test is skipped"; exit 0; }

---

issue (non-blocking): On the ready road, a replication server publishes ready before its first membership check. On the pending road, ready means it is already a member.

opendj-packages/opendj-docker/bootstrap/join.sh:1104-1112

The comment's premise, "this volume holds the data of the topology", does not hold for a replication server. Two servers reach this road: an RS that the survivors no longer register, and a legacy srs volume restarted as simple with REPLICATION_ROLE=replication. Either one advertises ready until reset_if_unregistered publishes rejoining in its first round. A pending peer that enables through it in that window may be counted a member through the RS's own registry. Not run: this needs a multi-container run, and I have not traced whether that leaves a lasting island or heals after the RS resets. Publishing pending for ROLE=replication until a round finds it a member would match the pending road.

---

suggestion (non-blocking): The dj-nm/dj-nmr case does not pin that a peer which publishes no role is taken for combined.

.github/scripts/docker-test-replication.sh:742-747

dj-nm runs no join, so published_role falls through to *) echo combined (join.sh:400-406). Consider a mutant that returns directory for a peer that publishes nothing. It passes --noReplicationServer1 for dj-nm, dj-nmr's own RS carries the initialize, and both assertions stay green while the legacy master is left without a replication server.

runs_replication_server dj-nm || fail "dj-nm, which publishes no role, was not given a replication server"

Pin: after :747, kills the mutant above.

---

suggestion (non-blocking): legacy_role, which infers the role of a volume set up before roles, has no CI observable.

opendj-packages/opendj-docker/bootstrap/join.sh:1018-1048

It runs only when .replication-role is missing. Every replication volume in the script is set up by this image, which writes the file, and the released-image upgrade step sets no OPENDJ_REPLICATION_TYPE. A mutant such as server) always echoing combined stays green. Under that mutant, an srs replication server upgraded with REPLICATION_ROLE=replication is recorded as combined for good.

docker exec dj-r0 rm /opt/opendj/data/.replication-role
docker restart dj-r0 >/dev/null
wait_healthy dj-r0
[ "$(recorded_role_of dj-r0)" = replication ] || fail "dj-r0 was recorded $(recorded_role_of dj-r0) from its replication configuration"

Pin: the same on a combined volume, expecting combined.

---

suggestion (non-blocking): No case exercises setup.sh refusing an unknown REPLICATION_ROLE, which is the road a real container takes.

opendj-packages/opendj-docker/bootstrap/setup.sh:35-46, .github/scripts/docker-test-replication.sh:722-725

The only unknown-role case runs join.sh through --entrypoint, with no volume, no run.sh and no setup.sh. A mutant that drops setup.sh's *) … exit 1 writes bogus to .replication-role, sets the volume up as a data holder, and stays green.

start_node dj-bogus dj-0,dj-bogus -e REPLICATION_ROLE=bogus
wait_until 120 "setup refuses the role" logs_have dj-bogus "REPLICATION_ROLE=bogus is not one of combined, directory or replication"
if docker exec dj-bogus test -e /opt/opendj/data/.replication-role; then fail "setup recorded an unknown role"; fi

---

suggestion (non-blocking): "one replication server is enough" stops dj-r1 without checking that any directory server was connected to it.

.github/scripts/docker-test-replication.sh:868-871

If both directory servers sit on dj-r0, the step passes with no failover at all. Not traced: whether the broker always spreads two equal-weight replication servers one per directory server. Stopping each RS in turn makes the step independent of that choice.

docker start dj-r1 >/dev/null; wait_healthy dj-r1
docker stop dj-r0 >/dev/null
add_ou dj-d0 roles-other-rs
wait_has dj-d1 "ou=roles-other-rs,dc=example,dc=com"

---

nitpick (non-blocking): roles_hold's comment "every server joined through dj-d0 at some point" is false at its third call.

.github/scripts/docker-test-replication.sh:791-792

Before :866, dj-r1 and dj-d1 come back on empty volumes with dj-d0 stopped, and the script asserts that they joined through dj-r0 and dj-r. The check still holds because dj-r0 joined through dj-d0. Suggested wording: "dj-r0 joined through dj-d0: an enable that took dj-d0 for a combined server would have given it a replication server".

[vharseko]

Thank you for the review. All nine points hold against the code; they are in dd6113e, on master f3c7a3c (which brought #1179).

dj-bf against .bootstrap-pending. Confirmed, and it is my miss: locally I ran the REPLICATION_ROLE block and the new steps, never the whole script, so the #1115 case never met the #1182 branch. I took the #1182 contract, as the issue's Expected states it ("whatever the replication settings") and as the README already described it. The initialize brings the data of the topology, but nothing else a BOOTSTRAP was to configure. dj-bf fails exactly that way: its custom part fails after setup.sh. A replication server never carries INITIALIZE_PENDING, so the bypass would not have covered it anyway. dj-bf now follows your first sketch. It checks the pending log line after the restart, uses stays_unhealthy rather than one look at the status (a single look right after a restart reads starting), and checks that no join ran. It then removes the mark and expects the initialize from dj-0. The README paragraph now says that a background-join volume takes the same road and joins once the mark is gone. The Installation Guide from #1179 says so as well: a restart over a failed first start stays unhealthy, and removing the mark after finishing the setup by hand is the alternative to removing the volume.

Upgrade over a pending volume. Taken as suggested. A failed upgrade is logged and the server is started anyway.

#1185 in this PR. The upgrade counterpart of #1182 is in the same commit. run.sh marks a volume .upgrade-pending before it upgrades it to another version: the major.minor.point of data/config/buildinfo differs from the image's template/config/buildinfo, as BuildVersion.equals compares them. It removes the mark once upgrade succeeded, post-upgrade tasks included. On a later start, a mark over a volume whose version is already the image's can only come from an upgrade that recorded the version and did not complete. That start stays unhealthy and says what to do: rebuild the indexes upgrade.log names and remove the mark, or restore the backup. An upgrade that failed before it recorded the version is run again, as before. The upgrade step of build.yml now also checks three things: a successful upgrade leaves no mark; a mark over an upgraded volume keeps the container unhealthy until it is removed; an upgrade failed by a read-only buildinfo (it fails in changeBuildInfoVersion, after the upgrade tasks) leaves the mark. The "do not restart" advice in To Upgrade a Docker Container is replaced accordingly.

Repair adds an unanswering directory server. Confirmed: the peers that every list holds are never asked, so mixed could not see C's role. I took the deferral for every role and dropped mixed. The cost is what the comment already accepted for a replication server left out. A peer that was down also adds this server to its own lists when it starts, which connects the two already. The README paragraph is updated as well.

Released image not on Docker Hub yet. Confirmed against release.yml (release-docker needs release-maven). Taken as suggested, with a comment on why.

ready before the first membership check on a replication server. I agree with the premise and the fix, with one change: the replication server publishes rejoining, not pending, until a round finds it a member, and ready from joined() then. Both keep peers from enabling through it, since only ready is trusted. But all_others_pending counts pending as "waits for data". A first peer that starts from scratch at the same time could then seed next to the topology that this replication server serves. rejoining keeps any_other_may_hold_data true for it, as ready did. The state comment at the top of join.sh and the README name this. Like you, I have not built a multi-container case for the window itself. The restarts of dj-r0 and dj-r1 in the role block now go through it, and dj-r1 still joins through dj-r0.

dj-nm pin. Taken. runs_replication_server and recorded_role_of moved up with the other helpers, because the pin runs before the role block where they were defined.

legacy_role pin. Taken, for all three roles: dj-1 (combined) loses .replication-role while it is down in the restart case of the early block. dj-r0 (replication) and dj-d1 (directory) lose it in a restart of their own in the role block, and roles_hold reads what they recorded.

setup.sh and an unknown role. Taken, with one change: the container exits once setup.sh fails, because nothing creates data/config. docker exec would then fail and pass the check vacuously. The volume is therefore read through a throwaway container of the image.

One replication server stopped. Taken: dj-r1 and then dj-r0 are stopped in turn, with a change from the other directory server each time.

roles_hold comment. Taken as worded.

Docs. Besides the two Installation Guide procedures above, the replication NOTE of the Administration Guide now names REPLICATION_ROLE and links the standalone directory server and replication server sections. All four Docker chapters render with AsciidoctorJ 2.5.3 -v without warnings.

Testing (local, 1-CPU Docker VM, images built from this branch, with the local timeouts ×3):

• the whole docker-test-replication.sh on Debian: passed, in 20403 s;
• the upgrade step, with the Docker image: a restart after a failed post-upgrade task reports healthy, with the upgrade never completed #1185 checks, and the failed-bootstrap step, on Debian and Alpine: passed;
• the whole script on Alpine: failed at [#1086] Join replication in the background on every start of the Docker image #1115's dj-2, the third server's first join. Its first dsreplication enable with dj-0 ran past the 270 s attempt bound and was cut off (124). Every later round found "the peers register this server but its own cn=admin data does not", and the reset's dsreplication disable exited 8 (ERROR_CONNECTING) each time. reset_replication is master's here, except that it skips hold_writes for a replication server, and the same case passed on Debian. So I read it as the recovery after an enable cut off half-way, which an overloaded host exposed, rather than as something this round changed. I have not proven that, and the output of that disable does not reach the log. CI runs both jobs on faster runners; if dj-2 fails there too, I will take it up in a separate issue.

[maximthomas]

praise: Round 2 closes every issue of round 1 where it was, and pins the hardest road of the role topology end to end.

• .github/scripts/docker-test-replication.sh:881-891 stops dj-d0, brings dj-r1 back on an empty volume and requires "joined the replication topology through dj-r0", so a replication server enabling through another replication server (--onlyReplicationServer on both sides) is shown to work, green in both Docker jobs.
• .github/workflows/build.yml:849-866 pins the #1185 mark from both sides: a clean upgrade leaves no .upgrade-pending, and a mark over an upgraded volume keeps the container unhealthy until it is removed (run.sh:83-97).
• The round-1 roads are fixed where they were: dj-bf removes .bootstrap-pending (docker-test-replication.sh:435), the .bootstrap-pending start still runs the upgrade (run.sh:181-188), and unanswering peers stay out of the replication server lists (join.sh:891).

---

issue (non-blocking): The new pre-roles volume cases restart every volume under the REPLICATION_ROLE it was set up with, so they cannot tell legacy_role from an echo of $REPLICATION_ROLE.

.github/scripts/docker-test-replication.sh:860-867, :337-346

dj-1 loses .replication-role at :337 and restarts under the default combined; dj-r0 and dj-d1 lose it at :861, and docker restart keeps replication and directory from :789-791. Each case expects exactly the env's value, and the swap at :869-870 comes after the role is recorded again, so it never reaches legacy_role. The mutant legacy_role() { echo "$REPLICATION_ROLE"; } passes the assert at :345 and both roles_hold calls. The description lists "the role of a volume set up before roles inferred again, for combined, directory and replication volumes" among what the script gains. The property the inference exists for has no observable: a pre-roles directory or combined volume started under another role keeps its configured role. A second mutant survives too: the server) branch always echoing replication.

# after the loop at :866: a pre-roles directory volume under another REPLICATION_ROLE
docker exec dj-d1 rm /opt/opendj/data/.replication-role
docker rm -f dj-d1 >/dev/null
start_node dj-d1 $ROLE_PEERS -e REPLICATION_ROLE=combined -e REPLICATION_RETRY_COUNT=30
wait_until 300 "the pre-roles dj-d1 is a member again" logs_have dj-d1 "already a member of the replication topology"
[ "$(recorded_role_of dj-d1)" = directory ] || fail "dj-d1 was recorded $(recorded_role_of dj-d1) from its directory replication configuration"

Pin: the echo mutant records combined and fails the last line.

---

issue (non-blocking): dj-bogus and vol-dj-bogus are missing from NODES and VOLUMES.

.github/scripts/docker-test-replication.sh:26-27, :749

Only the success path removes them by hand (:754-755). When the wait at :750 times out or the check at :751 fails, the ERR trap's dump_logs loops over $NODES and never prints the one container log that explains the failure. cleanup also leaves dj-bogus and its volume behind, so on a reused host the next run's docker run --name dj-bogus fails on the name. On ephemeral runners only the missing log matters.

NODES="... dj-nm dj-nmr dj-bogus"
VOLUMES="... vol-dj-bogus"

---

suggestion (non-blocking): The released-image pull turns any failure into a green skip, not only a tag that is not on Docker Hub yet.

.github/workflows/build.yml:836, :1317

docker pull "$RELEASED" || { ...; exit 0; } ends the step with success before the first upgrade assertion on a rate limit, a registry or network outage, or a renamed repository, and the warning names the release window whatever the cause. The jobs API then reports "Docker test upgrade from a released image" as passed either way. Run 37955316819 did pull: it carries no such warning.

if ! out=$(docker pull "$RELEASED" 2>&1); then
  case "$out" in
    *"manifest unknown"*|*"not found"*) echo "::warning::$RELEASED is not on Docker Hub yet, the upgrade test is skipped"; exit 0 ;;
    *) echo "$out"; exit 1 ;;
  esac
fi

---

suggestion (non-blocking): Neither join.sh change of this round has a case that fails without it.

opendj-packages/opendj-docker/bootstrap/join.sh:1108, :891

The script checks published_state only on the combined dj-0/dj-1/dj-2. The replication-server restarts (:850-856, :860-865, :869-872, :903-913) check the "already a member" line, health and roles_hold, which reads the role, not the state. No case leaves a registered peer down while another server repairs its lists. Two reverts leave the script green: dropping || [ "$ROLE" = replication ] at :1108, which brings back ready at a replication server's start, and restoring the mixed=no fallback that adds an unanswering peer.

# with dj-r0's first membership round held back, as the dj-held block does for dj-2 (:498-503)
[ "$(published_state dj-r0)" = rejoining ] || fail "dj-r0 publishes itself ready before its membership check"

Pin: the first revert publishes ready and fails this line. For the second, stop a registered peer missing from one list, restart another server, and assert that the list still lacks it.

---

suggestion (non-blocking): The setup.sh refusal case checks that no role was recorded, not that the refusal comes "before it sets anything up".

.github/scripts/docker-test-replication.sh:748-753

Move the role check in setup.sh after the /opt/opendj/setup call (:92). The mutant then sets the instance up, logs the same refusal and exits before it records a role, so the wait at :750 and the test -e at :751 both pass. That leaves the hazard setup.sh:30-32 names unpinned: a volume set up for a role nobody asked for.

if docker run --rm --entrypoint test -v vol-dj-bogus:/opt/opendj/data "$IMAGE" -e /opt/opendj/data/config/buildinfo; then
  fail "setup.sh set the instance up before it refused REPLICATION_ROLE=bogus"
fi

[vharseko]

Thank you for the second round. All five points hold against the code. Four are in ee8b068, on master 5b210f3 (#1187); the fifth is in half. Each new case was checked against the mutant it is for, as listed under Testing.

legacy_role echoing $REPLICATION_ROLE. Confirmed: every pre-roles case restarted the volume under the role it was set up with. dj-r0 and dj-d1 now lose .replication-role and are started again as new containers under REPLICATION_ROLE=combined, which they log as a role they keep. roles_hold then expects replication and directory. This pin also caught a real hole. The replication server dj-r0 has a replication server, no domain and no backend for BASE_DN. Under any REPLICATION_ROLE other than replication, the server) branch took that for a combined server whose enable had stopped half-way. Only this image sets a volume up without that backend, and only for a replication server. A volume set up before roles always has one, since setup.sh created userRoot unconditionally. So server) now answers replication when the backend is missing as well. The round-2 image fails the new case.

For the second mutant (server) always replication) there is a case of its own on the combined dj-1. Its domain of dc=example,dc=com is deleted, which leaves the half-way state of an image before roles: a replication server and a backend, no domain. .replication-role is removed and the server is restarted under the default role. It must be recorded combined, join again through dj-0 and replicate.

dj-bogus in NODES and VOLUMES. Taken.

The released-image pull. Taken, narrowed to manifest unknown. Against Docker Hub, a missing tag answers manifest for openidentityplatform/opendj:<tag> not found: manifest unknown: manifest unknown. A missing repository answers pull access denied ..., repository does not exist or may require 'docker login'; with *"not found"* in the pattern a renamed repository would still have been a skip, so "not found" is left out. Any other failure prints the output of the pull and fails the step.

setup.sh refusing before it sets anything up. Taken, with one change. The test -e .../config/buildinfo alone did not catch the mutant I tried first. That mutant logs the refusal where setup.sh logs it today, then runs the setup anyway and fails only after it. The wait at :750 of round 2 then returned while /opt/opendj/setup was still running, and the volume had no buildinfo yet. The case now waits for the line of run.sh that comes after the bootstrap returned (... setup.sh failed, this container will not report itself healthy), and only then looks at the volume. That mutant now fails at the new check.

The two join.sh changes of round 2.

• The unanswering peer in the repair. Taken. The pin runs where the dropped mixed=no fallback applied: an all-combined topology, here dj-0/dj-1/dj-2. dj-2 is removed from the list of dj-1's replication server and stopped, and dj-1 is restarted. dj-1 must log that it leaves dj-2 for a later start and must not add it. Once dj-2 is back, the next restart of dj-1 must add it. The lines are counted rather than looked for after the last Starting OpenDJ. Right after docker restart the log still ends in the start before, and my first version passed on the line of an earlier join. With the fallback restored, dj-1 adds dj-2 and the case fails.
• rejoining on a replication server's start (:1108 of round 2). Not pinned, and I record it as such. The dj-held block can hold dj-2 because the reset waits for the join lock. But reset_replication publishes rejoining for every role by itself, so it cannot tell this change apart. On a replication server that is a member, the window between publish_state and the first round's is_member is one local search. No lock or peer sits in it. A road that would keep it open is a replication server that is not a member and is not reset either, which only comes from removing its configuration by hand. That would test the hand edit more than the join. A deterministic pin needs a test hook in join.sh, such as a wait before the first round, set from the environment. I would rather not put one into the image's script, but will add it if you think the window is worth it.

One more thing the local runs found. The fail-over step from round 2 was racy. It made the change right after docker stop, before the directory servers had moved to the replication server left. In one run, dj-d0 refused the write (53, "not able to connect to any Replication Server"). In another, dj-d1 took the write while its broker was still on the replication server going down, and the change never reached dj-d0. The step now waits until both directory servers are connected to the remaining replication server before it writes. It reads that from connected-to in the replication monitor of their dc=example,dc=com domain, which names the replication server by its address. Whether a change taken in that window should be resent later is a question for replication itself, not for this PR.

With that wait, the full run on Alpine showed what the race had hidden. dj-d0 never connected to dj-r1 after dj-r1 came back on an empty volume while dj-d0 was down: every handshake failed with bad_certificate. On a stand kept alive at that point, the instance key of the new dj-r1 is in cn=admin data of dj-d0, from the same single add as on the other servers, but without ds-cfg-public-key-certificate. So CryptoManagerSync cannot put it into the ads-truststore, and says so at debug level only. Nothing in the Docker join takes part in that, so it is #1188, with the evidence and the steps to reproduce. The fail-over step now runs before the joins on empty volumes, while every server trusts every other, which is what it is there to show.

Timings. Each case now logs when it starts, each wait over 20 s logs how long it took, and every run ends, failed or not, with the time per case, longest first. The replication step took 68 min in each Docker job of the round-2 run (37955316819), most of the jobs' time; the time per case is where making it faster starts.

Testing (local, 1-CPU Docker VM, images built from this branch, with the local timeouts ×3):

• the new cases on dj-0/dj-1/dj-2: passed;
• the REPLICATION_ROLE block, on Debian: passed, in 4914 s;
• each against its mutant, failed where expected:
  • legacy_role echoing $REPLICATION_ROLE: the role-less dj-r0 under combined;
  • server) always replication: the domain-less dj-1;
  • the mixed=no fallback restored: the repair case;
  • setup.sh refusing after the setup: setup.sh set the instance up before it refused REPLICATION_ROLE=bogus;
  • the join.sh of round 2: the role-less dj-r0;
• the whole script on Debian: passed, in 13401 s, before the fail-over step moved and the timings were added;
• the whole script on Alpine: the run before the move failed at the fail-over step, which is how A directory server that was down does not trust a replication server set up again meanwhile: its replayed instance key has no certificate #1188 was found. Two runs after it failed earlier, at cases this round does not change: the parallel start of three fresh servers, and the first start of the Missing Changes count is piling up during peak load testing with OpenDJ 4.9.4 #534 topology. In both, every join stopped logging after its first rounds, while another test stand ran on the same 1-CPU VM and its disk was 98% full. So the fail-over step in its new place has not run locally on Alpine; CI runs it in both jobs.

[maximthomas]

praise: The round answers both round-2 test gaps, and the CI step now tells "not published yet" apart from "pull broken".

• build.yml skips the released-image upgrade test only on a missing tag (*"manifest unknown"*). A rate limit or a registry outage now fails the step, so it no longer passes untested.
• "roles: role-less volumes" restarts dj-r0 and dj-d1 under REPLICATION_ROLE=combined, so the recorded role must come from the replication configuration and not from the environment.
• "repair defers an unanswering peer" asserts that no add is logged at all while dj-2 is down (docker-test-replication.sh:702), not just that the server list lacks it.

---

issue (non-blocking): added_back compares a server-list-only count against a baseline that also counts domain-list adds.

.github/scripts/docker-test-replication.sh:696, :706

adds counts every adding replication server dj-2: line, which covers both the "domain list" and the "server list" lines from join.sh:900. The wait counts only …dj-2:8989 to the server list. Only the server list loses dj-2 (:693), so the last restart logs exactly one server-list line. If dj-1's log since :605 already holds one domain-list add for dj-2, added_back never turns true and wait_until 300 fails the Docker job on correct code. The comment at :694 expects this case ("may have logged either line already"). Here is one way that line gets logged: in "rejoin together", dj-1 enables first, and its lock-free repair polls until dj-2 registers. It then reads its own lists after dj-2's replication server is configured but before dsreplication has added dj-2 to them, so it adds dj-2 to its domain lists itself. Run 38311485177 is green on both Docker jobs, so the race did not fire there.

server_adds=$(logs_count dj-1 "adding replication server dj-2:8989 to the server list")
added_back() { [ "$(logs_count dj-1 "adding replication server dj-2:8989 to the server list")" -gt "$server_adds" ]; }

Keep adds for the no-add check at :702.

---

suggestion (non-blocking): The released-image skip matches only the classic image store's wording for a missing tag.

.github/workflows/build.yml:841, :1330

On a containerd-store daemon (Docker 29.6.2, API 1.55), pulling a missing tag of openidentityplatform/opendj fails with … failed to resolve reference "docker.io/openidentityplatform/opendj:<tag>": …: not found and no manifest unknown. Today's runner reports API 1.48 (classic store), so the step is correct now. Once the runner's Docker moves to the containerd store, which fresh Engine 29 installs use by default, every build between a GitHub release and its Docker Hub push fails both Docker jobs instead of skipping. The failure is fail-safe (red, never a false green).

case $out in
  *"manifest unknown"*|*"failed to resolve reference"*": not found"*) echo "::warning::$RELEASED is not on Docker Hub yet, the upgrade test is skipped"; exit 0 ;;
  *) exit 1 ;;
esac

---

suggestion (non-blocking): No case reaches legacy_role's [ "$REPLICATION_ROLE" = replication ] || clause any more.

opendj-packages/opendj-docker/bootstrap/join.sh:1020, .github/scripts/docker-test-replication.sh:669-687, :953-963

The script reaches the server) arm twice. dj-1 at :669-687 has a backend and runs under combined. dj-r0 at :953-963 has no backend, so the new ! grep -qx backend arm answers alone. At dd6113e, the restart of dj-r0 under REPLICATION_ROLE=replication was the only case that pinned the clause. Dropping the clause changes the answer only for a pre-roles srs volume: a replication server with a backend, no domain, started under REPLICATION_ROLE=replication. The description and the comment at join.sh:1010-1013 name exactly that volume, and the mutant would record it combined for good.

dsconfig_on dj-1 delete-replication-domain --provider-name "Multimaster Synchronization" --domain-name "dc=example,dc=com"
docker exec dj-1 rm /opt/opendj/data/.replication-role
docker rm -f dj-1 >/dev/null
start_node dj-1 dj-0,dj-1,dj-2 -e REPLICATION_ROLE=replication -e REPLICATION_RETRY_COUNT=30
wait_healthy dj-1
[ "$(recorded_role_of dj-1)" = replication ] || fail "dj-1 was recorded $(recorded_role_of dj-1) from a replication server and a backend under REPLICATION_ROLE=replication"

Pin: put this where the volume is not needed as combined afterwards. With the clause dropped, the last line fails.

[vharseko]

Thank you for the third round. All three points hold against the code and are in 081a974, still on master 5b210f3.

added_back against the wrong baseline. Confirmed: adds counts both the domain-list and the server-list lines, while the wait counts only the server-list ones. So a single domain-list add of dj-2 anywhere in dj-1's log since its start in "rejoin together" made the wait impossible. The baseline of the wait is now server_adds, counted with the same text as added_back, and adds stays for the no-add check while dj-2 is down. The comment above says why the two counts differ.

The containerd image store. Taken, with your pattern, in both jobs. I checked it against a live pull of a missing tag on both stores. The classic one (Docker 29.7.2, overlay2) answers manifest for openidentityplatform/opendj:<tag> not found: manifest unknown: manifest unknown. The containerd one (docker:29-dind, 29.9.0, io.containerd.snapshotter.v1) answers no matching manifest for linux/amd64/v3 in the manifest list entries: failed to resolve reference "docker.io/openidentityplatform/opendj:<tag>": docker.io/openidentityplatform/opendj:<tag>: not found. The odd prefix does not matter, and both skip. Sample messages for a containerd rate limit (429), a refused repository (insufficient_scope), a DNS failure and a classic toomanyrequests all still fail the step. The comment above the pull now names both wordings.

legacy_role's REPLICATION_ROLE=replication clause. Confirmed: since the role-less dj-r0 is started under combined and has no backend, nothing reached the clause. The new case "pre-roles srs volume" builds that volume on dj-1, at the end of the sdsr case, where dj-1 is not used any more. dj-1 loses its domain of dc=example,dc=com and its .replication-role, and is started again as a new container under REPLICATION_ROLE=replication. It must be recorded replication, be a member again and turn healthy. With the clause dropped, the case fails with dj-1 was recorded combined from a replication server and a backend under REPLICATION_ROLE=replication.

Testing (local, Docker VM now with 4 CPUs, images built from this branch, local timeouts ×3):

• dj-0/dj-1/dj-2 through the domain-less volume, the repair case and the new case: passed on Debian (1221 s) and on Alpine (679 s);
• the same against the mutant without the clause: failed at the new case, with the line above;
• the whole script was not run again locally; CI run 38311485177 passed it in both Docker jobs on ee8b068, and this round changes only the three places above.

[maximthomas]

praise: This round closes the three items from round 3 in the places where they occur.

• server_adds (.github/scripts/docker-test-replication.sh:699) counts the same "to the server list" pattern that added_back waits on (:709). A domain-list add from an earlier start can no longer keep the wait false.
• The "pre-roles srs volume" phase (:753-769) reaches the REPLICATION_ROLE = replication clause of legacy_role. Without that clause the volume is recorded as combined and :766 fails.
• Both upgrade steps (.github/workflows/build.yml:843, :1334) now also skip on the containerd store's wording for a missing tag. A 429 or a DNS failure still fails the step.