Docker image: a restart after a failed first start reports healthy over a half-bootstrapped volume #1182

Problem

Suppose the first start of a container without the background join fails after setup has written data/config: for example dsconfig create-backend fails (setup.sh#L86-L88), or the import of the base entry does. That start never reports healthy, as intended (run.sh#L195-L198).

The next start over the same volume can come from docker restart, docker start, a restart policy, or docker rm plus docker run. On that start run.sh finds ./data/config (run.sh#L137) and takes the volume for an installed instance. It then runs upgrade -n --force. On equal versions that is a successful no-op (Upgrade.java#L994-L999). Without the join, nothing else is checked, so run.sh#L161 writes the bootstrap marker. The container reports itself healthy, with no userRoot backend or no base entry.

The background join (OPENDJ_REPLICATION_TYPE=simple with REPLICATION_PEERS) does not have this problem. run.sh marks such a volume INITIALIZE_PENDING before its bootstrap (run.sh#L192-L194), and a later start waits for the join instead of writing the marker. Every other container has no such guard.

Reproduce

The image was built from master 3f4deb9178 (its run.sh is identical to master). The repro makes create-backend fail with an unknown backend type:

docker run -d --name repro --init -e ADD_BASE_ENTRY=--addBaseEntry -e BACKEND_TYPE=bogus \
  -v repro_data:/opt/opendj/data <image>
# logs: The sub-type "bogus" is not a recognized type of Backend.
#       /opt/opendj/bootstrap/setup.sh failed, this container will not report itself healthy
docker restart -t 60 repro
# logs: * OpenDJ has already been upgraded to version ...
#       Starting OpenDJ
docker inspect -f '{{.State.Health.Status}}' repro          # healthy
docker exec repro /opt/opendj/bin/dsconfig list-backends -h localhost -p 4444 \
  -D "cn=Directory Manager" -w password -X -n               # adminRoot only
docker exec repro /opt/opendj/bin/ldapsearch --port 1389 -D "cn=Directory Manager" -w password \
  -b dc=example,dc=com -s base "(objectClass=*)" dn         # 32 (No Such Entry)

A container killed during its first start, for example an OOM kill or a node lost in the middle of setup, takes the same road once data/config exists.

Expected

The container never reports healthy over a volume whose bootstrap did not complete, whatever the replication settings, and its logs say what to do: remove the volume and start over, or finish the setup by hand and mark it done.

For example, run.sh could mark every volume it bootstraps before the bootstrap starts, the way INITIALIZE_PENDING does for the join. It would remove that mark only after the bootstrap succeeds. A start over a volume that still carries the mark would then not write the bootstrap marker. Volumes written by earlier images carry no mark and keep their current behaviour.

Notes

• Until this is fixed, the Installation Guide tells users to remove the container and its volume after a failed first start (PR Describe running, upgrading and removing the Docker image in the Installation Guide #1179, review round 1).
• A related observation that has not been checked on master: on the released 5.1.2 image, a volume whose setup was killed half-way went into upgrade on the next start and failed without a keystore.