Docker image: a restart after a failed post-upgrade task reports healthy, with the upgrade never completed #1185

Problem

When a container upgrades its volume on start and a post-upgrade task fails, the container is unhealthy, as intended. The next start over the same volume reports it healthy, although the task never completed.

Upgrade.upgrade records the new version in buildinfo (Upgrade.java#L877) before it runs the post-upgrade tasks (Upgrade.java#L887-L891). These are the index rebuilds, and from 5.2.0 also the verification of the DN equality indexes, which every upgrade from 5.1.x runs (Upgrade.java#L661-L662) and which rebuilds the indexes it finds inconsistent.

1. A rebuild fails (UpgradeTasks.java#L946-L948, #L1022-L1028). performPostUpgradeTasks marks the exit code as an error and only postpones the tasks after it (Upgrade.java#L912-L936).
2. upgrade exits non-zero, so run.sh starts the server without the health marker (run.sh#L142, #L171-L174). The container is unhealthy.
3. The container is started again: docker restart, a restart policy, a Kubernetes liveness restart, or docker rm plus docker run. run.sh runs upgrade -n --force again. isVersionCanBeUpdated finds the version in buildinfo equal to the new one and exits with success before any task runs (Upgrade.java#L816, #L991-L999).
4. run.sh writes the marker (run.sh#L161). The container reports itself healthy. Its indexes are as the failed rebuild left them, and the tasks that were to follow the failed one never ran. Nothing in the logs of that start says so.

This is the upgrade counterpart of #1182, which covers a failed bootstrap. PR #1184 fixes #1182 with a BOOTSTRAP_PENDING mark that only the bootstrap writes, so it does not cover this case.

The same ordering affects the ZIP and the native packages: a second upgrade after a failed post-upgrade task reports "already upgraded". There a person reads the output of the first run, though. In a container, a restart policy can hide that run.

Reproduce

Not reproduced on an image. A run would need a volume of an older version whose index rebuild fails during the upgrade, for example because the backend cannot be read. The analysis above comes from the code at a4a5cd8e4c.

Expected

A container never reports healthy over a volume whose upgrade did not complete, its post-upgrade tasks included. Its logs say what to do: rebuild the indexes named, or revert to the backup.

Possible approaches:

• In run.sh, mark the volume before upgrade runs (say .upgrade-pending, next to INITIALIZE_PENDING), and remove the mark only when upgrade succeeded. A start over a marked volume does not write the health marker. To get past it, the user rebuilds the indexes and removes the mark, or reverts.
• In Upgrade.upgrade, record the new version only after the post-upgrade tasks succeeded, or record which ones are pending, so that the next run of upgrade performs them again. That fixes every distribution, but changes what a re-run of upgrade does on the ZIP and the packages.

Notes

• Until this is fixed, the Installation Guide tells users not to restart a container to get past a failed upgrade, and to revert or rebuild the indexes instead (PR Describe running, upgrading and removing the Docker image in the Installation Guide #1179, review round 2).
• The upgrade log is /opt/opendj/data/logs/upgrade.log in the image; docker logs shows only "An error occurred during post upgrade task. Process aborted. Please check log for further details" and the names of the indexes being rebuilt.