Add a warm validation server so the JVM is reused between validations

[corneliusroemer-agent]

Validating one submission costs a fresh java -jar readtools.jar, and roughly half of that call is classloading and JIT warm-up that the process then throws away. The measurement that motivated this: JVM boot to main is only 0.036s, so almost none of the waste is process startup — it is the compiled parse/validate loop being discarded and rebuilt every single time. Running the identical validation repeatedly inside one JVM goes 1.66s on the first pass and settles at ~0.81s by the fourth, with distinct inputs each round so it is not a page-cache effect.

java -jar readtools.jar server --port N --threads N keeps one JVM warm and takes validations over loopback HTTP. Measured here on 100k-read pairs: ~1.3s per cold CLI run, ~0.9s warm, and ~0.3s per validation when four run concurrently in the one JVM.

Concurrency is the part I most want a second opinion on, so here is the evidence rather than an assertion. Every validation constructs its own ValidatorWrapper and shares nothing with any other. I grepped the whole v2 validation path for non-final static fields: the only ones are InsdcReadsValidator's ERROR_* and INVALID_FILE strings, and nothing in src/main ever reassigns them. Everything else static on that path is a method. Beyond that reading, a 360-validation soak at 9-way concurrency against 4 slots, mixing valid, content-invalid and structurally-invalid inputs, returned zero wrong verdicts.

Confirmed on a live deployment, not only on a bench

Four 53 MB paired-FASTQ submissions (1,455,282 reads / 190 Mbp) through the Loculus preview in loculus-project/loculus#7188:

	fork per call	warm server
readtools validation, per submission	~3 s	0.90 / 0.96 / 1.01 / 1.13 s
pod memory, peak	4.89 GB	5.15 GB

About 3x in production, against ~1.5x on a 16-vCPU dev box — a cold JVM costs a constrained pod more than it costs a fat container, so the local ratio understates this.

The server's own startup output shows the JIT curve, which is the most direct evidence that what is being reclaimed is compilation rather than process startup:

warm-up 1/3: 1.06s
warm-up 2/3: 0.71s
warm-up 3/3: 0.55s
readtools validation server ready

Total warm-up: 3 seconds, once, at pod start. The resident JVM adds ~0.26 GB.

Full-file validation stresses this in a way quick mode does not

--full (every read, rather than the first 100k per file) on a 53 MB pair, 727,641 reads per mate:

	quick	full
cold CLI	~1.15 s	~8.6 s
warm server	0.86 s	7.40 s
peak memory	0.39 GB	1.33 GB

The warm-JVM saving is roughly fixed, so it is ~25% of quick mode and only ~14% of full mode. Worth saying plainly: this server is much less valuable if full validation is ever made the default.

Memory is the sharper edge. ValidatorWrapper sizes the pairing Bloom filter from readCountLimit / 2, so --full allocates for 50M reads regardless of how small the file actually is, and getCopy() duplicates it per mate. Four concurrent --full validations against -Xmx1g exhausted the heap on three of four. Nothing here changes that sizing - it is upstream behaviour - but it is the reason --threads and the heap have to be chosen together, and the reason the OOM-handling commit exists.

The thing I'd push back on if I were reviewing

/validate returns the exact stdout, stderr and exit code the CLI would have printed, rather than a JSON schema describing the verdict. That looks lazy and it is deliberate: Loculus parses the CLI's text today, so returning anything else creates two ways to phrase the same validation error and a chance for them to drift apart, on a path whose whole job is telling a submitter precisely why their file was rejected. ValidationRunner is the one implementation both entry points call, and ValidateServerTest asserts the server's bytes equal the CLI's for the same input. If you'd rather have a structured response, the clean version is to add fields alongside the text, not to replace it.

The second judgement call: validation concurrency is bounded by a semaphore, and the HTTP thread pool is left unbounded, which is backwards from how these are usually written. With a fixed pool of --threads, a health check queues behind in-flight validations, so a merely busy server fails its liveness probe and gets restarted — and in the Loculus pod that restart also evicts deacon's multi-gigabyte in-memory index, turning a busy minute into a several-minute outage. Threads are cheap; what actually needs bounding is how many validations allocate read buffers at once, which is exactly what the semaphore bounds. There's a test that health stays answerable with zero free slots (measured 50ms worst case under saturation).

Deliberately not in here

The two hot-loop inefficiencies profiling turned up — the autoboxed HashSet<Character> lookup per base in InsdcReadsValidator.validate, and the leading-.* regex run per read name in PairedFastqReadsValidator — are together about 60% of steady-state time and are being worked on separately. They compose with this rather than competing with it: this change removes the warm-up cost, those attack what remains.

Also left alone: the webin-cli-validator:2.+ floating version range. Resolving it hits maven.imagej.net, which returned 503 throughout this work, so a clean build needs --offline against a warm cache right now. Worth pinning, but it isn't this change's business.

Note for whoever merges

ValidateCli keeps its existing behaviour and command line exactly; server is a new first argument. I checked the CLI's output is byte-identical to the released v1.0.0 jar across valid, content-error and structural cases before touching anything else.

Loculus consumes this in loculus-project/loculus#7188. So that PR is actually testable rather than pointing at a tag that does not exist, there is a preview build of this branch at https://github.com/corneliusroemer-agent/readtools/releases/tag/v1.1.0-rc1 and the Loculus Dockerfile pulls from there for now. It is on a personal fork only because the agent account cannot push here; cutting the real release on this repo is a maintainer call and I have not assumed it. Once you do, the Loculus PR is a one-line repoint.