diff --git a/CHANGELOG.md b/CHANGELOG.md index 41f7869..0752541 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,39 @@ grouped by date and milestone rather than semantic version. The content tracks the [readingbat-core](https://github.com/readingbat/readingbat-core) platform, so many entries reflect dependency and toolchain upgrades. +## [1.2.0] - 2026-10-13 + +Dependency upkeep, and the memory fix that the new Kotlin scripting runtime made +necessary. No challenge content changed. + +### Changed +- Bumped `version` in `gradle.properties` to `1.2.0`. + +- Upgraded dependencies: readingbat-core `3.4.0` → `3.5.0`, Kotlin `2.4.10` → `2.4.21`, + common-utils `3.2.2` → `5.1.1`, and the Gradle versions plugin `0.64.0` → `0.66.0`. + Ktor (`3.6.0`), Kotest (`6.2.5`), kotlinter (`5.7.0`), detekt (`2.0.0-alpha.6`), and the + JVM toolchain (**25**) are unchanged. + +- Upgraded the Gradle wrapper `9.7.1` → `9.8.1`, which also regenerates `gradlew.bat`. + +- `gradle.properties` turns the Gradle configuration cache on, and gives the Gradle and + Kotlin daemons a 6 GB heap with `-XX:+HeapDumpOnOutOfMemoryError`. + +- `make versions` now prints only the dependencies with newer versions, any that + could not be checked, and a pending Gradle upgrade, via a custom `outputFormatter` + on `dependencyUpdates` in `build.gradle.kts`. The new `make versions-all` target + passes `--output-formatter=plain` for the plugin's full report, which also lists + every dependency already up to date. + +- Held the runtime Kotlin scripting jars (`kotlin-scripting-*`, `kotlin-compiler-embeddable`, + `kotlin-daemon-embeddable`, `kotlin-script-runtime`) at `2.4.10` through a new + `kotlin-scripting` catalog version, applied by `configureScripting()` to the runtime and + test classpaths only; the project itself builds with Kotlin `2.4.21`. From Kotlin + `2.4.20` on, the JSR-223 engine never disposes a script state's compiler environment, so + each state keeps about 27 MB reachable. readingbat-core `3.5.0`, which brings in the + `2.4.21` scripting jars, resets its pooled engine after every Kotlin challenge, so + `ContentTests` ran out of heap partway through the Kotlin sweep, even with a 2 GB heap. + ## [1.1.0] - 2026-09-21 Test-reliability and dependency upkeep. No challenge content changed. diff --git a/CLAUDE.md b/CLAUDE.md index 8a2c201..756cf40 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,7 +7,7 @@ This is a **ReadingBat content repository** — it defines Java and Kotlin progr ## Conventions Record notable changes in `CHANGELOG.md` under an `[Unreleased]` heading; it becomes the -next version's section at release time. Releases are tagged without a `v` prefix (`1.1.0`), +next version's section at release time. Releases are tagged without a `v` prefix (`1.2.0`), and the version lives in `gradle.properties`. `.gitattributes` governs line endings: stored blobs are LF, `gradlew` stays LF, and @@ -21,6 +21,23 @@ and the version lives in `gradle.properties`. ./gradlew test -Dkotest.filter.tests="" # Filter Kotest cases by name ``` +`gradle.properties` turns the configuration cache on, so build logic has to be +configuration-cache safe. `dependencyUpdates` is marked incompatible, and the `make versions` +targets pass `--no-configuration-cache` for it. + +## Dependencies + +Every version lives in `gradle/libs.versions.toml`. One is deliberately out of step: +`kotlin-scripting` holds the runtime Kotlin scripting jars (`kotlin-scripting-*`, +`kotlin-compiler-embeddable`, `kotlin-daemon-embeddable`, `kotlin-script-runtime`) at 2.4.10 +while `kotlin` moves on, and `configureScripting()` in `build.gradle.kts` applies it to +`runtimeClasspath` and `testRuntimeClasspath` only. From Kotlin 2.4.20 the JSR-223 engine +never disposes a script state's compiler environment (about 27 MB each), and readingbat-core +starts a state for every Kotlin challenge, so without the pin `ContentTests` runs out of heap +in the Kotlin sweep. Leave it in place when upgrading Kotlin. Remove it only after a Kotlin +release fixes the engine or readingbat-core pins the jars itself, and confirm with +`make clean tests`. + ## Testing `src/test/kotlin/ContentTests.kt` sweeps every challenge through a Ktor test host, one diff --git a/Makefile b/Makefile index fcba971..c0525f3 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ .PHONY: default help clean build lint format detekt detekt-baseline tests uberjar uber \ - cc run versions upgrade-wrapper + cc run versions versions-all upgrade-wrapper GRADLE_VERSION := $(shell sed -n 's/^gradle-wrapper = "\(.*\)"/\1/p' gradle/libs.versions.toml) @@ -42,9 +42,17 @@ cc: ## Continuous compilation (watches for changes) run: ## Start the content server locally (port 8080) ./gradlew run -versions: ## Check for dependency updates +# dependencyUpdates is marked incompatible with the configuration cache (build.gradle.kts), +# which gradle.properties turns on, so --no-configuration-cache skips the cache for these +# runs rather than building an entry only to discard it. build.gradle.kts gives the task a +# formatter that lists only what is behind; versions-all names the plugin's own plain +# report, which also lists everything up to date. +versions: ## List the dependencies with newer versions, and any that could not be checked ./gradlew dependencyUpdates --no-configuration-cache --no-parallel +versions-all: ## The plugin's full dependency report, up-to-date dependencies included + ./gradlew dependencyUpdates --no-configuration-cache --no-parallel --output-formatter=plain + upgrade-wrapper: _require-gradle-version ## Upgrade the Gradle wrapper to the catalog version # Gradle's documented upgrade procedure: the first run rewrites # gradle-wrapper.properties using the *old* wrapper jar; the second run diff --git a/README.md b/README.md index 28400a3..ad6f1ff 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,8 @@ make cc # Continuous compilation (watches for changes) make run # Start the content server locally (port 8080) make uberjar # Build fat jar at build/libs/server.jar make uber # Build the uberjar and run it -make versions # Check for dependency updates +make versions # List dependencies with newer versions +make versions-all # Full dependency report, up-to-date dependencies included ``` When running locally, challenge files are loaded directly from disk (via @@ -102,7 +103,11 @@ gradle/libs.versions.toml # Centralized dependency & toolchain versions ## Tech Stack - Kotlin / JVM, Gradle (Kotlin DSL) with a version catalog -- [readingbat-core](https://github.com/readingbat/readingbat-core) — the platform library (from JitPack) +- [readingbat-core](https://github.com/readingbat/readingbat-core) — the platform library (from Maven Central) +- Kotlin's JSR-223 scripting runtime, which computes the Kotlin challenges' answers, is held at + 2.4.10 by the `kotlin-scripting` catalog version while the project builds with a newer Kotlin. + From 2.4.20 the engine never releases a script's compiler environment (about 27 MB each), and + the challenge sweep runs out of heap; see `configureScripting()` in `build.gradle.kts`. - Ktor test host + Kotest — challenge validation - detekt + kotlinter — static analysis and formatting diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 7c44cb1..df4f389 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -8,6 +8,40 @@ changes, see [CHANGELOG.md](CHANGELOG.md). --- +## v1.2.0 (October 2026) + +A dependency release that turned into a memory investigation. Moving to readingbat-core +3.5.0 and Kotlin 2.4.21 made `ContentTests` fail partway through the Kotlin sweep with +`OutOfMemoryError`, and a larger test heap did not help: with 2 GB, full collections +freed almost nothing, and the heap stayed full. + +The cause is a regression in Kotlin's JSR-223 script engine, which readingbat-core uses +to compute each Kotlin challenge's answers. From Kotlin 2.4.20 the engine is built on the +K2 REPL, and every new script state builds a complete compiler environment (a project, +FIR sessions, and an index of the whole classpath) that is never disposed, so each one +stays reachable for the life of the JVM, about 27 MB apiece here. readingbat-core resets +its pooled engine after every challenge, which starts a new state, so the sweep piled up +an environment per Kotlin challenge. The 2.4.10 engine released the same states, and +creating a fresh engine per script leaks just as badly on 2.4.21, so this is Kotlin's, +not the pooling's. + +The fix keeps the runtime scripting jars on 2.4.10 while everything else moves forward. +A new `kotlin-scripting` catalog version is applied by `configureScripting()` to the +runtime and test classpaths only, so the project still compiles with Kotlin 2.4.21 and +challenge code is compiled by the 2.4.10 engine. With the pin in place the full suite +passes in about 30 seconds. The pin is temporary: it can go once a Kotlin release disposes +the state, or once readingbat-core carries the same pin itself. + +Two build changes ship alongside. `make versions` now lists only what is behind (plus +anything it could not check and a pending Gradle upgrade), and `make versions-all` keeps +the versions plugin's full report. `gradle.properties` turns the Gradle configuration +cache on and gives the Gradle and Kotlin daemons a 6 GB heap with heap dumps on +`OutOfMemoryError`. + +Dependencies: readingbat-core 3.5.0, Kotlin 2.4.21 (scripting runtime held at 2.4.10), +common-utils 5.1.1, the versions plugin 0.66.0, and the Gradle wrapper 9.8.1. No +challenge content changed in this release. + ## v1.1.0 (September 2026) A test-reliability release, prompted by a CI failure on a pull request that contained no diff --git a/build.gradle.kts b/build.gradle.kts index 3d1ba48..4ffcd7d 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -38,6 +38,7 @@ tasks.named("build") { } configureKotlin() +configureScripting() configureDetekt() configureKotlinter() configureKtor() @@ -61,6 +62,28 @@ fun Project.configureKotlin() { } } +fun Project.configureScripting() { + // Kotlin 2.4.20 and later leave each JSR-223 script state's compiler environment + // reachable (about 27 MB apiece) once the engine moves to a new one. readingbat-core + // resets its pooled engines after every Kotlin challenge it evaluates, so the + // ContentTests sweep exhausts any test heap. readingbat-core brings in the scripting + // jars at its own Kotlin version; this holds them at one that releases the state. Only + // the runtime classpaths are pinned: the compiler that builds this project is untouched. + val scriptingVersion = libs.versions.kotlin.scripting.get() + val compilerModules = setOf("kotlin-compiler-embeddable", "kotlin-daemon-embeddable", "kotlin-script-runtime") + + configurations.matching { it.name in setOf("runtimeClasspath", "testRuntimeClasspath") }.configureEach { + resolutionStrategy.eachDependency { + if (requested.group == "org.jetbrains.kotlin" && + (requested.name.startsWith("kotlin-scripting-") || requested.name in compilerModules) + ) { + useVersion(scriptingVersion) + because("Kotlin 2.4.20+ scripting retains every script state's compiler environment") + } + } + } +} + fun Project.configureDetekt() { detekt { source.setFrom("src/main/kotlin", "src/test/kotlin") @@ -124,5 +147,23 @@ fun Project.configureVersions() { rejectVersionIf { isNonStable(candidate.version) && !isNonStable(currentVersion) } + + // Prints only what is behind, and what could not be checked. The plugin's own + // report, which `make versions-all` asks for with --output-formatter=plain, first + // lists every dependency that is already up to date. + outputFormatter { + val behind = outdated.dependencies.sortedBy { "${it.group}:${it.name}" } + if (behind.isEmpty()) println("All dependencies are up to date") + behind.forEach { d -> + val latest = with(d.available) { release ?: milestone ?: integration } + println(" - ${d.group}:${d.name} ${d.version} -> $latest") + } + unresolved.dependencies.sortedBy { "${it.group}:${it.name}" }.forEach { + println(" ? ${it.group}:${it.name}:${it.version} could not be checked") + } + if (gradle.current.isUpdateAvailable) { + println(" - Gradle ${gradle.running.version} -> ${gradle.current.version}") + } + } } } diff --git a/gradle.properties b/gradle.properties index ef1db44..dba8557 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,7 +1,8 @@ group=com.readingbat -version=1.1.0 +version=1.2.0 kotlin.code.style=official -org.gradle.caching=true -org.gradle.configuration-cache=false org.gradle.daemon=true +org.gradle.caching=true +org.gradle.configuration-cache=true +org.gradle.jvmargs=-Xmx6g -Dkotlin.daemon.jvm.options=-Xmx6g -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index faccb44..c8e8281 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -1,15 +1,17 @@ [versions] detekt = "2.0.0-alpha.6" -gradle-wrapper = "9.7.1" +gradle-wrapper = "9.8.1" jvm = "25" kotest = "6.2.5" -kotlin = "2.4.10" +kotlin = "2.4.21" +# The Kotlin scripting jars challenges run on; held back from kotlin (see configureScripting) +kotlin-scripting = "2.4.10" kotlinter = "5.7.0" ktor = "3.6.0" logging = "8.0.4" -readingbat = "3.4.0" -utils = "3.2.2" -versions = "0.64.0" +readingbat = "3.5.0" +utils = "5.1.1" +versions = "0.66.0" [libraries] core-utils = { module = "com.pambrose.common-utils:core-utils", version.ref = "utils" } diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar index eddabd2..9fa7f28 100644 Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index ad7845b..2e7bb9f 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,6 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-9.7.1-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-9.8.1-bin.zip networkTimeout=10000 retries=0 retryBackOffMs=500 diff --git a/gradlew.bat b/gradlew.bat index a51ec4f..3185a43 100644 --- a/gradlew.bat +++ b/gradlew.bat @@ -26,6 +26,33 @@ @rem Set local scope for the variables, and ensure extensions are enabled setlocal EnableExtensions +@rem Catch executions from older scripts and ensure they exit cleanly. +@rem This can be removed once we can be reasonably confident that few people +@rem will be migrating directly to this new wrapper. +goto afterSafetyNet +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +goto exitWithErrorLevel +:afterSafetyNet + set DIRNAME=%~dp0 if "%DIRNAME%"=="" set DIRNAME=. @rem This is normally unused @@ -45,13 +72,14 @@ set JAVA_EXE=java.exe %JAVA_EXE% -version >NUL 2>&1 if %ERRORLEVEL% equ 0 goto execute -echo. 1>&2 -echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2 -echo. 1>&2 -echo Please set the JAVA_HOME variable in your environment to match the 1>&2 -echo location of your Java installation. 1>&2 +1>&2 echo. +1>&2 echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +1>&2 echo. +1>&2 echo Please set the JAVA_HOME variable in your environment to match the +1>&2 echo location of your Java installation. "%COMSPEC%" /c exit 1 +goto exitWithErrorLevel :findJavaFromJavaHome set JAVA_HOME=%JAVA_HOME:"=% @@ -59,13 +87,14 @@ set JAVA_EXE=%JAVA_HOME%/bin/java.exe if exist "%JAVA_EXE%" goto execute -echo. 1>&2 -echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2 -echo. 1>&2 -echo Please set the JAVA_HOME variable in your environment to match the 1>&2 -echo location of your Java installation. 1>&2 +1>&2 echo. +1>&2 echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +1>&2 echo. +1>&2 echo Please set the JAVA_HOME variable in your environment to match the +1>&2 echo location of your Java installation. "%COMSPEC%" /c exit 1 +goto exitWithErrorLevel :execute @rem Setup the command line @@ -75,8 +104,9 @@ echo location of your Java installation. 1>&2 @rem Execute gradlew @rem endlocal doesn't take effect until after the line is parsed and variables are expanded @rem which allows us to clear the local environment before executing the java command -endlocal & "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %* & call :exitWithErrorLevel +endlocal & "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %* & call :exitWithErrorLevel & goto exitWithErrorLevel +@rem This label must not be changed. We rely on old scripts being able to jump to this point. :exitWithErrorLevel @rem Use "%COMSPEC%" /c exit to allow operators to work properly in scripts "%COMSPEC%" /c exit %ERRORLEVEL% diff --git a/llms.txt b/llms.txt index 3ede593..7261e71 100644 --- a/llms.txt +++ b/llms.txt @@ -20,7 +20,9 @@ Each challenge is a standalone source file. The `main()` function prints expecte ## Tech Stack - Kotlin / JVM, Gradle (Kotlin DSL) with a version catalog. The JVM toolchain version is centralized in `gradle/libs.versions.toml` under the `jvm` key. -- readingbat-core (platform library, from JitPack) +- readingbat-core (platform library, from Maven Central) +- Kotlin's JSR-223 scripting runtime is pinned to 2.4.10 (`kotlin-scripting` in the version catalog, applied by `configureScripting()` to the runtime and test classpaths only) while the project builds with a newer Kotlin. From 2.4.20 the engine never disposes a script state's compiler environment (~27 MB each), and readingbat-core starts a state per Kotlin challenge, so the test sweep runs out of heap without the pin. +- The Gradle configuration cache is on (`gradle.properties`); `dependencyUpdates` opts out, so the `make versions` targets pass `--no-configuration-cache`. - Kotest + Ktor test host (testing) - detekt + kotlinter (static analysis and formatting, enforced in CI) @@ -35,7 +37,8 @@ Run `make help` for the authoritative target list. - `make run` — start the content server on http://localhost:8080 - `make uberjar` — build fat jar at `build/libs/server.jar` - `make uber` — build uberjar and run it -- `make versions` — check for dependency updates +- `make versions` — list only the dependencies (and Gradle) with newer versions, plus any that could not be checked +- `make versions-all` — the versions plugin's full plain report, up-to-date dependencies included - `./gradlew test -Dkotest.filter.tests=""` — filter Kotest cases by name ## Testing