Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 37 additions & 3 deletions build.gradle.kts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import java.security.MessageDigest

plugins {
kotlin("jvm") version "2.4.10"
kotlin("plugin.serialization") version "2.4.10"
kotlin("jvm") version "2.4.21"
kotlin("plugin.serialization") version "2.4.21"
application
jacoco
}
Expand Down Expand Up @@ -63,6 +63,11 @@ dependencies {
testImplementation("org.jetbrains.pty4j:pty4j:0.13.12")
testImplementation("com.h2database:h2:2.3.232")
testImplementation(platform("org.springframework.boot:spring-boot-dependencies:3.5.16"))
testImplementation(platform("com.fasterxml.jackson:jackson-bom:2.21.7"))
testImplementation(platform("org.apache.logging.log4j:log4j-bom:2.25.5"))
listOf("core", "el", "websocket").forEach { module ->
testImplementation("org.apache.tomcat.embed:tomcat-embed-$module:10.1.61")
}
testImplementation("org.springframework.boot:spring-boot-starter-web")
testImplementation("org.springframework.boot:spring-boot-starter-jdbc")
testImplementation(kotlin("reflect"))
Expand Down Expand Up @@ -195,12 +200,33 @@ tasks.register<JavaExec>("resumeReadCheck") {
args(providers.gradleProperty("resumeReadResults").getOrElse("build/resume-read-results"))
}

tasks.register<JavaExec>("captureContinuationCheck") {
group = "verification"
description = "Measure repeated profile-capture resumes against bounded single-process continuation."
dependsOn(tasks.testClasses)
classpath = sourceSets.test.get().runtimeClasspath
mainClass.set("dev.scouterctl.CaptureContinuationCheckKt")
maxHeapSize = "512m"
args(providers.gradleProperty("continuationResults").getOrElse("build/capture-continuation-results"))
}

tasks.register<JavaExec>("timezoneStorageCheck") {
group = "verification"
description = "Verify reported time zones and DST storage-date limits against an isolated Collector (opt-in)."
dependsOn(tasks.testClasses, "fatJar")
classpath = sourceSets.test.get().runtimeClasspath
mainClass.set("dev.scouterctl.TimezoneStorageCheckKt")
args(providers.gradleProperty("scouterRoot").getOrElse("../scouter"),
providers.gradleProperty("timezoneResults").getOrElse("build/timezone-storage-results"))
systemProperty("cli.jar", layout.buildDirectory.file("libs/scouterctl-${project.version}-all.jar").get().asFile.absolutePath)
}

// Opt-in fixture for matching legacy Collector/Agent releases on Java 8.
val legacyFixture = sourceSets.create("legacyFixture")
configurations.named(legacyFixture.implementationConfigurationName) {
extendsFrom(configurations.implementation.get())
}
dependencies { add(legacyFixture.implementationConfigurationName, "com.h2database:h2:1.4.200") }
dependencies { add(legacyFixture.implementationConfigurationName, "com.h2database:h2:2.2.224") }
tasks.named<org.jetbrains.kotlin.gradle.tasks.KotlinJvmCompile>("compileLegacyFixtureKotlin") {
setSource(files("src/test/kotlin/dev/scouterctl/fixture/TestService.kt"))
compilerOptions.jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_1_8)
Expand Down Expand Up @@ -230,6 +256,14 @@ tasks.register<JavaExec>("agentInspectionSmoke") {
layout.buildDirectory.file("libs/scouterctl-${project.version}-all.jar").get().asFile.absolutePath))
}

tasks.register<JavaExec>("agentDiagnosticsSmoke") {
dependsOn(tasks.testClasses, "fatJar")
classpath = sourceSets.test.get().runtimeClasspath
mainClass.set("dev.scouterctl.AgentSmokeKt")
args(providers.gradleProperty("scouterRoot").getOrElse("../scouter"), "--diagnostics-only")
systemProperty("cli.jar", layout.buildDirectory.file("libs/scouterctl-${project.version}-all.jar").get().asFile.absolutePath)
}


tasks.register<JavaExec>("counterSmoke") {
dependsOn(tasks.testClasses, "fatJar")
Expand Down
4 changes: 3 additions & 1 deletion docs/capture-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ Pages that were already committed are validated before resume. An incomplete pag

Windows can deny an atomic file replacement while another program holds the destination open. The CLI briefly retries access-denied replacements, preserving the committed file and the atomic replacement operation. Persistent denial still returns `IO_ERROR` (exit 10); other filesystem failures are not retried. An access-denied error can also mean insufficient permissions.

The message distinguishes local file I/O from Collector transport I/O. Close programs using the capture and check its path, permissions and available disk space. If the directory contains a valid committed manifest, rerun the original command with `--resume` after correcting the cause. If the first manifest could not be committed, preserve that directory and start a new capture in an unused directory. Do not promote a `.partial` file to a manifest or remove referenced pages to force recovery.
The message distinguishes local file I/O from Collector transport I/O and includes a safe exception category, for example `Local file I/O failed (AccessDeniedException)`. The category identifies the failure reported by Java; it does not establish which process denied access or prove an antivirus cause. Raw paths and operating-system error messages are excluded. Other categories are `NoSuchFileException`, `FileAlreadyExistsException`, `AtomicMoveNotSupportedException`, `FileSystemException`, and `IOException`. Error codes and exit codes are unchanged; scripts should use the error code rather than matching message text.

Close programs using the capture and check its path, permissions and available disk space. If the directory contains a valid committed manifest, rerun the original command with `--resume` after correcting the cause. If the first manifest could not be committed, preserve that directory and start a new capture in an unused directory. Do not promote a `.partial` file to a manifest or remove referenced pages to force recovery.

## Recovery limits

Expand Down
22 changes: 21 additions & 1 deletion docs/capture-resume-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,24 @@ For an ordinary incomplete capture with no unavailable profiles, the existing-pa

Combining the passes halves these prefix scans; it does not eliminate their cumulative growth. With B new pages per invocation and P = mB total pages, ordinary repeated resumes scan Bm(m-1)/2 = (P²/B - P)/2 existing pages. Each individual resume remains linear in the committed prefix. Transaction-ID membership still requires memory proportional to committed records. Collecting through the existing validated callback retains the application's ID set while the store's uniqueness set is live, increasing simultaneous set storage; peak heap was not measured here.

The CLI still allows `--max-pages` 1..40 and profile page sizes up to 100, with the existing capture limit of 10,000 pages. A bounded single-process continuation could retain IDs between batches and avoid repeated prefix validation, but it would need an explicit total-work budget, cancellation/checkpoint behavior and rules for on-disk changes between batches. Changing the current per-invocation budget contract is outside this fix. Persisting an unchecked ID cache or skipping validation on later invocations would weaken integrity guarantees and is not a substitute.
The default invocation still allows `--max-pages` 1..40 and profile page sizes up to 100, with the existing capture limit of 10,000 pages. Repeated separate invocations retain the cumulative cost above.

## Bounded single-process continuation

Profile captures can opt into `--max-total-pages N` to continue beyond the ordinary `--max-pages` budget in the same process. N must be between `--max-pages` and 10,000 and counts **new pages in this invocation**, including after `--resume`; it is not the final capture size. Defaults and behavior without this option are unchanged. This option requires `capture --include-profiles`.

```sh
scouterctl capture --include-profiles --resume --directory captures/incident \
--date 20261101 --from 2026-11-01T00:00:00Z --to 2026-11-01T01:00:00Z \
--timezone UTC --object-hash 42 --max-total-pages 1000 --no-input
```

The process validates the existing prefix once, retains transaction IDs in memory and checkpoints each new page before requesting the next. Reaching the budget without finishing returns the ordinary incomplete capture result (exit 8). Resume still fully validates all committed pages; no persistent unchecked ID cache or format change is introduced. Profile recovery can reread pages whose profiles were unavailable.

Profile writers hold an exclusive `.writer.lock` for the invocation: a file lock on Unix and an exclusive file handle on Windows, including Windows access to WSL UNC paths where byte-range locking is unavailable. Concurrent writers using the same platform fail with `IO_ERROR`; readers remain available. Do not mix Windows and Linux writers on a WSL-shared directory: these platform mechanisms do not establish cross-platform coordination. A changed or removed manifest is detected before checkpoint replacement and is not overwritten. Atomic moves and advisory locks do not defend against an external process editing immutable page files, replacing lock files, or racing the final compare-and-replace. Do not modify a live capture directory. Stop the writer and use a fresh full validation/resume after external edits. This restriction also applies between ordinary page-budget boundaries inside a continued invocation.

Cancellation is checked before each page and profile request and before committing a new page. A cooperative cancellation reports `PARTIAL_RESULT` with the last committed cursor intact; retry with `--resume`. On terminal termination the CLI allows up to five seconds for in-flight work to notice cancellation and release its writer lock. An uninterruptible network call or forced termination can outlive that grace period; already committed pages remain recoverable and unreferenced partial/page files are ignored. Cancellation does not guarantee upstream cancellation.

ID storage is still O(committed records). Continuation removes repeated prefix scans between separately launched invocations, but it does not bound record memory independently of capture size. Choose a finite total budget for the available heap and workload.

Run `./gradlew captureContinuationCheck -PcontinuationResults=NEW_DIRECTORY` for a file-backed comparison of repeated 40-page resumes and one continued invocation after a common 40-page seed. It measures 120 and 400 pages with 100 records and 20 structural SQL steps per profile, under a 512 MiB heap. The JSON report includes positive existing-page read bytes from JFR, measured capture elapsed time, per-heap-pool peak usage and post-operation heap usage. Pool peaks can occur at different instants, include measurement overhead and are not simultaneous peak heap or process RSS. GC timing and shared-JVM warmup affect memory and time observations; no universal speedup or memory reduction follows from these synthetic local runs.
2 changes: 2 additions & 0 deletions docs/cli-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ A consumer should accept the stream schema, process records, then require both t

Online window queries reject offset mismatches against the reported Collector JVM zone before streaming or writing captures. `doctor` and `collector status` expose the property's ID and a client-rule offset with explicit provenance; unavailable metadata remains `null`. These checks do not verify the storage calendar. See [Collector time zones](collector-timezone.md).

`xlog search` and `counter history` repeat `timezoneCheck` in both `start` and `complete`; `capture` and `counter export` include it in the outer result envelope. Its `status` is `reported-offset-match` when the interval comparison succeeds, or `unavailable` when no usable Collector zone was supplied. `collectorTimezone` contains the reported ID and client-derived offset with provenance, or `null`. All four commands retain `storageDateVerification: "unverified"`: neither a reported-zone match nor exit 0 proves that an empty result means no data existed. Saved manifest formats remain unchanged.

`counter history` uses its own `scouterctl.counter-stream/v1` schema with the same start/record/complete lifecycle. Its start includes the dated target and counter metadata, and records contain stored timestamps and numeric strings. See [counter history](counter-history.md) for missing values, units and query limits.

`counter export` returns a single manifest result and writes JSONL chunk files. Its `scouterctl.counter-capture/v1` format is separate from XLog captures. See [resumable counter export](counter-export.md) for budgets, checkpoints and completion semantics.
Expand Down
23 changes: 23 additions & 0 deletions docs/collector-timezone.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,31 @@ scouterctl xlog search --object-hash 42 --date 20261109 --timezone Asia/Seoul \

The same instants declared as `America/Los_Angeles` with date `20261108` are rejected. A UTC timestamp suffix is still valid input with `--timezone Asia/Seoul`; the timestamp syntax itself is not required to use the Collector's offset.

## Query verification in live output

Window queries include a `timezoneCheck` object in live output: the `start` and `complete` events of `xlog search` and `counter history`, and the outer result envelope of `capture` and `counter export`.

```json
{"timezoneCheck":{"status":"unavailable","collectorTimezone":null},"storageDateVerification":"unverified"}
```

`status: "reported-offset-match"` means the declared zone matched the reported JVM zone throughout the requested interval under the client's rules. `collectorTimezone` contains the same ID, reference time, offset and provenance fields as `doctor`; its offset is for that reference time, not necessarily the historical query. `status: "unavailable"` means the Collector supplied no usable zone metadata, so no comparison was possible. Transport and malformed-protocol failures still fail the command instead of producing this status.

The check is repeated in the terminal stream event so a consumer retaining only the completion receipt can distinguish the two states. Neither state verifies the storage calendar, and `storageDateVerification` remains `"unverified"` for all four commands. Treat an empty successful result as a completed extraction from an unverified partition, not proof that no traffic or counter samples existed. These are additive live fields; saved manifest schemas and `timezoneVerification: "operator-declared"` do not change.

## Storage dates remain unverified

This is a comparison with a **reported JVM zone**, not confirmation of the storage partition. XLog `storageDateVerification: "unverified"` and capture `timezoneVerification: "operator-declared"` remain unchanged. Capture formats and strict-reader compatibility are unchanged; the new diagnostic fields are live output only.

Scouter's `DateTimeHelper` in 2.8.1 and 2.21.3 builds a calendar from local `20000101` with fixed 86,400,000-ms days. That calendar can differ from civil time around DST. For example, in `America/Chicago`, `2026-07-01T05:30:00Z` is July 1 by current zone rules but June 30 in that storage calendar. A matching zone check cannot certify the correct partition or prove that an empty result means no traffic. Confirm storage dates with the Collector operator when results are unexpected. This change does not repair that upstream calendar behavior or verify the server's cached calendar through the protocol.

## Reproduce the storage-calendar check

With a built Scouter 2.21.3 checkout used read-only and an installed Java 8 runtime:

```sh
SCOUTER_COLLECTOR_JAVA=/path/to/java8/bin/java ./gradlew timezoneStorageCheck \
-PscouterRoot=/path/to/scouter -PtimezoneResults=build/timezone-storage-trial
```

The result directory must not exist. The opt-in check starts isolated UTC and `America/Chicago` Collectors, writes synthetic XLogs at the same fixed instant, reads the stored partition, and runs the packaged CLI. It retains JSON output, command arguments, Collector logs and teardown receipts. It stops its processes and removes temporary account files. This verifies the 2.21.3 calendar behavior with synthetic telemetry, not every supported Collector or Agent combination.
20 changes: 20 additions & 0 deletions docs/inspection.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,26 @@ Both commands go through the selected Collector. Status includes the Agent name,

Configuration lists the Agent's runtime settings and defaults. Visible settings include selected numeric and boolean controls, such as `trace_sql_parameter_max_length`, `profile_http_parameter_enabled` and `profile_http_querystring_enabled`, plus the text settings `net_collector_ip`, `obj_name` and `obj_type`. Other settings report null for current and default values with `redacted: true`, including when selected by `--key`. Neither command writes configuration. An unknown target, an offline Agent's configuration, unavailable configuration or a missing setting returns exit 7. Malformed responses return exit 6. The CLI remains on JDK 21 even when the Agent/application runs on Java 8.

## Active requests and thread dumps

Use one nonzero Java Agent hash from `object list`:

```sh
scouterctl agent active-requests --object-hash 123456
scouterctl agent active-requests --object-hash 123456 --include-text
scouterctl agent thread-dump --object-hash 123456
```

`active-requests` takes one live snapshot. It returns `collector`, `agent`, `requests`, `complete` and `textIncluded`. Each request has a decimal-string `txid` and `threadId`, the Agent's thread `state`, `elapsedMs` and decimal-string `threadCpuTimeMs`. CPU time is the thread's cumulative CPU time in milliseconds, not CPU used by that request; some Agents return zero when measurement is unavailable. A reported -1 is represented as null. Deferred or reactive requests may have thread ID `"0"` and state `"n/a"`. Rows are ordered by elapsed time descending. Only requests currently represented by the Agent are included; this is not a historical search or a count of all application activity.

Service and thread names are null unless `--include-text` is supplied. Even with that option, SQL statements, client IP addresses, login names and arbitrary request descriptions are excluded. The decimal `txid` can be used with `profile get` after the request finishes and reaches Collector storage. A complete, empty snapshot succeeds; a snapshot explicitly reported incomplete returns the rows with `executionComplete: false` and exit 8.

`thread-dump` explicitly requests raw diagnostic text and returns it in `data.lines`, with Collector and Agent identity. Thread names, stack traces, lock details and other JVM-produced text can contain sensitive data; review the output before sharing. No remote dump file or thread-control command is requested. The Agent/JVM may require attachment support; Java 9+ attachment-based Agents can require `-Djdk.attach.allowAttachSelf=true` at application startup. The CLI does not change that setting. Agent-side failures return a generic diagnostic without echoing raw attachment errors or local paths.

Dump `completeness` is `"unknown"`: the upstream protocol does not provide a reliable truncation indicator and some implementations cap output at 100,000 lines. Command `executionComplete` means the returned response was processed, not that every thread or frame was captured. The CLI rejects over 100,000 lines or 8,388,608 decoded characters, and active snapshots over 10,000 rows. These are post-decode checks, not bounds on Agent work or total transport allocation. A timeout does not guarantee upstream cancellation.

Unknown/offline targets, missing diagnostic replies and unavailable JVM dumps exit 7 (`NOT_FOUND`); malformed responses exit 6 (`PROTOCOL_ERROR`). The protocol cannot reliably distinguish an unsupported diagnostic from an Agent that disconnected after discovery. The commands use the existing Collector account, whose `effectivePermission` remains `unknown`; successful authentication does not establish per-operation authorization. Platform, JVM and Agent combinations need separate validation from the general Collector version list.

## Real-time counters

Use a target hash from `object list`:
Expand Down
Loading
Loading