From 926381334e9cf9fb9ec018e62aa19a3cfce9da52 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Mon, 9 Nov 2026 17:08:44 +0900 Subject: [PATCH 01/17] =?UTF-8?q?docs:=20=F0=9F=93=9D=20Document=20issue-b?= =?UTF-8?q?ased=20branch=20names?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document issue-based branch names and issue references in both contributor guides and pull request templates. --- .github/PULL_REQUEST_TEMPLATE/english.md | 4 +++- .github/PULL_REQUEST_TEMPLATE/korean.md | 4 +++- CONTRIBUTING.ko.md | 8 ++++++-- CONTRIBUTING.md | 8 ++++++-- 4 files changed, 18 insertions(+), 6 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE/english.md b/.github/PULL_REQUEST_TEMPLATE/english.md index 483a069..4d2fbbc 100644 --- a/.github/PULL_REQUEST_TEMPLATE/english.md +++ b/.github/PULL_REQUEST_TEMPLATE/english.md @@ -6,7 +6,9 @@ See CONTRIBUTING.md for PR targets and merge methods. ## Why - + ## Changes diff --git a/.github/PULL_REQUEST_TEMPLATE/korean.md b/.github/PULL_REQUEST_TEMPLATE/korean.md index cdddf2e..31ac354 100644 --- a/.github/PULL_REQUEST_TEMPLATE/korean.md +++ b/.github/PULL_REQUEST_TEMPLATE/korean.md @@ -6,7 +6,9 @@ PR 대상 브랜치와 병합 방식은 CONTRIBUTING.md를 참고해 주세요. ## 변경 이유 - + ## 주요 변경 diff --git a/CONTRIBUTING.ko.md b/CONTRIBUTING.ko.md index 2701853..56db909 100644 --- a/CONTRIBUTING.ko.md +++ b/CONTRIBUTING.ko.md @@ -37,10 +37,14 @@ Linux 명령 ZIP의 빌드·검증: | 변경 | 시작 브랜치 | PR 대상 | | --- | --- | --- | -| 기능·일반 수정·테스트·문서 (`feature/*`) | `develop` | `develop` | -| 긴급 배포 수정 (`hotfix/*`) | `master` | `master` | +| 기능·일반 수정·테스트·문서 (`feature/<이슈번호>[-<설명>]`) | `develop` | `develop` | +| 긴급 배포 수정 (`hotfix/<이슈번호>[-<설명>]`) | `master` | `master` | | 배포할 변경 통합 | `develop` | `master` | +`feature/59`처럼 `#`을 뺀 이슈 번호를 사용합니다. 작업을 알아보기 쉽도록 `feature/59-object-list`처럼 소문자와 하이픈으로 짧은 설명을 붙이는 것을 권장하며, 설명은 생략할 수 있습니다. 연결할 이슈가 없는 소규모 유지보수는 `feature/<설명>`을 사용합니다. Dependabot과 release-please는 자동 생성된 브랜치명을 사용합니다. + +`develop` 대상 PR에는 `Refs #59`로 이슈를 참조합니다. 해결분을 포함한 `master` 대상 PR에는 `Closes #59`를 적어 병합 시 이슈가 닫히도록 합니다. + `master`는 기본·배포 브랜치이고 `develop`은 다음 변경을 통합합니다. 별도 release 브랜치는 현재 사용하지 않습니다. PR에는 문제·변경 후 동작·실제로 수행한 검증·기존 사용자에게 미치는 영향을 적습니다. 비교할 브랜치를 고른 뒤 기본 PR 설명의 **Preview** 탭에서 **English** 또는 **한국어**를 선택하세요. 언어를 바꾸면 양식이 다시 열리므로 본문 작성 전에 선택합니다. GitHub가 작성 중인 내용을 복원하면 언어 링크를 새 탭에서 열고, 브랜치를 바꿨다면 언어를 다시 선택하세요. 양식은 기본 브랜치에서 불러옵니다. 로컬 템플릿도 사용할 수 있습니다. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7df5123..bcee0e2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -37,10 +37,14 @@ For a local Linux installation, use `./gradlew installDist` and run `build/insta | Change | Start from | PR target | | --- | --- | --- | -| Feature, regular fix, tests or documentation (`feature/*`) | `develop` | `develop` | -| Urgent release fix (`hotfix/*`) | `master` | `master` | +| Feature, regular fix, tests or documentation (`feature/[-]`) | `develop` | `develop` | +| Urgent release fix (`hotfix/[-]`) | `master` | `master` | | Release-ready integration | `develop` | `master` | +Use the issue number without `#`, as in `feature/59`. A short lowercase, hyphen-separated description is optional and recommended for readability, as in `feature/59-object-list`. For small maintenance changes without an issue, use `feature/`. Dependabot and release-please use their generated branch names. + +Reference the issue with `Refs #59` in a PR to `develop`. When the fix is included in a PR to `master`, use `Closes #59` to close the issue on merge. + `master` is the default and release branch; `develop` integrates upcoming changes. Release branches are not currently used. Keep each PR focused and explain the problem, resulting behavior, validation actually performed, and compatibility risks. Choose the base and compare branches before selecting **English** or **한국어** from the default PR description's **Preview** tab. Switching languages reloads the form, so select a language before writing the description. If GitHub restores a draft, open the language link in a new tab; reselect it after changing branches. Templates are loaded from the default branch. Alternatively, use a local template: From 8d0cbf5c04ff218f463859847fdc3c86fccaed34 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Mon, 9 Nov 2026 17:56:06 +0900 Subject: [PATCH 02/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Report=20unverified?= =?UTF-8?q?=20XLog=20storage=20dates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Report the verification limit in XLog search and capture output. Refs #57. --- QUICKSTART.ko.md | 2 ++ QUICKSTART.md | 2 ++ docs/cli-contract.md | 6 ++++-- .../kotlin/dev/scouterctl/cli/JsonOutput.kt | 6 +++++- .../dev/scouterctl/cli/OnlineCommands.kt | 4 ++-- .../dev/scouterctl/CliOutputContractTest.kt | 21 +++++++++++++++++++ 6 files changed, 36 insertions(+), 5 deletions(-) diff --git a/QUICKSTART.ko.md b/QUICKSTART.ko.md index 92e845d..2e83694 100644 --- a/QUICKSTART.ko.md +++ b/QUICKSTART.ko.md @@ -49,6 +49,8 @@ scouterctl capture --object-hash 123456 --date 20261107 \ --include-profiles --directory captures/incident-001 --no-input ``` +캡처 결과의 `storageDateVerification: "unverified"`는 입력한 저장 날짜를 Collector에서 확인하지 않았다는 뜻입니다. 조회가 완료됐는데 결과가 비어 있으면 저장 날짜를 확인하세요. + 중단되거나 일부만 저장됐으면 같은 명령에 `--resume`을 붙여 재개합니다. 서비스명·SQL·오류 원문을 저장하려는 경우에만 캡처할 때 `--include-text`를 추가하세요. 민감한 애플리케이션 정보가 포함될 수 있습니다. 자세한 사용법은 [캡처와 오프라인 분석(영문)](docs/captures.md)에 있습니다. ## 4. Collector 없이 분석 diff --git a/QUICKSTART.md b/QUICKSTART.md index 8421c01..03040e3 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -47,6 +47,8 @@ scouterctl capture --object-hash 123456 --date 20261107 \ --include-profiles --directory captures/incident-001 --no-input ``` +The capture result reports `storageDateVerification: "unverified"`. Check the supplied storage date when a completed query returns no records. + Repeat the same command with `--resume` after interruption or partial completion. Add `--include-text` at capture time only if you intend to store potentially sensitive service names, SQL and error text. See [capture and offline analysis](docs/captures.md). ## 4. Inspect without a Collector diff --git a/docs/cli-contract.md b/docs/cli-contract.md index 03b2e3e..b03bf95 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -51,7 +51,7 @@ A mid-response EOF currently maps to `IO_ERROR` (10), whereas socket exceptions/ `xlog search` uses `scouterctl.stream/v1` on its first `start` event: ```text -{"type":"start","schemaVersion":"scouterctl.stream/v1","source":{...},"query":{...}} +{"type":"start","schemaVersion":"scouterctl.stream/v1","source":{...},"query":{...},"storageDateVerification":"unverified"} {"type":"record","data":{...}} {"type":"complete","executionComplete":true,"upstreamCoverage":"unknown","sampling":"unknown"} ``` @@ -62,7 +62,9 @@ A successful stream has exactly one final `complete` event and exit 0. Failure a A consumer should accept the stream schema, process records, then require both the final `complete` event and exit 0 before marking the query complete. Use `capture` and its checkpoints when resumable extraction is needed. -`counter history` uses its own `scouterctl.counter-stream/v1` schema with the same start/record/complete lifecycle. Its start includes the dated target and counter metadata, and records contain stored timestamps and numeric strings. See [counter history](counter-history.md) for missing values, units and query limits. XLog stream fields and schemas are unchanged. +`storageDateVerification: "unverified"` means the CLI used the supplied XLog storage date without confirmation from the Collector. An incorrect date can produce an empty completed query. XLog `capture` reports the same field in its outer result envelope. + +`counter history` uses its own `scouterctl.counter-stream/v1` schema with the same start/record/complete lifecycle. Its start includes the dated target and counter metadata, and records contain stored timestamps and numeric strings. See [counter history](counter-history.md) for missing values, units and query limits. `counter export` returns a single manifest result and writes JSONL chunk files. Its `scouterctl.counter-capture/v1` format is separate from XLog captures. See [resumable counter export](counter-export.md) for budgets, checkpoints and completion semantics. diff --git a/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt b/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt index fa89008..17d9bed 100644 --- a/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt +++ b/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt @@ -10,6 +10,8 @@ import java.io.PrintStream import java.net.SocketException import java.net.SocketTimeoutException +private const val STORAGE_DATE_UNVERIFIED = "unverified" + class JsonOutput(private val stream: PrintStream) { fun run(operation: () -> Int): Int { try { @@ -32,12 +34,13 @@ class JsonOutput(private val stream: PrintStream) { return failure(OperationFailure(ErrorCode.PROTOCOL_ERROR, "Response could not be decoded")) } } - fun result(data: JsonElement, complete: Boolean = true): Int { + fun result(data: JsonElement, complete: Boolean = true, xlogCapture: Boolean = false): Int { emit(buildJsonObject { put("schemaVersion", "scouterctl.result/v1") put("ok", true) put("data", data) put("executionComplete", complete) + if (xlogCapture) put("storageDateVerification", STORAGE_DATE_UNVERIFIED) }) return if (complete) 0 else 8 } @@ -48,6 +51,7 @@ class JsonOutput(private val stream: PrintStream) { put("schemaVersion", "scouterctl.stream/v1") put("source", json.encodeToJsonElement(event.source.toDto())) put("query", json.encodeToJsonElement(event.query.toDto())) + put("storageDateVerification", STORAGE_DATE_UNVERIFIED) }) is XLogEvent.Record -> emit(buildJsonObject { put("type", "record") diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 9780d7e..1a50c1d 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -56,10 +56,10 @@ class OnlineCommands( val options = CaptureOptions(request.resume, request.maxPages, "${endpoint.host}:${endpoint.port}", request.includeText) return if (request.includeProfiles) { val manifest = captureEvidence(source, request.query, options, evidence(request.directory)) - output.result(json.encodeToJsonElement(manifest.toDto()), manifest.complete) + output.result(json.encodeToJsonElement(manifest.toDto()), manifest.complete, xlogCapture = true) } else { val manifest = capture(source, request.query, options, captures(request.directory)) - output.result(json.encodeToJsonElement(manifest.toDto()), manifest.executionComplete) + output.result(json.encodeToJsonElement(manifest.toDto()), manifest.executionComplete, xlogCapture = true) } } } diff --git a/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt b/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt index 2719784..d800969 100644 --- a/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt +++ b/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt @@ -4,6 +4,7 @@ import io.kotest.core.spec.style.FunSpec import io.kotest.engine.spec.tempdir import io.kotest.matchers.shouldBe import kotlinx.serialization.json.* +import java.nio.file.Files class CliOutputContractTest : FunSpec({ test("single-result envelopes distinguish available data from complete execution") { @@ -34,6 +35,25 @@ class CliOutputContractTest : FunSpec({ } } + test("XLog capture reports the unverified date outside the persisted manifest") { + val root = tempdir().toPath() + val cli = ContractCli(root) + val window = listOf("--date", "20261101", "--from", "2026-11-01T00:00:00Z", + "--to", "2026-11-01T00:01:00Z", "--timezone", "UTC", "--object-hash", "42") + WireFixture().use { source -> + for (includeProfiles in listOf(false, true)) { + val destination = root.resolve(if (includeProfiles) "profiles" else "xlogs") + val capture = listOf("capture", "--directory", destination.toString()) + window + val result = cli.run(source, if (includeProfiles) capture + "--include-profiles" else capture) + result.exit shouldBe 0 + result.events.single()["storageDateVerification"] shouldBe JsonPrimitive("unverified") + val manifest = Json.parseToJsonElement(Files.readString(destination.resolve("manifest.json"))).jsonObject + result.data() shouldBe manifest + manifest.containsKey("storageDateVerification") shouldBe false + } + } + } + test("parser and command failures use the same machine-readable error envelope") { val cli = ContractCli(tempdir().toPath()) for (args in listOf(listOf("doctor", "--unknown-option"), listOf("raw", "call"))) { @@ -62,6 +82,7 @@ class CliOutputContractTest : FunSpec({ result.stderr shouldBe "" result.events.first()["schemaVersion"] shouldBe JsonPrimitive("scouterctl.stream/v1") result.events.first()["type"] shouldBe JsonPrimitive("start") + result.events.first()["storageDateVerification"] shouldBe JsonPrimitive("unverified") val records = result.events.drop(1).dropLast(1) records.size shouldBe if (limited) 2 else 3 records.forEachIndexed { index, record -> From 2ad5f496750a4b452d1f2847f7582e90fa97925d Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Mon, 9 Nov 2026 19:58:05 +0900 Subject: [PATCH 03/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Clarify=20sensitive?= =?UTF-8?q?=20values=20in=20profile=20error=20text?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify the profile warning for error text that can contain sensitive values. Refs #63. --- .../kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index 64292d9..da5e521 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -279,7 +279,7 @@ class CollectorSource private constructor(private val socket: Socket, private va transaction = if (transaction == null) ProfileTransaction("unavailable") else ProfileTransaction( "available", DictionaryText(transaction.service, service?.get(transaction.service)), if (transaction.error == 0) null else DictionaryText(transaction.error, errors[transaction.error])), - detail = "service, SQL and error dictionary text included; bind fields and other free text omitted; dictionary text may contain sensitive values", + detail = "Service, SQL and error dictionary text included. Bind fields and other free text omitted. Dictionary text may contain sensitive values. Error text can include bound values or stored row values.", ) } From 8a25b30ac581de685a09f8832856aca25b67ea26 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:11:07 +0900 Subject: [PATCH 04/17] =?UTF-8?q?feat:=20=E2=9C=A8=20List=20objects=20by?= =?UTF-8?q?=20Collector=20storage=20date?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expose stored object hashes through object list --date. Refs #59. --- QUICKSTART.ko.md | 2 ++ QUICKSTART.md | 2 ++ .../adapters/collector/CollectorSource.kt | 11 ++++-- .../collector/CounterHistoryDecoder.kt | 35 +++++++++++-------- .../dev/scouterctl/application/Ports.kt | 2 +- src/main/kotlin/dev/scouterctl/cli/Cli.kt | 2 +- .../dev/scouterctl/cli/CommandInterpreter.kt | 10 ++++-- .../dev/scouterctl/cli/CommandRequest.kt | 2 +- .../dev/scouterctl/cli/OnlineCommands.kt | 2 +- .../dev/scouterctl/CliOutputContractTest.kt | 23 ++++++++++++ src/test/kotlin/dev/scouterctl/MemoryPorts.kt | 2 +- 11 files changed, 68 insertions(+), 25 deletions(-) diff --git a/QUICKSTART.ko.md b/QUICKSTART.ko.md index 2e83694..84a24fa 100644 --- a/QUICKSTART.ko.md +++ b/QUICKSTART.ko.md @@ -40,6 +40,8 @@ scouterctl counter get --object-hash 123456 --counter HeapUsed --no-input ## 3. 트랜잭션과 프로파일 저장 +Collector를 재시작한 뒤에도 `scouterctl object list --date 20261107 --no-input`으로 해당 저장 날짜에 기록된 `objHash`를 찾을 수 있습니다. 저장된 목록에는 실시간 상태가 없어 날짜를 지정하면 `alive: false`로 표시합니다. 현재 상태는 날짜를 빼고 `object list`로 확인하세요. + 조회할 자료가 저장된 날짜와 Collector의 시간대를 사용합니다. 아래는 UTC 예시입니다. `captures`가 없으면 먼저 만들고, 캡처마다 새 하위 디렉터리를 지정하세요. ```sh diff --git a/QUICKSTART.md b/QUICKSTART.md index 03040e3..fd7e011 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -38,6 +38,8 @@ See [status, settings and current counters](docs/inspection.md) for available fi ## 3. Save transactions and profiles +Use `scouterctl object list --date 20261107 --no-input` to find an `objHash` recorded on that storage date, even after a Collector restart. Dated results use `alive: false` because the stored list has no live status. Use `object list` without a date to check current status. + Use the Collector's storage date and timezone. This example assumes UTC. Create the parent `captures` directory if it does not exist, and use a new capture directory: ```sh diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index da5e521..cb174f4 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -141,8 +141,7 @@ class CollectorSource private constructor(private val socket: Socket, private va } override fun counterHistory(query: CounterHistoryQuery): CounterHistory = collectorOperation { - val targets = exchange(RequestCmd.OBJECT_LIST_LOAD_DATE, MapPack().apply { put("date", query.date) }) - val target = decodeCounterTarget(query.objHash, targets) + val target = decodeCounterTarget(query.objHash, datedObjects(query.date)) val metadata = exchange(RequestCmd.GET_XML_COUNTER) val response = exchange(RequestCmd.COUNTER_PAST_TIME, MapPack().apply { put("objHash", query.objHash.toLong()); put("counter", query.counter) @@ -197,7 +196,13 @@ class CollectorSource private constructor(private val socket: Socket, private va } } - override fun objects(): List = collectorOperation { + private fun datedObjects(date: String): List = + decodeDatedObjects(exchange(RequestCmd.OBJECT_LIST_LOAD_DATE, MapPack().apply { put("date", date) })) + + override fun objects(date: String?): List = collectorOperation { + if (date != null) return@collectorOperation datedObjects(date).map { + ObjectRecord(info.serverId, it.objHash, it.objName, it.objType, alive = false) + } exchange(RequestCmd.OBJECT_LIST_REAL_TIME).map { val p = it as? ObjectPack ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Expected object pack") ObjectRecord(info.serverId, p.objHash, p.objName ?: "", p.objType ?: "", p.alive) diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CounterHistoryDecoder.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CounterHistoryDecoder.kt index 721e844..4db2fee 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CounterHistoryDecoder.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CounterHistoryDecoder.kt @@ -5,22 +5,29 @@ import scouter.lang.pack.MapPack import scouter.lang.pack.Pack import scouter.lang.value.* -internal fun decodeCounterTarget(objHash: Int, targets: List): CounterTarget { - val objects = targets.singleOrNull() as? MapPack ?: invalidHistory() - val hashes = objects.get("objHash") as? ListValue ?: invalidHistory() - val names = objects.get("objName") as? ListValue ?: invalidHistory() - val types = objects.get("objType") as? ListValue ?: invalidHistory() - if (hashes.size() > 10000 || names.size() != hashes.size() || types.size() != hashes.size()) invalidHistory() - val matches = (0 until hashes.size()).map { index -> - val hash = (hashes.get(index) as? DecimalValue)?.value ?: invalidHistory() - if (hash !in Int.MIN_VALUE..Int.MAX_VALUE) invalidHistory() - val name = (names.get(index) as? TextValue)?.value ?: invalidHistory() - val type = (types.get(index) as? TextValue)?.value ?: invalidHistory() +internal fun decodeDatedObjects(response: List): List { + val objects = response.singleOrNull() as? MapPack ?: invalidDatedObjects() + val hashes = objects.get("objHash") as? ListValue ?: invalidDatedObjects() + val names = objects.get("objName") as? ListValue ?: invalidDatedObjects() + val types = objects.get("objType") as? ListValue ?: invalidDatedObjects() + if (hashes.size() > 10000 || names.size() != hashes.size() || types.size() != hashes.size()) invalidDatedObjects() + return (0 until hashes.size()).map { index -> + val hash = (hashes.get(index) as? DecimalValue)?.value ?: invalidDatedObjects() + if (hash !in Int.MIN_VALUE..Int.MAX_VALUE) invalidDatedObjects() + val name = (names.get(index) as? TextValue)?.value ?: invalidDatedObjects() + val type = (types.get(index) as? TextValue)?.value ?: invalidDatedObjects() CounterTarget(hash.toInt(), name, type) - }.filter { it.objHash == objHash } + } +} + +private fun invalidDatedObjects(): Nothing = + throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid dated object list response") + +internal fun decodeCounterTarget(objHash: Int, targets: List): CounterTarget { + val matches = targets.filter { it.objHash == objHash } if (matches.size > 1) invalidHistory() - val target = matches.singleOrNull() ?: throw OperationFailure(ErrorCode.NOT_FOUND, "Object was not found on the requested date") - return target + return matches.singleOrNull() + ?: throw OperationFailure(ErrorCode.NOT_FOUND, "Object was not found on the requested date") } internal fun decodeCounterHistory(query: CounterHistoryQuery, target: CounterTarget, metadata: List, response: List): CounterHistory { diff --git a/src/main/kotlin/dev/scouterctl/application/Ports.kt b/src/main/kotlin/dev/scouterctl/application/Ports.kt index cd055f2..00b356b 100644 --- a/src/main/kotlin/dev/scouterctl/application/Ports.kt +++ b/src/main/kotlin/dev/scouterctl/application/Ports.kt @@ -10,7 +10,7 @@ interface ObservationSource : AutoCloseable { fun agentSettings(objHash: Int): AgentConfiguration fun counterHistory(query: CounterHistoryQuery): CounterHistory fun realtimeCounters(objHash: Int): RealtimeCounters - fun objects(): List + fun objects(date: String? = null): List fun page(query: Query, cursor: Cursor): XLogPage fun profile(date: String, txid: Long, includeText: Boolean = false): ProfileRecord } diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index bb62066..10f85ab 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -24,7 +24,7 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut @Option(names = ["--key"], description = ["collector config / agent config: exact setting name"]) var settingKey: String? = null @Option(names = ["--chunk-minutes"], description = ["counter export: 1..1440; default 60, split at Collector midnight"]) var chunkMinutes: Int? = null @Option(names = ["--max-chunks"], description = ["counter export: chunks per invocation; default 24"]) var maxChunks: Int? = null - @Option(names = ["--date"]) var date: String? = null + @Option(names = ["--date"], description = ["Collector storage date (YYYYMMDD). For object list, selects recorded objects."]) var date: String? = null @Option(names = ["--from"]) var from: String? = null @Option(names = ["--to"]) var to: String? = null @Option(names = ["--timezone"], description = ["Operator-declared collector IANA timezone"] ) var timezone: String? = null diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt index 388861f..d0c3b3e 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt @@ -109,15 +109,14 @@ private fun Cli.onlineRequest(command: String): OnlineRequest { if (maxPages !in 1..40) invalid("Use --max-pages 1..40") val query = if (command in setOf("xlog search", "capture")) query() else null if (command == "profile get") { - if (!required(date, "--date").matches(Regex("[0-9]{8}"))) invalid("Use an 8-digit storage date") - LocalDate.parse(required(date, "--date"), DateTimeFormatter.BASIC_ISO_DATE) + validateStorageDate(required(date, "--date")) required(txid, "--txid") } if (command == "capture") required(directory, "--directory") if (includeProfiles && required(query, "query").pageSize > 100) invalid("Profile capture page-size must be 1..100") return when (command) { "doctor", "auth login" -> OnlineRequest.Doctor(command == "auth login") - "object list" -> OnlineRequest.Objects + "object list" -> OnlineRequest.Objects(date?.also(::validateStorageDate)) "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText) "xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages) "capture" -> OnlineRequest.Capture( @@ -127,6 +126,11 @@ private fun Cli.onlineRequest(command: String): OnlineRequest { } } +private fun validateStorageDate(date: String) { + if (!date.matches(Regex("[0-9]{8}"))) invalid("Use an 8-digit storage date") + LocalDate.parse(date, DateTimeFormatter.BASIC_ISO_DATE) +} + private fun Cli.query(): Query = Query(required(date, "--date"), Instant.parse(required(from, "--from")).toEpochMilli(), Instant.parse(required(to, "--to")).toEpochMilli(), required(timezone, "--timezone"), required(objHash, "--object-hash"), pageSize ?: if (includeProfiles) 25 else 1000).also { it.validate() } diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt index a42cfc2..e4cfc80 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt @@ -25,7 +25,7 @@ sealed interface OnlineRequest : CommandRequest { data class CounterExport(val query: CounterExportQuery, val directory: Path, val resume: Boolean, val maxChunks: Int) : OnlineRequest data class CounterHistory(val query: CounterHistoryQuery) : OnlineRequest data class Counters(val objHash: Int, val name: String?) : OnlineRequest - data object Objects : OnlineRequest + data class Objects(val date: String?) : OnlineRequest data class Profile(val date: String, val txid: Long, val includeText: Boolean) : OnlineRequest data class Search(val query: Query, val maxPages: Int) : OnlineRequest data class Capture(val query: Query, val directory: Path, val resume: Boolean, diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 1a50c1d..0995b76 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -37,7 +37,7 @@ class OnlineCommands( 0 } is OnlineRequest.Counters -> output.result(json.encodeToJsonElement(realtimeCounters(observation, request.objHash, request.name).toDto(observation.info))) - OnlineRequest.Objects -> output.result(json.encodeToJsonElement(observation.objects().map { it.toDto() })) + is OnlineRequest.Objects -> output.result(json.encodeToJsonElement(observation.objects(request.date).map { it.toDto() })) is OnlineRequest.Profile -> profile(observation, request) is OnlineRequest.Search -> { streamXLogs(observation, request.query, request.maxPages, output::event) diff --git a/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt b/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt index d800969..cea4685 100644 --- a/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt +++ b/src/test/kotlin/dev/scouterctl/CliOutputContractTest.kt @@ -35,6 +35,29 @@ class CliOutputContractTest : FunSpec({ } } + test("object list with a storage date returns the recorded objects in the usual result shape") { + val cli = ContractCli(tempdir().toPath()) + WireFixture().use { source -> + val result = cli.run(source, listOf("object", "list", "--date", "20260101")) + result.exit shouldBe 0 + result.stderr shouldBe "" + result.events.single()["data"] shouldBe Json.parseToJsonElement( + """[{"serverId":"fixture","objHash":42,"objName":"/historical-fixture","objType":"java","alive":false}]""", + ) + } + } + + test("object list rejects invalid storage dates before connecting") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(Fault.AUTH).use { source -> + for (date in listOf("2026-01-01", "20260229")) { + val result = cli.run(source, listOf("object", "list", "--date", date)) + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["code"] shouldBe JsonPrimitive("INVALID_ARGUMENT") + } + } + } + test("XLog capture reports the unverified date outside the persisted manifest") { val root = tempdir().toPath() val cli = ContractCli(root) diff --git a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt index 5aa4e3a..c42acf4 100644 --- a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt +++ b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt @@ -46,7 +46,7 @@ internal class MemorySource( override fun agentSettings(objHash: Int): AgentConfiguration = throw OperationFailure(ErrorCode.NOT_FOUND, "No Agent in memory fixture") override fun counterHistory(query: CounterHistoryQuery): CounterHistory = throw OperationFailure(ErrorCode.NOT_FOUND, "No history in memory fixture") override fun realtimeCounters(objHash: Int): RealtimeCounters = throw OperationFailure(ErrorCode.NOT_FOUND, "No Agent in memory fixture") - override fun objects(): List = emptyList() + override fun objects(date: String?): List = emptyList() override fun page(query: Query, cursor: Cursor) = fetch(cursor) override fun profile(date: String, txid: Long, includeText: Boolean) = profile(txid) override fun close() { closed = true } From 08a59fd0e2ae4ad3f884a16483dba26ce051c3f0 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:42:42 +0900 Subject: [PATCH 05/17] =?UTF-8?q?feat:=20=E2=9C=A8=20Resolve=20XLog=20dict?= =?UTF-8?q?ionary=20text=20on=20request?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expose stored service and error dictionary text through the existing XLog search opt-in. Refs #60. --- docs/captures.md | 2 + docs/cli-contract.md | 2 + .../adapters/collector/CollectorSource.kt | 8 ++- .../dev/scouterctl/application/Ports.kt | 2 +- .../dev/scouterctl/application/StreamXLogs.kt | 17 ++++-- src/main/kotlin/dev/scouterctl/cli/Cli.kt | 2 +- .../dev/scouterctl/cli/CommandInterpreter.kt | 6 +- .../dev/scouterctl/cli/CommandRequest.kt | 2 +- .../kotlin/dev/scouterctl/cli/JsonOutput.kt | 10 +++- .../dev/scouterctl/cli/OnlineCommands.kt | 2 +- .../kotlin/dev/scouterctl/core/Observation.kt | 6 +- src/test/kotlin/dev/scouterctl/MemoryPorts.kt | 2 +- src/test/kotlin/dev/scouterctl/WireFixture.kt | 3 +- .../kotlin/dev/scouterctl/XLogTextTest.kt | 60 +++++++++++++++++++ 14 files changed, 107 insertions(+), 17 deletions(-) create mode 100644 src/test/kotlin/dev/scouterctl/XLogTextTest.kt diff --git a/docs/captures.md b/docs/captures.md index ca285bf..a0416cf 100644 --- a/docs/captures.md +++ b/docs/captures.md @@ -27,6 +27,8 @@ scouterctl capture \ --include-profiles --directory captures/incident-001 ``` +Add `--include-text` to `xlog search` to resolve service names and error text without downloading profiles. Error text can contain sensitive values, including bound values or values from existing database rows. + The destination directory must not exist for a new capture. If the export is interrupted or partially completes, add `--resume` to the same capture command. Each query covers one date and one object over `[from,to)`. Set `--timezone` to the Collector's actual time zone. The example saves profiles in a v3 bundle. Add `--include-text` to the capture command only when you intend to store potentially sensitive service, SQL and error text; the default page size is 25 and the maximum is 100. Resume v3 with the same flags and query. Existing v2 bundles remain readable but cannot be resumed by this version; see [Capture compatibility](capture-compatibility.md). Missing profiles are recorded and retried on resume; connection failures preserve earlier page checkpoints. See [Capture recovery](capture-recovery.md) for interruption handling and the limits of recovery. diff --git a/docs/cli-contract.md b/docs/cli-contract.md index b03bf95..3e6cfbb 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -58,6 +58,8 @@ A mid-response EOF currently maps to `IO_ERROR` (10), whereas socket exceptions/ The abbreviated objects illustrate event shape, not executable JSON fixtures. There are zero or more `record` events. `record` and `complete` events do not repeat `schemaVersion`; they inherit the stream schema from `start`. +With `xlog search --include-text`, each record adds `service` and `error` dictionary fields alongside the existing hashes. Dictionary entries have `hash`, `text`, and `status`. Missing entries have `status: "unresolved"` and `text: null`. A zero error hash yields `error: null`. Both dictionary fields are absent by default. Error text may contain sensitive values. + A successful stream has exactly one final `complete` event and exit 0. Failure after streaming starts ends with the `scouterctl.error/v1` error object and a nonzero exit, without `complete`. Failure before streaming starts may emit only the error object. A page budget produces `PARTIAL_RESULT` and exit 8; earlier records must not be mistaken for the full query. A killed process or failed output transport may leave no terminal event. A consumer should accept the stream schema, process records, then require both the final `complete` event and exit 0 before marking the query complete. Use `capture` and its checkpoints when resumable extraction is needed. diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index cb174f4..4d11f71 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -209,7 +209,7 @@ class CollectorSource private constructor(private val socket: Socket, private va } } - override fun page(query: Query, cursor: Cursor): XLogPage = collectorOperation { + override fun page(query: Query, cursor: Cursor, includeText: Boolean): XLogPage = collectorOperation { query.validate() val request = MapPack().apply { put("date", query.date); put("stime", query.from); put("etime", query.to - 1) @@ -237,7 +237,11 @@ class CollectorSource private constructor(private val socket: Socket, private va if (more && (records.isEmpty() || next.bucketTime == 0L || next.txid == "0" || next == cursor)) { throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Non-advancing XLog cursor") } - XLogPage(records, next, more) + val text = if (includeText) XLogPageText( + dictionary(query.date, TextTypes.SERVICE, records.map { it.serviceHash }), + dictionary(query.date, TextTypes.ERROR, records.map { it.errorHash }.filter { it != 0 }), + ) else null + XLogPage(records, next, more, text) } override fun profile(date: String, txid: Long, includeText: Boolean): ProfileRecord = collectorOperation { diff --git a/src/main/kotlin/dev/scouterctl/application/Ports.kt b/src/main/kotlin/dev/scouterctl/application/Ports.kt index 00b356b..d961f76 100644 --- a/src/main/kotlin/dev/scouterctl/application/Ports.kt +++ b/src/main/kotlin/dev/scouterctl/application/Ports.kt @@ -11,7 +11,7 @@ interface ObservationSource : AutoCloseable { fun counterHistory(query: CounterHistoryQuery): CounterHistory fun realtimeCounters(objHash: Int): RealtimeCounters fun objects(date: String? = null): List - fun page(query: Query, cursor: Cursor): XLogPage + fun page(query: Query, cursor: Cursor, includeText: Boolean = false): XLogPage fun profile(date: String, txid: Long, includeText: Boolean = false): ProfileRecord } diff --git a/src/main/kotlin/dev/scouterctl/application/StreamXLogs.kt b/src/main/kotlin/dev/scouterctl/application/StreamXLogs.kt index 7677228..226f8ae 100644 --- a/src/main/kotlin/dev/scouterctl/application/StreamXLogs.kt +++ b/src/main/kotlin/dev/scouterctl/application/StreamXLogs.kt @@ -4,16 +4,25 @@ import dev.scouterctl.core.* sealed interface XLogEvent { data class Start(val source: ServerInfo, val query: Query) : XLogEvent - data class Record(val value: XLogRecord) : XLogEvent + data class Record(val value: XLogRecord, val text: Text?) : XLogEvent + data class Text(val service: DictionaryText, val error: DictionaryText?) data object Complete : XLogEvent } -fun streamXLogs(source: ObservationSource, query: Query, maxPages: Int, consume: (XLogEvent) -> Unit) { +fun streamXLogs(source: ObservationSource, query: Query, maxPages: Int, includeText: Boolean, consume: (XLogEvent) -> Unit) { consume(XLogEvent.Start(source.info, query)) var cursor = Cursor() repeat(maxPages) { - val page = source.page(query, cursor) - page.records.forEach { consume(XLogEvent.Record(it)) } + val page = source.page(query, cursor, includeText) + page.records.forEach { row -> + val text = page.text?.let { + XLogEvent.Text( + DictionaryText(row.serviceHash, it.services[row.serviceHash]), + if (row.errorHash == 0) null else DictionaryText(row.errorHash, it.errors[row.errorHash]), + ) + } + consume(XLogEvent.Record(row, text)) + } cursor = page.cursor if (!page.hasMore) { consume(XLogEvent.Complete) diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index 10f85ab..b1814fb 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -37,7 +37,7 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut @Option(names = ["--after"], description = ["inspect compare: candidate v2/v3 capture directory"]) var after: Path? = null @Option(names = ["--resume"]) var resume: Boolean = false @Option(names = ["--include-profiles"], description = ["Capture profiles in a v3 offline bundle"]) var includeProfiles: Boolean = false - @Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in profile get, profile capture or inspect"]) + @Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in xlog search, profile get, profile capture or inspect"]) var includeText: Boolean = false @Option(names = ["--limit"], description = ["inspect search/summary/compare: 1..1000 requests/services; default 20"]) var limit: Int? = null @Option(names = ["--sort"], description = ["inspect summary: errors, elapsed (maximum), requests; default errors"]) var summarySort: String? = null diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt index d0c3b3e..df30597 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt @@ -100,8 +100,8 @@ private fun Cli.validateGeneralOptions(command: String) { invalid("Search filters require inspect search") } if (includeProfiles && command != "capture") invalid("--include-profiles requires capture") - if (includeText && command !in setOf("profile get", "inspect", "inspect search") && !(command == "capture" && includeProfiles)) { - invalid("--include-text requires profile get, inspect or capture --include-profiles") + if (includeText && command !in setOf("xlog search", "profile get", "inspect", "inspect search") && !(command == "capture" && includeProfiles)) { + invalid("--include-text requires xlog search, profile get, inspect or capture --include-profiles") } } @@ -118,7 +118,7 @@ private fun Cli.onlineRequest(command: String): OnlineRequest { "doctor", "auth login" -> OnlineRequest.Doctor(command == "auth login") "object list" -> OnlineRequest.Objects(date?.also(::validateStorageDate)) "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText) - "xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages) + "xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages, includeText) "capture" -> OnlineRequest.Capture( required(query, "query"), required(directory, "--directory"), resume, maxPages, includeProfiles, includeText, ) diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt index e4cfc80..29519c9 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt @@ -27,7 +27,7 @@ sealed interface OnlineRequest : CommandRequest { data class Counters(val objHash: Int, val name: String?) : OnlineRequest data class Objects(val date: String?) : OnlineRequest data class Profile(val date: String, val txid: Long, val includeText: Boolean) : OnlineRequest - data class Search(val query: Query, val maxPages: Int) : OnlineRequest + data class Search(val query: Query, val maxPages: Int, val includeText: Boolean = false) : OnlineRequest data class Capture(val query: Query, val directory: Path, val resume: Boolean, val maxPages: Int, val includeProfiles: Boolean, val includeText: Boolean) : OnlineRequest } diff --git a/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt b/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt index 17d9bed..cecb845 100644 --- a/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt +++ b/src/main/kotlin/dev/scouterctl/cli/JsonOutput.kt @@ -55,7 +55,15 @@ class JsonOutput(private val stream: PrintStream) { }) is XLogEvent.Record -> emit(buildJsonObject { put("type", "record") - put("data", json.encodeToJsonElement(event.value.toDto())) + val record = json.encodeToJsonElement(event.value.toDto()).jsonObject + val data = event.text?.let { text -> + buildJsonObject { + record.forEach { (key, value) -> put(key, value) } + put("service", json.encodeToJsonElement(text.service.toDto())) + put("error", json.encodeToJsonElement(text.error?.toDto())) + } + } ?: record + put("data", data) }) XLogEvent.Complete -> emit(buildJsonObject { put("type", "complete") diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 0995b76..ebf3f28 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -40,7 +40,7 @@ class OnlineCommands( is OnlineRequest.Objects -> output.result(json.encodeToJsonElement(observation.objects(request.date).map { it.toDto() })) is OnlineRequest.Profile -> profile(observation, request) is OnlineRequest.Search -> { - streamXLogs(observation, request.query, request.maxPages, output::event) + streamXLogs(observation, request.query, request.maxPages, request.includeText, output::event) 0 } is OnlineRequest.Capture -> capture(observation, endpoint, request) diff --git a/src/main/kotlin/dev/scouterctl/core/Observation.kt b/src/main/kotlin/dev/scouterctl/core/Observation.kt index f73ac5f..689d3cb 100644 --- a/src/main/kotlin/dev/scouterctl/core/Observation.kt +++ b/src/main/kotlin/dev/scouterctl/core/Observation.kt @@ -62,4 +62,8 @@ data class Query(val date: String, val from: Long, val to: Long, val timezone: S } } -data class XLogPage(val records: List, val cursor: Cursor, val hasMore: Boolean) +data class XLogPageText(val services: Map, val errors: Map) + +data class XLogPage( + val records: List, val cursor: Cursor, val hasMore: Boolean, val text: XLogPageText? = null, +) diff --git a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt index c42acf4..c94ae63 100644 --- a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt +++ b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt @@ -47,7 +47,7 @@ internal class MemorySource( override fun counterHistory(query: CounterHistoryQuery): CounterHistory = throw OperationFailure(ErrorCode.NOT_FOUND, "No history in memory fixture") override fun realtimeCounters(objHash: Int): RealtimeCounters = throw OperationFailure(ErrorCode.NOT_FOUND, "No Agent in memory fixture") override fun objects(date: String?): List = emptyList() - override fun page(query: Query, cursor: Cursor) = fetch(cursor) + override fun page(query: Query, cursor: Cursor, includeText: Boolean) = fetch(cursor) override fun profile(date: String, txid: Long, includeText: Boolean) = profile(txid) override fun close() { closed = true } } diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index 74e4995..c661752 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -32,7 +32,7 @@ enum class Fault { } class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPassword: String? = null, - private val collectorVersion: String = "2.21.3 test") : AutoCloseable { + private val collectorVersion: String = "2.21.3 test", private val xlogErrorHash: Int = 0) : AutoCloseable { private val server = ServerSocket(0, 8, InetAddress.getLoopbackAddress()) val port: Int = server.localPort private val executor = Executors.newSingleThreadExecutor() @@ -241,6 +241,7 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas for (index in start until end) send(output, XLogPack().apply { this.txid = ids[index]; gxid = Long.MIN_VALUE + 1; caller = 9007199254740993L objHash = 42; endTime = baseTime + index; elapsed = 30; service = 10 + if (index == 0) error = xlogErrorHash text1 = "secret-token"; ipaddr = byteArrayOf(127, 0, 0, 1) }) if (fault != Fault.MISSING_META) send(output, MapPack().apply { diff --git a/src/test/kotlin/dev/scouterctl/XLogTextTest.kt b/src/test/kotlin/dev/scouterctl/XLogTextTest.kt new file mode 100644 index 0000000..d27d3b0 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/XLogTextTest.kt @@ -0,0 +1,60 @@ +package dev.scouterctl + +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.* + +class XLogTextTest : FunSpec({ + val search = listOf("xlog", "search", "--date", "20261101", + "--from", "2026-11-01T00:00:00Z", "--to", "2026-11-01T00:01:00Z", + "--timezone", "UTC", "--object-hash", "42", "--page-size", "2") + + test("XLog search omits dictionary fields and avoids dictionary requests by default") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(Fault.STRUCTURAL_ONLY, xlogErrorHash = -789).use { source -> + val result = cli.run(source, search) + result.exit shouldBe 0 + val row = result.events.first { it["type"] == JsonPrimitive("record") }["data"]!!.jsonObject + row["serviceHash"] shouldBe JsonPrimitive(10) + row["errorHash"] shouldBe JsonPrimitive(-789) + row.containsKey("service") shouldBe false + row.containsKey("error") shouldBe false + result.output shouldNotContain "secret-token" + } + } + + for (fault in listOf(Fault.NONE, Fault.EMPTY_TEXT)) { + test("XLog text opt-in preserves record order and reports dictionary availability with $fault") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(fault, xlogErrorHash = -789).use { source -> + val result = cli.run(source, search + "--include-text") + result.exit shouldBe 0 + result.stderr shouldBe "" + result.events.last()["type"] shouldBe JsonPrimitive("complete") + val rows = result.events.filter { it["type"] == JsonPrimitive("record") }.map { it["data"]!!.jsonObject } + rows.map { it["txid"] } shouldBe WireFixture.ids.map { JsonPrimitive(it.toString()) } + val service = if (fault == Fault.NONE) + """{"hash":10,"text":"/fixture","status":"resolved"}""" + else """{"hash":10,"text":null,"status":"unresolved"}""" + rows.map { it["service"] } shouldBe List(rows.size) { Json.parseToJsonElement(service) } + val error = if (fault == Fault.NONE) + """{"hash":-789,"text":"fixture duplicate key error","status":"resolved"}""" + else """{"hash":-789,"text":null,"status":"unresolved"}""" + rows.map { it["error"] } shouldBe listOf(Json.parseToJsonElement(error), JsonNull, JsonNull) + result.output shouldNotContain "secret-token" + } + } + } + + test("A malformed XLog dictionary reply terminates the stream with a protocol error") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(Fault.TEXT_WRONG_HASH, xlogErrorHash = -789).use { source -> + val result = cli.run(source, search + "--include-text") + result.exit shouldBe 6 + result.events.map { it["type"] } shouldBe listOf(JsonPrimitive("start"), JsonPrimitive("error")) + result.events.last()["error"]!!.jsonObject["code"] shouldBe JsonPrimitive("PROTOCOL_ERROR") + } + } +}) From 672dacf7ddb9d4d6c8aa6a8357ab0ec8db3931dd Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Tue, 10 Nov 2026 00:05:16 +0900 Subject: [PATCH 06/17] =?UTF-8?q?feat:=20=E2=9C=A8=20Show=20timing=20and?= =?UTF-8?q?=20network=20outcomes=20in=20live=20profiles?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add recorded timing, parent steps and network outcomes to live profiles. Refs #61. --- docs/captures.md | 2 + .../adapters/collector/CollectorSource.kt | 13 ++-- .../adapters/collector/ProfileStepDecoder.kt | 32 +++++++++ .../adapters/serialization/LiveProfileJson.kt | 25 +++++++ .../dev/scouterctl/application/Ports.kt | 2 +- .../dev/scouterctl/cli/OnlineCommands.kt | 4 +- .../kotlin/dev/scouterctl/core/Observation.kt | 11 ++- src/test/kotlin/dev/scouterctl/AgentSmoke.kt | 2 +- .../kotlin/dev/scouterctl/CliContractTest.kt | 7 +- src/test/kotlin/dev/scouterctl/MemoryPorts.kt | 2 +- .../kotlin/dev/scouterctl/ProfileTextTest.kt | 7 +- .../dev/scouterctl/ProfileTimingTest.kt | 70 +++++++++++++++++++ .../dev/scouterctl/SpringOfflineCheck.kt | 2 +- .../dev/scouterctl/SpringProfileTextCheck.kt | 5 +- src/test/kotlin/dev/scouterctl/SpringSmoke.kt | 2 +- src/test/kotlin/dev/scouterctl/WireFixture.kt | 21 +++++- 16 files changed, 181 insertions(+), 26 deletions(-) create mode 100644 src/main/kotlin/dev/scouterctl/adapters/collector/ProfileStepDecoder.kt create mode 100644 src/main/kotlin/dev/scouterctl/adapters/serialization/LiveProfileJson.kt create mode 100644 src/test/kotlin/dev/scouterctl/ProfileTimingTest.kt diff --git a/docs/captures.md b/docs/captures.md index a0416cf..026761b 100644 --- a/docs/captures.md +++ b/docs/captures.md @@ -33,6 +33,8 @@ The destination directory must not exist for a new capture. If the export is int The example saves profiles in a v3 bundle. Add `--include-text` to the capture command only when you intend to store potentially sensitive service, SQL and error text; the default page size is 25 and the maximum is 100. Resume v3 with the same flags and query. Existing v2 bundles remain readable but cannot be resumed by this version; see [Capture compatibility](capture-compatibility.md). Missing profiles are recorded and retried on resume; connection failures preserve earlier page checkpoints. See [Capture recovery](capture-recovery.md) for interruption handling and the limits of recovery. +Live `profile get` steps include `startTimeMs` relative to the transaction start and the recorded `parent` step index when the Collector provides them. Compare step start times to identify gaps. `SOCKET`, `APICALL` and `APICALL2` steps also report `elapsedMs` and `errorHash` (zero means no recorded error). When `--include-text` is set, `target` can include the recorded address and socket port. Treat those addresses as sensitive. Capture files retain the existing v3 step layout. + Profile results include `completeness`: `truncated` means the Collector explicitly reported omitted blocks; `unknown` does not certify completeness. Known truncation makes `profile get` and `inspect` return exit 8 with the available data. See [Profile completeness](profile-completeness.md), including compatibility with older capture readers. Inspect a saved transaction after disconnecting from the Collector: diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index 4d11f71..3d99f30 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -11,8 +11,6 @@ import scouter.lang.pack.* import scouter.lang.TextTypes import scouter.lang.step.Step import scouter.lang.step.StepControl -import scouter.lang.step.SqlStep -import scouter.lang.step.SqlStep3 import scouter.lang.value.BooleanValue import scouter.lang.value.ListValue import scouter.lang.value.TextValue @@ -244,7 +242,7 @@ class CollectorSource private constructor(private val socket: Socket, private va XLogPage(records, next, more, text) } - override fun profile(date: String, txid: Long, includeText: Boolean): ProfileRecord = collectorOperation { + override fun profile(date: String, txid: Long, includeText: Boolean, includeStepDetails: Boolean): ProfileRecord = collectorOperation { val request = MapPack().apply { put("date", date); put("txid", txid); put("max", 1000) } val packs = exchange(RequestCmd.TRANX_PROFILE, request) if (packs.isEmpty()) throw OperationFailure(ErrorCode.NOT_FOUND, "Profile unavailable") @@ -253,9 +251,7 @@ class CollectorSource private constructor(private val socket: Socket, private va val p = it as? XLogProfilePack ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Expected profile pack") (Step.toObjects(p.profile) ?: emptyArray()).map { step -> if (step is StepControl && step.code == 0 && step.message == " ** Profile Truncated ** ") truncated = true - val sql = if (step is SqlStep) SqlProfile(step.hash, step.elapsed, step.error, - (step as? SqlStep3)?.updated) else null - ProfileStep(step.stepTypeName, step.order, sql) + step.toProfileStep(includeStepDetails = includeStepDetails, includeText = includeText) } } val profile = ProfileRecord(txid.toString(), steps, completeness = @@ -280,6 +276,9 @@ class CollectorSource private constructor(private val socket: Socket, private va val errors = dictionary(date, TextTypes.ERROR, (sql.map { it.errorHash } + listOfNotNull(transaction?.error)).filter { it != 0 }) val service = transaction?.let { dictionary(date, TextTypes.SERVICE, listOf(it.service)) } + val targetNotice = if (profile.steps.any { it.details?.target != null }) + " Socket and API call targets are included when recorded and may contain sensitive addresses." + else "" return profile.copy( steps = profile.steps.map { step -> step.copy(sql = step.sql?.let { it.copy(statement = DictionaryText(it.hash, statements[it.hash]), @@ -288,7 +287,7 @@ class CollectorSource private constructor(private val socket: Socket, private va transaction = if (transaction == null) ProfileTransaction("unavailable") else ProfileTransaction( "available", DictionaryText(transaction.service, service?.get(transaction.service)), if (transaction.error == 0) null else DictionaryText(transaction.error, errors[transaction.error])), - detail = "Service, SQL and error dictionary text included. Bind fields and other free text omitted. Dictionary text may contain sensitive values. Error text can include bound values or stored row values.", + detail = "Service, SQL and error dictionary text included.$targetNotice Bind fields and other free text omitted. Dictionary text may contain sensitive values. Error text can include bound values or stored row values.", ) } diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/ProfileStepDecoder.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/ProfileStepDecoder.kt new file mode 100644 index 0000000..dd6ccef --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/ProfileStepDecoder.kt @@ -0,0 +1,32 @@ +package dev.scouterctl.adapters.collector + +import dev.scouterctl.core.ProfileStep +import dev.scouterctl.core.ProfileStepDetails +import dev.scouterctl.core.ProfileTarget +import dev.scouterctl.core.SqlProfile +import scouter.lang.step.ApiCallStep +import scouter.lang.step.SocketStep +import scouter.lang.step.SqlStep +import scouter.lang.step.SqlStep3 +import scouter.lang.step.Step +import scouter.lang.step.StepSingle +import java.net.InetAddress + +internal fun Step.toProfileStep(includeStepDetails: Boolean, includeText: Boolean): ProfileStep { + val sql = if (this is SqlStep) SqlProfile(hash, elapsed, error, (this as? SqlStep3)?.updated) else null + val details = if (includeStepDetails && this is StepSingle) details(includeText) else null + return ProfileStep(stepTypeName, order, sql, details) +} + +private fun StepSingle.details(includeText: Boolean): ProfileStepDetails = when (this) { + is SocketStep -> ProfileStepDetails(start_time, parent, elapsed, error, if (includeText) target() else null) + is ApiCallStep -> ProfileStepDetails(start_time, parent, elapsed, error, + if (includeText) address?.takeIf { it.isNotBlank() }?.let { ProfileTarget(it) } else null) + else -> ProfileStepDetails(start_time, parent) +} + +private fun SocketStep.target(): ProfileTarget? { + val address = ipaddr?.takeIf { it.size == 4 || it.size == 16 }?.let { InetAddress.getByAddress(it).hostAddress } + val targetPort = port.takeIf { it in 1..65535 } + return if (address == null && targetPort == null) null else ProfileTarget(address, targetPort) +} diff --git a/src/main/kotlin/dev/scouterctl/adapters/serialization/LiveProfileJson.kt b/src/main/kotlin/dev/scouterctl/adapters/serialization/LiveProfileJson.kt new file mode 100644 index 0000000..8be20b0 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/serialization/LiveProfileJson.kt @@ -0,0 +1,25 @@ +package dev.scouterctl.adapters.serialization + +import dev.scouterctl.core.ProfileRecord +import kotlinx.serialization.json.* + +internal fun ProfileRecord.toLiveJson(): JsonObject { + val record = json.encodeToJsonElement(toDto()).jsonObject + val liveSteps = record.getValue("steps").jsonArray.mapIndexed { index, step -> + val details = steps[index].details ?: return@mapIndexed step + buildJsonObject { + step.jsonObject.forEach { (key, value) -> put(key, value) } + put("startTimeMs", details.startTimeMs) + put("parent", details.parent) + details.elapsedMs?.let { put("elapsedMs", it) } + details.errorHash?.let { put("errorHash", it) } + details.target?.let { target -> + putJsonObject("target") { + target.address?.let { put("address", it) } + target.port?.let { put("port", it) } + } + } + } + } + return JsonObject(record + ("steps" to JsonArray(liveSteps))) +} diff --git a/src/main/kotlin/dev/scouterctl/application/Ports.kt b/src/main/kotlin/dev/scouterctl/application/Ports.kt index d961f76..448e184 100644 --- a/src/main/kotlin/dev/scouterctl/application/Ports.kt +++ b/src/main/kotlin/dev/scouterctl/application/Ports.kt @@ -12,7 +12,7 @@ interface ObservationSource : AutoCloseable { fun realtimeCounters(objHash: Int): RealtimeCounters fun objects(date: String? = null): List fun page(query: Query, cursor: Cursor, includeText: Boolean = false): XLogPage - fun profile(date: String, txid: Long, includeText: Boolean = false): ProfileRecord + fun profile(date: String, txid: Long, includeText: Boolean = false, includeStepDetails: Boolean = false): ProfileRecord } interface CaptureRepository { diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index ebf3f28..053ae70 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -48,8 +48,8 @@ class OnlineCommands( } } private fun profile(source: ObservationSource, request: OnlineRequest.Profile): Int { - val profile = source.profile(request.date, request.txid, request.includeText) - return output.result(json.encodeToJsonElement(profile.toDto()), profile.completeness != ProfileCompleteness.TRUNCATED) + val profile = source.profile(request.date, request.txid, request.includeText, includeStepDetails = true) + return output.result(profile.toLiveJson(), profile.completeness != ProfileCompleteness.TRUNCATED) } private fun capture(source: ObservationSource, endpoint: Endpoint, request: OnlineRequest.Capture): Int { diff --git a/src/main/kotlin/dev/scouterctl/core/Observation.kt b/src/main/kotlin/dev/scouterctl/core/Observation.kt index 689d3cb..8413fee 100644 --- a/src/main/kotlin/dev/scouterctl/core/Observation.kt +++ b/src/main/kotlin/dev/scouterctl/core/Observation.kt @@ -32,7 +32,16 @@ data class ProfileTransaction( val status: String, val service: DictionaryText? = null, val error: DictionaryText? = null, ) -data class ProfileStep(val type: String, val order: Int, val sql: SqlProfile? = null) +data class ProfileTarget(val address: String?, val port: Int? = null) + +data class ProfileStepDetails( + val startTimeMs: Int, val parent: Int, val elapsedMs: Int? = null, val errorHash: Int? = null, + val target: ProfileTarget? = null, +) + +data class ProfileStep( + val type: String, val order: Int, val sql: SqlProfile? = null, val details: ProfileStepDetails? = null, +) enum class ProfileCompleteness { UNKNOWN, TRUNCATED } diff --git a/src/test/kotlin/dev/scouterctl/AgentSmoke.kt b/src/test/kotlin/dev/scouterctl/AgentSmoke.kt index d760c7c..c62d47f 100644 --- a/src/test/kotlin/dev/scouterctl/AgentSmoke.kt +++ b/src/test/kotlin/dev/scouterctl/AgentSmoke.kt @@ -199,7 +199,7 @@ fun main(args: Array) { for (route in expected.keys) { val row = rows.first { it.serviceHash == HashUtil.hash(route) } val event = cli.run("profile-${route.substringAfterLast('/')}", listOf("profile", "get", "--date", date, "--txid", row.txid)).single() - val profile = json.decodeFromJsonElement(event["data"]!!).toModel() + val profile = decodeLiveProfile(event["data"]!!) check(profile.steps.any { it.type.startsWith("METHOD") }) { "Missing method instrumentation: $route" } if (route == "/poc/sql") check(profile.steps.any { it.type.startsWith("SQL") }) { "Missing JDBC instrumentation" } } diff --git a/src/test/kotlin/dev/scouterctl/CliContractTest.kt b/src/test/kotlin/dev/scouterctl/CliContractTest.kt index 33f77a4..58d468a 100644 --- a/src/test/kotlin/dev/scouterctl/CliContractTest.kt +++ b/src/test/kotlin/dev/scouterctl/CliContractTest.kt @@ -28,7 +28,7 @@ class CliContractTest : FunSpec({ listOf(doctor, objects, profile).forEach { it.exit shouldBe 0 } doctor.data()["group"]?.jsonPrimitive?.content shouldBe "guest" objects.events.single()["data"]!!.jsonArray.single().jsonObject["objName"]!!.jsonPrimitive.content shouldBe "/fixture" - val decoded = json.decodeFromJsonElement(profile.data()).toModel() + val decoded = decodeLiveProfile(profile.data()) decoded.steps shouldContainExactly listOf(ProfileStep("MESSAGE", 1), ProfileStep("SQL3", 2, SqlProfile(123, 7, 0, 1)), ProfileStep("SQL3", 3, SqlProfile(456, 11, -789, -3))) @@ -124,6 +124,11 @@ class CliContractTest : FunSpec({ } }) +private val liveProfileJson = Json(json) { ignoreUnknownKeys = true } + +internal fun decodeLiveProfile(data: JsonElement): ProfileRecord = + liveProfileJson.decodeFromJsonElement(data).toModel() + internal class ContractCli(private val temp: Path) { fun run(server: WireFixture, args: List): Outcome = run(args + listOf("--port", server.port.toString(), "--user", "guest"), fixture = server::diagnostics) diff --git a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt index c94ae63..33e2e08 100644 --- a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt +++ b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt @@ -48,7 +48,7 @@ internal class MemorySource( override fun realtimeCounters(objHash: Int): RealtimeCounters = throw OperationFailure(ErrorCode.NOT_FOUND, "No Agent in memory fixture") override fun objects(date: String?): List = emptyList() override fun page(query: Query, cursor: Cursor, includeText: Boolean) = fetch(cursor) - override fun profile(date: String, txid: Long, includeText: Boolean) = profile(txid) + override fun profile(date: String, txid: Long, includeText: Boolean, includeStepDetails: Boolean) = profile(txid) override fun close() { closed = true } } diff --git a/src/test/kotlin/dev/scouterctl/ProfileTextTest.kt b/src/test/kotlin/dev/scouterctl/ProfileTextTest.kt index 936be86..6bdfe96 100644 --- a/src/test/kotlin/dev/scouterctl/ProfileTextTest.kt +++ b/src/test/kotlin/dev/scouterctl/ProfileTextTest.kt @@ -1,7 +1,6 @@ package dev.scouterctl import dev.scouterctl.core.* -import dev.scouterctl.adapters.serialization.* import io.kotest.core.spec.style.FunSpec import io.kotest.engine.spec.tempdir @@ -18,7 +17,7 @@ class ProfileTextTest : FunSpec({ WireFixture(fault).use { server -> val result = cli.run(server, arguments) result.exit shouldBe 0 - val profile = json.decodeFromJsonElement(result.data()).toModel() + val profile = decodeLiveProfile(result.data()) profile.txid shouldBe WireFixture.ids.first().toString() profile.steps[0] shouldBe ProfileStep("MESSAGE", 1) val sql = profile.steps.mapNotNull { it.sql } @@ -43,7 +42,7 @@ class ProfileTextTest : FunSpec({ WireFixture(Fault.MANY_SQL).use { server -> val result = cli.run(server, arguments) result.exit shouldBe 0 - val profile = json.decodeFromJsonElement(result.data()).toModel() + val profile = decodeLiveProfile(result.data()) profile.steps.map { it.sql?.statement } shouldBe (0..100).map { DictionaryText(it, "statement $it") } } } @@ -53,7 +52,7 @@ class ProfileTextTest : FunSpec({ WireFixture(Fault.EMPTY_TEXT).use { server -> val result = cli.run(server, arguments) result.exit shouldBe 0 - val profile = json.decodeFromJsonElement(result.data()).toModel() + val profile = decodeLiveProfile(result.data()) profile.transaction shouldBe ProfileTransaction("available", DictionaryText(10), DictionaryText(-789)) profile.steps.mapNotNull { it.sql } shouldBe listOf( SqlProfile(123, 7, 0, 1, DictionaryText(123)), diff --git a/src/test/kotlin/dev/scouterctl/ProfileTimingTest.kt b/src/test/kotlin/dev/scouterctl/ProfileTimingTest.kt new file mode 100644 index 0000000..a3571a4 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/ProfileTimingTest.kt @@ -0,0 +1,70 @@ +package dev.scouterctl + +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.* +import java.nio.file.Files + +class ProfileTimingTest : FunSpec({ + val txid = WireFixture.ids.first().toString() + val profile = listOf("profile", "get", "--date", "20261101", "--txid", txid) + + test("live profiles expose recorded timing and network outcomes without targets") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(Fault.PROFILE_NETWORK).use { server -> + val result = cli.run(server, profile) + result.exit shouldBe 0 + val steps = result.data()["steps"]!!.jsonArray.map { it.jsonObject }.associateBy { it["type"]!!.jsonPrimitive.content } + steps.getValue("HASHED_MESSAGE")["startTimeMs"] shouldBe JsonPrimitive(0) + steps.getValue("SQL3")["startTimeMs"] shouldBe JsonPrimitive(350) + steps.getValue("SQL3")["sql"]!!.jsonObject["elapsedMs"] shouldBe JsonPrimitive(7) + for ((type, timing) in mapOf("SOCKET" to listOf(400, 25, -510), "APICALL" to listOf(430, 180, -519), + "APICALL2" to listOf(650, 50, 0))) { + val step = steps.getValue(type) + step["parent"] shouldBe JsonPrimitive(1) + listOf(step["startTimeMs"], step["elapsedMs"], step["errorHash"]) shouldBe timing.map(::JsonPrimitive) + step.containsKey("target") shouldBe false + } + val control = steps.getValue("CONTROL") + control.containsKey("startTimeMs") shouldBe false + control.containsKey("parent") shouldBe false + result.output shouldNotContain "192.0.2.17" + result.output shouldNotContain "internal-target.example" + result.output shouldNotContain "private-control-message" + } + } + + test("only the live text opt-in exposes recorded socket and API targets") { + val cli = ContractCli(tempdir().toPath()) + WireFixture(Fault.PROFILE_NETWORK).use { server -> + val result = cli.run(server, profile + "--include-text") + result.exit shouldBe 0 + val steps = result.data()["steps"]!!.jsonArray.map { it.jsonObject }.associateBy { it["type"]!!.jsonPrimitive.content } + steps.getValue("SOCKET")["target"] shouldBe Json.parseToJsonElement("""{"address":"192.0.2.17","port":443}""") + steps.getValue("APICALL")["target"] shouldBe Json.parseToJsonElement("""{"address":"internal-target.example:9443"}""") + steps.getValue("APICALL2").containsKey("target") shouldBe false + result.output shouldNotContain "private-control-message" + } + } + + test("capturing a profile with network steps preserves the v3 step layout and excludes targets") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + WireFixture(Fault.PROFILE_NETWORK).use { server -> + val args = listOf("capture", "--include-profiles", "--include-text", "--directory", directory.toString()) + evidenceWindow + val capture = ContractCli(work).run(server, args) + capture.exit shouldBe 0 + capture.data()["schemaVersion"] shouldBe JsonPrimitive("scouterctl.capture/v3") + val page = capture.data()["pages"]!!.jsonArray.first().jsonObject["name"]!!.jsonPrimitive.content + val record = Json.parseToJsonElement(Files.readAllLines(directory.resolve(page)).first()).jsonObject + val steps = record["profile"]!!.jsonObject["steps"]!!.jsonArray + steps.map { it.jsonObject.keys }.distinct() shouldBe listOf(setOf("type", "order", "sql")) + val result = runOfflineCli(work, listOf("inspect", "--directory", directory.toString(), "--txid", txid, "--include-text")) + result.exit shouldBe 0 + result.output shouldNotContain "internal-target.example" + result.output shouldNotContain "192.0.2.17" + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/SpringOfflineCheck.kt b/src/test/kotlin/dev/scouterctl/SpringOfflineCheck.kt index 76c92b2..b9f4e94 100644 --- a/src/test/kotlin/dev/scouterctl/SpringOfflineCheck.kt +++ b/src/test/kotlin/dev/scouterctl/SpringOfflineCheck.kt @@ -30,7 +30,7 @@ internal class SpringOfflineCheck(private val work: Path) { val resolved = runOfflineCli(work, args + "--include-text") check(resolved.exit == 0 && resolved.stderr.isEmpty()) val inspection = json.decodeFromJsonElement(resolved.data()).toModel() - check(inspection.record.profile == json.decodeFromJsonElement(expected).toModel()) + check(inspection.record.profile == decodeLiveProfile(expected)) check(inspection.record.xlog == saved.xlog) val structural = runOfflineCli(work, args) check(structural.exit == 0 && structural.stderr.isEmpty()) diff --git a/src/test/kotlin/dev/scouterctl/SpringProfileTextCheck.kt b/src/test/kotlin/dev/scouterctl/SpringProfileTextCheck.kt index 2851c2d..0f6d1fa 100644 --- a/src/test/kotlin/dev/scouterctl/SpringProfileTextCheck.kt +++ b/src/test/kotlin/dev/scouterctl/SpringProfileTextCheck.kt @@ -1,15 +1,12 @@ package dev.scouterctl import dev.scouterctl.core.* -import dev.scouterctl.adapters.serialization.* - -import kotlinx.serialization.json.* class SpringProfileTextCheck(private val cli: AgentCli, private val date: String) { fun verify(row: XLogRecord, route: String) { val event = cli.run("text-profile-${row.txid}", listOf("profile", "get", "--date", date, "--txid", row.txid, "--include-text")).single() - val profile = json.decodeFromJsonElement(event["data"]!!).toModel() + val profile = decodeLiveProfile(event["data"]!!) val transaction = checkNotNull(profile.transaction) val serviceName = if (route == "/error") route else "$route" check(profile.txid == row.txid) diff --git a/src/test/kotlin/dev/scouterctl/SpringSmoke.kt b/src/test/kotlin/dev/scouterctl/SpringSmoke.kt index b69ec88..100f9fc 100644 --- a/src/test/kotlin/dev/scouterctl/SpringSmoke.kt +++ b/src/test/kotlin/dev/scouterctl/SpringSmoke.kt @@ -143,7 +143,7 @@ fun main(args: Array) { check((row.errorHash != 0) == expectedError) { "$route error flag: ${row.errorHash}" } val event = cli.run("profile-${route.substringAfterLast('/')}-${index + 1}", listOf("profile", "get", "--date", date, "--txid", row.txid)).single() - val profile = json.decodeFromJsonElement(event["data"]!!).toModel() + val profile = decodeLiveProfile(event["data"]!!) check(profile.txid == row.txid && profile.steps.isNotEmpty()) val sql = profile.steps.mapNotNull { it.sql } if (route == "/poc/ok" || route == "/error") check(sql.isEmpty()) else { diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index c661752..3ce4536 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -3,6 +3,10 @@ package dev.scouterctl import scouter.io.DataInputX import scouter.io.DataOutputX import scouter.lang.pack.* +import scouter.lang.step.ApiCallStep +import scouter.lang.step.ApiCallStep2 +import scouter.lang.step.HashedMessageStep +import scouter.lang.step.SocketStep import scouter.lang.step.MessageStep import scouter.lang.step.SqlStep3 import scouter.lang.step.Step @@ -28,7 +32,7 @@ enum class Fault { NONE, STATUS_MALFORMED, CONFIG_MALFORMED, CONFIG_DUPLICATE, CONFIG_INVALID_VISIBLE, VERSION, AUTH, SESSION, MISSING_META, INVALID_META, STALLED_CURSOR, DISCONNECT, EMPTY_OBJECTS, STRUCTURAL_ONLY, MISSING_TEXT, EMPTY_TEXT, MANY_SQL, TEXT_WRONG_TYPE, TEXT_WRONG_HASH, TEXT_DUPLICATE, TEXT_PACK, TEXT_DISCONNECT, TEXT_SESSION, XLOG_MISSING, XLOG_WRONG_ID, - PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, + PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, } class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPassword: String? = null, @@ -260,7 +264,20 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas val missing = request.getLong("txid") == ids[1] && (fault == Fault.PROFILE_MISSING || (fault == Fault.PROFILE_MISSING_ONCE && !profileMissed)) if (missing) profileMissed = true else send(output, XLogProfilePack().apply { - profile = if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( + profile = if (fault == Fault.PROFILE_NETWORK) Step.toBytes(arrayOf( + HashedMessageStep().apply { index = 0; hash = 77 }, + SqlStep3().apply { index = 1; start_time = 350; hash = 123; elapsed = 7 }, + SocketStep().apply { + index = 2; parent = 1; start_time = 400; elapsed = 25; error = -510 + ipaddr = byteArrayOf(192.toByte(), 0, 2, 17); port = 443 + }, + ApiCallStep().apply { + index = 3; parent = 1; start_time = 430; elapsed = 180; error = -519 + opt = 1; address = "internal-target.example:9443" + }, + ApiCallStep2().apply { index = 4; parent = 1; start_time = 650; elapsed = 50 }, + StepControl().apply { code = 7; message = "private-control-message" }, + )) else if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( SqlStep3().apply { index = 1; hash = 123; elapsed = 7 }, StepControl().apply { code = if (fault == Fault.PROFILE_TRUNCATED) 0 else 7; message = " ** Profile Truncated ** " } )) else if (fault == Fault.PROFILE_MANY_STEPS) Step.toBytes(Array(1001) { number -> From 091d8059b2029edf3ee921f1a20a71116e590d0a Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Wed, 11 Nov 2026 16:14:23 +0900 Subject: [PATCH 07/17] =?UTF-8?q?ci:=20=F0=9F=91=B7=20Enforce=20coverage?= =?UTF-8?q?=20and=20report=20test=20results=20on=20PRs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enforce Linux line and branch coverage thresholds and publish test reports. Refs #72. --- .github/scripts/summarize-tests.py | 58 +++++++++ .github/workflows/ci-release.yml | 86 +++++++++++++- CONTRIBUTING.ko.md | 4 + CONTRIBUTING.md | 4 + build.gradle.kts | 3 + docs/local-ci.md | 2 +- docs/testing.md | 20 ++++ gradle/coverage.gradle.kts | 110 ++++++++++++++++++ .../kotlin/dev/scouterctl/CliContractTest.kt | 3 +- src/test/kotlin/dev/scouterctl/CliCoverage.kt | 26 +++++ .../kotlin/dev/scouterctl/CliCoverageTest.kt | 30 +++++ .../dev/scouterctl/CommandDistributionTest.kt | 1 + .../dev/scouterctl/ContextPrecedenceTest.kt | 3 +- .../scouterctl/CounterHistoryDecoderTest.kt | 62 ++++++++++ src/test/kotlin/dev/scouterctl/OfflineCli.kt | 3 +- src/test/kotlin/dev/scouterctl/TerminalCli.kt | 6 +- .../kotlin/dev/scouterctl/UnicodeCliTest.kt | 3 +- 17 files changed, 410 insertions(+), 14 deletions(-) create mode 100644 .github/scripts/summarize-tests.py create mode 100644 gradle/coverage.gradle.kts create mode 100644 src/test/kotlin/dev/scouterctl/CliCoverage.kt create mode 100644 src/test/kotlin/dev/scouterctl/CliCoverageTest.kt create mode 100644 src/test/kotlin/dev/scouterctl/CounterHistoryDecoderTest.kt diff --git a/.github/scripts/summarize-tests.py b/.github/scripts/summarize-tests.py new file mode 100644 index 0000000..ba021e1 --- /dev/null +++ b/.github/scripts/summarize-tests.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +"""Render Gradle test counts and the coverage summary without publishing logs.""" + +import argparse +from pathlib import Path +import xml.etree.ElementTree as ET + + +def summarize(tests_directory: Path, coverage_summary: Path) -> str: + lines = ["## Test results (Linux)", ""] + reports = sorted(tests_directory.glob("TEST-*.xml")) + try: + if not reports: + raise ValueError("No test reports") + totals = dict.fromkeys(("tests", "failures", "errors", "skipped"), 0) + for report in reports: + suite = ET.parse(report).getroot() + if suite.tag != "testsuite": + raise ValueError("Unexpected report root") + counts = {name: int(suite.attrib[name]) for name in totals} + if min(counts.values()) < 0 or counts["tests"] < sum(counts[name] for name in ("failures", "errors", "skipped")): + raise ValueError("Invalid report counts") + for name, value in counts.items(): + totals[name] += value + passed = totals["tests"] - totals["failures"] - totals["errors"] - totals["skipped"] + lines += [ + "| Total | Passed | Failed | Errors | Skipped |", + "| ---: | ---: | ---: | ---: | ---: |", + f"| {totals['tests']} | {passed} | {totals['failures']} | {totals['errors']} | {totals['skipped']} |", + ] + if not totals["tests"]: + lines += ["", "No tests were executed."] + except (ET.ParseError, OSError, KeyError, ValueError): + lines += ["Test results are unavailable or invalid. Check the build logs; this is not a passing test result."] + lines += [""] + try: + coverage = coverage_summary.read_text(encoding="utf-8").strip() + except (OSError, UnicodeError): + coverage = "" + if coverage: + lines += [coverage] + else: + lines += ["## Test coverage", "", "Coverage is unavailable. The build or tests may have stopped before a report was generated."] + return "\n".join(lines) + "\n" + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tests-directory", type=Path, default=Path("build/test-results/test")) + parser.add_argument("--coverage-summary", type=Path, default=Path("build/reports/jacoco/summary.md")) + parser.add_argument("--output", type=Path, default=Path("build/reports/ci/summary.md")) + args = parser.parse_args() + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(summarize(args.tests_directory, args.coverage_summary), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.yml index 7d81cab..7d044bb 100644 --- a/.github/workflows/ci-release.yml +++ b/.github/workflows/ci-release.yml @@ -67,8 +67,12 @@ jobs: env: EXPECTED_TAG: ${{ inputs.release_tag }} run: | + coverage_args=() + if [[ -z "${EXPECTED_TAG:-}" ]]; then + coverage_args=(-Pcoverage=true coverageCheck) + fi if [[ -f .github/scripts/verify-release-files.sh ]]; then - ./gradlew --no-daemon test verifyReleaseInputs + ./gradlew --no-daemon test "${coverage_args[@]}" verifyReleaseInputs printf 'format=commands\n' >> "$GITHUB_OUTPUT" printf 'package_path=build/release-inputs\n' >> "$GITHUB_OUTPUT" else @@ -83,6 +87,34 @@ jobs: [[ -z "${EXPECTED_TAG:-}" || "$EXPECTED_TAG" == "v$version" ]] printf 'version=%s\n' "$version" >> "$GITHUB_OUTPUT" printf 'sha=%s\n' "$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + - name: Summarize tests and coverage + if: ${{ always() && inputs.release_tag == '' }} + run: | + python3 .github/scripts/summarize-tests.py + cat build/reports/ci/summary.md >> "$GITHUB_STEP_SUMMARY" + - name: Upload PR test summary + if: ${{ always() && github.event_name == 'pull_request' && !env.ACT }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: pr-test-summary + path: build/reports/ci/summary.md + if-no-files-found: warn + retention-days: 14 + - name: Upload coverage reports + if: ${{ always() && !env.ACT && hashFiles('build/reports/jacoco/**') != '' }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: coverage-reports + path: build/reports/jacoco/ + if-no-files-found: error + retention-days: 14 + - name: Save local coverage reports + if: ${{ always() && env.ACT == 'true' && hashFiles('build/reports/jacoco/**') != '' }} + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2; act artifact API + with: + name: coverage-reports + path: build/reports/jacoco/ + if-no-files-found: error - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 if: ${{ always() && !env.ACT }} with: @@ -112,6 +144,58 @@ jobs: path: ${{ steps.package.outputs.package_path }} if-no-files-found: error + test-comment: + name: Comment test results + needs: build + if: >- + always() && !cancelled() && github.event_name == 'pull_request' && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.user.login != 'dependabot[bot]' + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + actions: read + pull-requests: write + steps: + - name: Download test summary + continue-on-error: true + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: pr-test-summary + path: ci-summary + - name: Create or update the PR comment + uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + env: + BUILD_RESULT: ${{ needs.build.result }} + with: + script: | + const fs = require('node:fs'); + const marker = ''; + const {owner, repo} = context.repo; + const issue_number = context.payload.pull_request.number; + const head = context.payload.pull_request.head.sha; + const {data: pr} = await github.rest.pulls.get({owner, repo, pull_number: issue_number}); + if (pr.state !== 'open' || pr.head.sha !== head) { + core.info('Skipping results for an outdated or closed pull request.'); + return; + } + // Read report text only. Never check out or execute PR code with this token. + const file = 'ci-summary/summary.md'; + const summary = fs.existsSync(file) && fs.statSync(file).size <= 16000 + ? fs.readFileSync(file, 'utf8') + : 'Test and coverage summary unavailable. Check the workflow logs.'; + const run = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`; + const body = `${marker}\n**Linux CI: ${process.env.BUILD_RESULT}** · Commit \`${head.slice(0, 7)}\`\n\n${summary}\n[Workflow logs and report artifacts](${run})\n\n_Updated on each CI run. Windows checks, when required, are reported separately in the workflow._`; + const comments = await github.paginate(github.rest.issues.listComments, + {owner, repo, issue_number, per_page: 100}); + const previous = comments.find(comment => + comment.user?.login === 'github-actions[bot]' && comment.body?.startsWith(marker)); + if (previous) { + await github.rest.issues.updateComment({owner, repo, comment_id: previous.id, body}); + } else { + await github.rest.issues.createComment({owner, repo, issue_number, body}); + } + portable: name: Windows command package needs: build diff --git a/CONTRIBUTING.ko.md b/CONTRIBUTING.ko.md index 56db909..b157e64 100644 --- a/CONTRIBUTING.ko.md +++ b/CONTRIBUTING.ko.md @@ -24,6 +24,10 @@ Linux 명령 ZIP의 빌드·검증: 개발용 설치는 `./gradlew installDist` 후 `build/install/scouterctl/bin/scouterctl`로 실행합니다. 네이티브 Windows 패키지는 별도 빌드 도구가 필요합니다. [릴리스 안내(영문)](docs/releasing.md)를 참고하세요. 실제 Collector·Agent·Spring 검사는 [테스트 안내(영문)](docs/testing.md)의 별도 작업이며, [로컬 CI(영문)](docs/local-ci.md)에서 act 실행 방법을 확인할 수 있습니다. +커버리지 리포트를 만들고 라인·분기 최소 기준을 검증하려면 `./gradlew -Pcoverage=true coverageCheck`를 실행합니다. 테스트 JVM과 CLI 자식 프로세스를 함께 집계하며, 라인 90%·분기 80% 미만이면 실패합니다. 기본 `test` 실행에는 커버리지를 적용하지 않습니다. HTML/XML 리포트와 요약은 `build/reports/jacoco/`에 생성됩니다. 측정 범위와 CI 동작은 [테스트 안내(영문)](docs/testing.md#coverage)를 참고하세요. + +같은 저장소의 PR에는 Linux 테스트 통과·실패·오류·스킵 수, 라인·분기 커버리지와 최소 기준, CI 결과와 실행 링크를 코멘트로 표시합니다. 재실행하면 기존 봇 코멘트를 갱신하며, 이전 커밋의 결과는 게시하지 않습니다. Windows 검사는 워크플로우에서 별도로 확인합니다. 읽기 전용 토큰을 사용하는 포크·Dependabot PR은 Actions 요약과 artifact를 제공하며 코멘트는 작성하지 않습니다. + ## 코드와 테스트 - CLI 기능은 Kotlin으로 작성합니다. Kotest `FunSpec`과 matcher를 사용하며 실행에는 JUnit Platform을 사용합니다. 네이티브 [Windows 실행기(영문)](https://github.com/oxspan/scouterctl/blob/master/native/windows/README.md)는 별도 플랫폼 구성 요소입니다. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bcee0e2..62f101f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,6 +24,10 @@ To build and verify the Linux command package: For a local Linux installation, use `./gradlew installDist` and run `build/install/scouterctl/bin/scouterctl`. Native Windows packaging additionally requires the pinned toolchain; see [releasing](docs/releasing.md). Real Collector/Agent/Spring checks are opt-in and described in [testing](docs/testing.md). Use [local CI](docs/local-ci.md) to run the workflow build with act when needed. +To generate coverage reports and enforce the line/branch minimums, run `./gradlew -Pcoverage=true coverageCheck`. This includes the test JVM and CLI subprocesses. See [testing](docs/testing.md#coverage) for measurement scope and report locations. + +Same-repository PRs receive a bot comment with Linux test counts, coverage/thresholds, the CI result and run links. Re-runs update the existing comment and outdated commits are skipped. Windows checks remain in the workflow; fork and Dependabot PRs receive the Actions summary and artifacts without a comment because their tokens are read-only. + ## Code and tests - Implement CLI behavior in Kotlin. Use Kotest `FunSpec` and matchers; JUnit Platform runs Kotest, while test APIs remain Kotest. The native [Windows launcher](https://github.com/oxspan/scouterctl/blob/master/native/windows/README.md) is a separate platform boundary. diff --git a/build.gradle.kts b/build.gradle.kts index 3966c28..48d30d3 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -4,7 +4,10 @@ plugins { kotlin("jvm") version "2.4.10" kotlin("plugin.serialization") version "2.4.10" application + jacoco } + +apply(from = "gradle/coverage.gradle.kts") group = "dev.scouterctl" version = providers.gradleProperty("candidateVersion").orElse( providers.fileContents(layout.projectDirectory.file("version.txt")).asText.map(String::trim)).get() diff --git a/docs/local-ci.md b/docs/local-ci.md index afd43b4..847ca2c 100644 --- a/docs/local-ci.md +++ b/docs/local-ci.md @@ -6,7 +6,7 @@ Use focused Gradle tests during development. Before a PR handoff or after changi scripts/ci-local.sh ``` -The wrapper selects only the `build` job in `.github/workflows/ci-release.yml`. It runs JDK setup, Kotest, canonical command packaging and package verification, then stores uploaded artifacts under `.local/act/artifacts/`. It does not select the portable, scale or release jobs. Local runs consume no GitHub-hosted runner minutes. +The wrapper selects only the `build` job in `.github/workflows/ci-release.yml`. It runs JDK setup, Kotest with coverage verification, canonical command packaging and package verification, then stores uploaded artifacts (including coverage reports) under `.local/act/artifacts/`. It does not select the portable, scale or release jobs. Local runs consume no GitHub-hosted runner minutes. ## WSL setup diff --git a/docs/testing.md b/docs/testing.md index 1d899a4..a74c8c7 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -11,6 +11,26 @@ Use `gradlew.bat` on Windows. The suite uses local protocol/file/process fixture The following integration tasks start isolated Collector/Agent/application processes. Use test installations and data, never production directories. Gradle still runs on JDK 21 even when a fixture uses Java 8. Each task prints its evidence directory, stops its child processes and removes its temporary Collector account. Preserve useful output and remove the temporary directory when finished. +## Coverage + +Run the complete suite, generate JaCoCo HTML/XML reports, and enforce minimum coverage: + +```sh +./gradlew -Pcoverage=true coverageCheck +``` + +The initial minimums are 90% for lines and 80% for branches. Coverage includes all production classes from the main source set, combining the test JVM with packaged CLI subprocesses (including the extracted command launcher). Dependencies, test fixtures, optional real-Collector smoke checks, and scale measurements are outside this scope. Ordinary `test` runs leave coverage disabled; use the full suite without `--tests` for a comparable baseline. Coverage requires the locally built CLI, so do not combine it with `-PcliJar`. + +Each coverage run executes fresh tests, clears old execution data/reports, and writes a separate execution file for each CLI process. JVM arguments enable the agent without startup messages on stdout/stderr. Missing test JVM or subprocess data, empty report counters, or coverage below either minimum fails verification. + +Reports are written to `build/reports/jacoco/test/html/index.html` and `build/reports/jacoco/test/jacocoTestReport.xml`; `build/reports/jacoco/summary.md` contains counts, percentages, thresholds, and pass/fail results. Linux CI runs coverage and package verification in one Gradle invocation, publishes the summary, and retains reports for 14 days even when a threshold fails. Local act runs save the same reports as artifacts. Historical release-asset repair does not apply new coverage checks to tagged code. + +No Codecov credentials are needed. PR coverage comparisons and configuring a required merge check are separate follow-up work. + +For same-repository PRs, CI creates one bot comment showing Linux test totals (passed, failed, errors, skipped), line/branch coverage and minimums, the build result, tested commit, and workflow/artifact links. Later runs update that comment. Failed builds still report available results; missing reports are marked unavailable. Results from outdated commits are not posted. Windows checks remain visible separately in the workflow. + +The comment job reads a Markdown artifact and has write permission only for PR comments; it does not check out or execute PR code. Fork and Dependabot PRs use read-only tokens and receive the Actions summary/artifacts without a bot comment. Local act runs still select only the build job. + ## Collector and Agent checks Use an isolated Collector installation and data directory. The optional smoke tools create their own configuration and accounts and stop the processes they start. Never point them at production data. diff --git a/gradle/coverage.gradle.kts b/gradle/coverage.gradle.kts new file mode 100644 index 0000000..872002a --- /dev/null +++ b/gradle/coverage.gradle.kts @@ -0,0 +1,110 @@ +import org.gradle.api.tasks.testing.Test +import org.gradle.testing.jacoco.plugins.JacocoPluginExtension +import org.gradle.testing.jacoco.plugins.JacocoTaskExtension +import org.gradle.testing.jacoco.tasks.JacocoReport +import org.gradle.testing.jacoco.tasks.JacocoCoverageVerification +import javax.xml.parsers.DocumentBuilderFactory + +val coverageEnabled = providers.gradleProperty("coverage").map(String::toBoolean).getOrElse(false) +val coverageDirectory = layout.buildDirectory.dir("jacoco") +val coverageSummary = layout.buildDirectory.file("reports/jacoco/summary.md") +val minimumLineCoverage = "0.90".toBigDecimal() +val minimumBranchCoverage = "0.80".toBigDecimal() + +extensions.configure { toolVersion = "0.8.14" } + +val coverageAgent = tasks.register("coverageAgent") { + from({ zipTree(configurations.getByName("jacocoAgent").singleFile) }) + include("jacocoagent.jar") + into(layout.buildDirectory.dir("coverage-agent")) +} + +tasks.named("test") { + extensions.configure { + isEnabled = coverageEnabled + includes = listOf("dev.scouterctl.*") + destinationFile = coverageDirectory.get().file("test.exec").asFile + } + if (coverageEnabled) { + dependsOn(coverageAgent) + outputs.dir(coverageDirectory) + // A coverage run must execute the whole suite and never reuse old subprocess data. + outputs.upToDateWhen { false } + outputs.doNotCacheIf("Coverage requires fresh JVM execution data") { true } + systemProperty("coverage.agent", layout.buildDirectory.file("coverage-agent/jacocoagent.jar").get().asFile.absolutePath) + systemProperty("coverage.directory", coverageDirectory.get().asFile.absolutePath) + doFirst { + check(!providers.gradleProperty("cliJar").isPresent) { "Coverage requires the JAR built from this checkout" } + delete(coverageDirectory, layout.buildDirectory.dir("reports/jacoco")) + coverageDirectory.get().asFile.mkdirs() + } + } +} + +val coverageData = tasks.register("coverageData") { + dependsOn("test") + doLast { + check(coverageEnabled) { "Enable coverage with -Pcoverage=true" } + val directory = coverageDirectory.get().asFile + check(directory.resolve("test.exec").let { it.isFile && it.length() > 0 }) { + "Missing test JVM coverage data" + } + check(directory.listFiles().orEmpty().any { it.name.startsWith("cli-") && it.extension == "exec" && it.length() > 0 }) { + "Missing CLI subprocess coverage data" + } + } +} + +tasks.named("jacocoTestReport") { + dependsOn(coverageData) + executionData.setFrom(fileTree(coverageDirectory) { include("*.exec") }) + reports { xml.required.set(true); html.required.set(true) } +} + +val summarizeCoverage = tasks.register("coverageSummary") { + dependsOn("jacocoTestReport") + doLast { + val report = tasks.named("jacocoTestReport").get().reports.xml.outputLocation.get().asFile + check(report.isFile) { "Missing JaCoCo XML report" } + val factory = DocumentBuilderFactory.newInstance().apply { + setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false) + setFeature("http://xml.org/sax/features/external-general-entities", false) + setFeature("http://xml.org/sax/features/external-parameter-entities", false) + } + val root = factory.newDocumentBuilder().parse(report).documentElement + val counters = (0 until root.childNodes.length).map { root.childNodes.item(it) } + .filterIsInstance().filter { it.tagName == "counter" } + .associateBy { it.getAttribute("type") } + val rows = listOf("LINE" to minimumLineCoverage, "BRANCH" to minimumBranchCoverage).map { (type, minimum) -> + val counter = checkNotNull(counters[type]) { "Missing $type coverage counter" } + val covered = counter.getAttribute("covered").toLong() + val total = covered + counter.getAttribute("missed").toLong() + check(total > 0) { "Empty $type coverage data" } + val ratio = covered.toBigDecimal().divide(total.toBigDecimal(), 8, java.math.RoundingMode.HALF_UP) + val percent = ratio.multiply(100.toBigDecimal()).setScale(2, java.math.RoundingMode.HALF_UP) + "| $type | $covered / $total | $percent% | ${minimum * 100.toBigDecimal()}% | ${if (ratio >= minimum) "PASS" else "FAIL"} |" + } + val summary = "## Test coverage\n\n| Metric | Covered / total | Coverage | Minimum | Result |\n| --- | --- | --- | --- | --- |\n" + + rows.joinToString("\n") + "\n\nIncludes the test JVM and CLI subprocesses. Excludes optional real-Collector smoke tests.\n" + coverageSummary.get().asFile.apply { parentFile.mkdirs(); writeText(summary) } + logger.lifecycle(summary) + } +} + +tasks.named("jacocoTestCoverageVerification") { + dependsOn(summarizeCoverage) + executionData.setFrom(fileTree(coverageDirectory) { include("*.exec") }) + violationRules { + rule { + limit { counter = "LINE"; minimum = minimumLineCoverage } + limit { counter = "BRANCH"; minimum = minimumBranchCoverage } + } + } +} + +tasks.register("coverageCheck") { + group = "verification" + description = "Run tests, report coverage, and enforce line/branch minimums (requires -Pcoverage=true)." + dependsOn("jacocoTestCoverageVerification") +} +if (coverageEnabled) tasks.named("check") { dependsOn("coverageCheck") } diff --git a/src/test/kotlin/dev/scouterctl/CliContractTest.kt b/src/test/kotlin/dev/scouterctl/CliContractTest.kt index 58d468a..d0a4761 100644 --- a/src/test/kotlin/dev/scouterctl/CliContractTest.kt +++ b/src/test/kotlin/dev/scouterctl/CliContractTest.kt @@ -141,8 +141,7 @@ internal class ContractCli(private val temp: Path) { val stderr = Files.createTempFile(temp, "stderr", ".txt") val stdout = Files.createTempFile(temp, "stdout", ".jsonl") val launchArgs = if (System.getProperty("os.name").startsWith("Windows")) listOf(utf8Arguments(temp, args)) else args - val process = ProcessBuilder(listOf(Path.of(System.getProperty("java.home"), "bin", "java").toString(), - "-jar", System.getProperty("cli.jar")) + launchArgs) + val process = ProcessBuilder(cliJavaCommand() + launchArgs) .directory(temp.toFile()) .redirectOutput(stdout.toFile()).redirectError(stderr.toFile()).apply { environment().remove("SCOUTERCTL_PASSWORD") diff --git a/src/test/kotlin/dev/scouterctl/CliCoverage.kt b/src/test/kotlin/dev/scouterctl/CliCoverage.kt new file mode 100644 index 0000000..98bde92 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CliCoverage.kt @@ -0,0 +1,26 @@ +package dev.scouterctl + +import java.nio.file.Path +import java.util.UUID + +/** Explicit JVM arguments keep coverage instrumentation out of stdout/stderr. */ +internal fun cliCoverageOptions(): List { + val agent = System.getProperty("coverage.agent") ?: return emptyList() + val directory = checkNotNull(System.getProperty("coverage.directory")) + val destination = Path.of(directory, "cli-${UUID.randomUUID()}.exec") + return listOf("-javaagent:$agent=destfile=$destination,append=false,includes=dev.scouterctl.*") +} + +internal fun cliJavaCommand( + jar: String = System.getProperty("cli.jar"), + jvmOptions: List = emptyList(), +): List = listOf(Path.of(System.getProperty("java.home"), "bin", "java").toString()) + + cliCoverageOptions() + jvmOptions + listOf("-jar", jar) + +/** The Gradle-generated command launcher reads JAVA_OPTS without printing it. */ +internal fun MutableMap.withCliCoverage() { + val options = cliCoverageOptions() + if (options.isNotEmpty()) { + this["JAVA_OPTS"] = listOfNotNull(this["JAVA_OPTS"], options.joinToString(" ") { "\"$it\"" }).joinToString(" ") + } +} diff --git a/src/test/kotlin/dev/scouterctl/CliCoverageTest.kt b/src/test/kotlin/dev/scouterctl/CliCoverageTest.kt new file mode 100644 index 0000000..41559bd --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CliCoverageTest.kt @@ -0,0 +1,30 @@ +package dev.scouterctl + +import dev.scouterctl.bootstrap.BuildInfo +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import java.nio.file.Files +import java.nio.file.Path +import java.util.concurrent.TimeUnit + +class CliCoverageTest : FunSpec({ + if (System.getProperty("coverage.agent") != null) { + test("CLI processes write separate coverage files without changing version output or stderr") { + val destinations = (1..2).map { + val command = cliJavaCommand() + val agent = command.single { it.startsWith("-javaagent:") } + val destination = Path.of(agent.substringAfter("destfile=").substringBefore(',')) + val process = ProcessBuilder(command + "--version").start() + try { + process.waitFor(15, TimeUnit.SECONDS) shouldBe true + process.exitValue() shouldBe 0 + process.inputStream.bufferedReader().readText().trim() shouldBe "scouterctl ${BuildInfo.version}" + process.errorStream.bufferedReader().readText() shouldBe "" + (Files.size(destination) > 0) shouldBe true + destination + } finally { process.destroyForcibly() } + } + destinations.distinct().size shouldBe 2 + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/CommandDistributionTest.kt b/src/test/kotlin/dev/scouterctl/CommandDistributionTest.kt index b330164..6eb0f16 100644 --- a/src/test/kotlin/dev/scouterctl/CommandDistributionTest.kt +++ b/src/test/kotlin/dev/scouterctl/CommandDistributionTest.kt @@ -105,6 +105,7 @@ private class InstalledCommand(work: Path) { environment().remove("SCOUTERCTL_PASSWORD") environment().remove("JAVA_OPTS") environment().remove("SCOUTERCTL_OPTS") + environment().withCliCoverage() environment()["PATH"] = if (javaHome == null) launcher.parent.toString() else "${launcher.parent}${java.io.File.pathSeparator}${environment()["PATH"]}" if (javaHome == null) environment().remove("JAVA_HOME") else environment()["JAVA_HOME"] = javaHome diff --git a/src/test/kotlin/dev/scouterctl/ContextPrecedenceTest.kt b/src/test/kotlin/dev/scouterctl/ContextPrecedenceTest.kt index 258384e..52279ed 100644 --- a/src/test/kotlin/dev/scouterctl/ContextPrecedenceTest.kt +++ b/src/test/kotlin/dev/scouterctl/ContextPrecedenceTest.kt @@ -19,8 +19,7 @@ class ContextPrecedenceTest : FunSpec({ val directory = tempdir().toPath() val stdout = directory.resolve("stdout.json") val stderr = directory.resolve("stderr.txt") - val process = ProcessBuilder(listOf(Path.of(System.getProperty("java.home"), "bin", "java").toString(), - "-jar", System.getProperty("cli.jar")) + args).apply { + val process = ProcessBuilder(cliJavaCommand() + args).apply { environment().remove("SCOUTERCTL_CONFIG") environment().remove("XDG_CONFIG_HOME") environment().remove("SCOUTERCTL_PASSWORD") diff --git a/src/test/kotlin/dev/scouterctl/CounterHistoryDecoderTest.kt b/src/test/kotlin/dev/scouterctl/CounterHistoryDecoderTest.kt new file mode 100644 index 0000000..c655af5 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CounterHistoryDecoderTest.kt @@ -0,0 +1,62 @@ +package dev.scouterctl + +import dev.scouterctl.adapters.collector.decodeDatedObjects +import dev.scouterctl.core.CounterTarget +import dev.scouterctl.core.ErrorCode +import dev.scouterctl.core.OperationFailure +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import scouter.lang.pack.MapPack +import scouter.lang.pack.Pack +import scouter.lang.value.DecimalValue +import scouter.lang.value.TextValue +import scouter.lang.value.Value + +class CounterHistoryDecoderTest : FunSpec({ + val invalidResponses: List>> = listOf( + "absent response" to emptyList(), + "multiple response packs" to listOf(datedObjects(), datedObjects()), + "absent hashes" to listOf(datedObjects(hashes = null)), + "absent names" to listOf(datedObjects(names = null)), + "absent types" to listOf(datedObjects(types = null)), + "scalar column" to listOf(datedObjects().apply { put("objHash", TextValue("private-value")) }), + "mismatched name count" to listOf(datedObjects(names = emptyList())), + "mismatched type count" to listOf(datedObjects(types = emptyList())), + "excessive object count" to listOf(datedObjects(hashes = List(10001) { DecimalValue(42) })), + "non-numeric hash" to listOf(datedObjects(hashes = listOf(TextValue("private-value")))), + "hash below signed 32-bit range" to listOf(datedObjects(hashes = listOf(DecimalValue(Int.MIN_VALUE.toLong() - 1)))), + "hash above signed 32-bit range" to listOf(datedObjects(hashes = listOf(DecimalValue(Int.MAX_VALUE.toLong() + 1)))), + "non-text name" to listOf(datedObjects(names = listOf(DecimalValue(42)))), + "non-text type" to listOf(datedObjects(types = listOf(DecimalValue(42)))), + ) + invalidResponses.forEach { (description, response) -> + test("dated object discovery rejects $description with a sanitized protocol error") { + val failure = shouldThrow { decodeDatedObjects(response) } + failure.code shouldBe ErrorCode.PROTOCOL_ERROR + failure.message shouldBe "Invalid dated object list response" + } + } + test("dated object discovery accepts an empty result") { + decodeDatedObjects(listOf(datedObjects(emptyList(), emptyList(), emptyList()))) shouldBe emptyList() + } + test("dated object discovery preserves signed 32-bit hash boundaries") { + for (hash in listOf(Int.MIN_VALUE, Int.MAX_VALUE)) { + decodeDatedObjects(listOf(datedObjects(hashes = listOf(DecimalValue(hash.toLong()))))) shouldBe + listOf(CounterTarget(hash, "/fixture", "java")) + } + } +}) + +private fun datedObjects( + hashes: List? = listOf(DecimalValue(42)), + names: List? = listOf(TextValue("/fixture")), + types: List? = listOf(TextValue("java")), +): MapPack = MapPack().apply { + listOf("objHash" to hashes, "objName" to names, "objType" to types).forEach { (column, values) -> + if (values != null) { + val list = newList(column) + values.forEach { list.add(it) } + } + } +} diff --git a/src/test/kotlin/dev/scouterctl/OfflineCli.kt b/src/test/kotlin/dev/scouterctl/OfflineCli.kt index 3803f4f..a5ae154 100644 --- a/src/test/kotlin/dev/scouterctl/OfflineCli.kt +++ b/src/test/kotlin/dev/scouterctl/OfflineCli.kt @@ -7,8 +7,7 @@ import java.util.concurrent.TimeUnit internal fun runOfflineCli(work: Path, args: List): Outcome { val output = Files.createTempFile(work, "inspect-", ".jsonl") val stderr = Files.createTempFile(work, "inspect-", ".stderr") - val child = ProcessBuilder(listOf(Path.of(System.getProperty("java.home"), "bin/java").toString(), - "-jar", System.getProperty("cli.jar")) + args).apply { + val child = ProcessBuilder(cliJavaCommand() + args).apply { environment().remove("SCOUTERCTL_PASSWORD") }.redirectOutput(output.toFile()).redirectError(stderr.toFile()).start() try { diff --git a/src/test/kotlin/dev/scouterctl/TerminalCli.kt b/src/test/kotlin/dev/scouterctl/TerminalCli.kt index 7565577..1b8c161 100644 --- a/src/test/kotlin/dev/scouterctl/TerminalCli.kt +++ b/src/test/kotlin/dev/scouterctl/TerminalCli.kt @@ -16,11 +16,9 @@ internal class TerminalCli(work: Path, args: List, password: String? = n put("SCOUTERCTL_CONFIG", work.resolve("config.json").toString()) put("TERM", "xterm") if (password != null) put("SCOUTERCTL_PASSWORD", password) + if (command != null) withCliCoverage() } - private val process = PtyProcessBuilder(((command ?: listOf( - Path.of(System.getProperty("java.home"), "bin", "java").toString(), - "-jar", System.getProperty("cli.jar"), - )) + args).toTypedArray()) + private val process = PtyProcessBuilder(((command ?: cliJavaCommand()) + args).toTypedArray()) .setEnvironment(environment).setDirectory(work.toString()) .setInitialColumns(2000).setInitialRows(50) .setUseWinConPty(true).setConPtyInheritCursor(false) diff --git a/src/test/kotlin/dev/scouterctl/UnicodeCliTest.kt b/src/test/kotlin/dev/scouterctl/UnicodeCliTest.kt index 55331a5..b809405 100644 --- a/src/test/kotlin/dev/scouterctl/UnicodeCliTest.kt +++ b/src/test/kotlin/dev/scouterctl/UnicodeCliTest.kt @@ -92,8 +92,7 @@ private class UnicodeCli(work: Path) { val arguments = utf8Arguments(install, args) val stdout = install.resolve("stdout.txt") val stderr = install.resolve("stderr.txt") - val process = ProcessBuilder(Path.of(System.getProperty("java.home"), "bin", "java").toString(), - "-Dfile.encoding=UTF-8", "-jar", "scouterctl.jar", arguments) + val process = ProcessBuilder(cliJavaCommand("scouterctl.jar", listOf("-Dfile.encoding=UTF-8")) + arguments) .directory(install.toFile()).redirectOutput(stdout.toFile()).redirectError(stderr.toFile()).apply { environment().remove("SCOUTERCTL_PASSWORD") environment()["SCOUTERCTL_CONFIG"] = config.toString() From 0507faf4ba288cc8fa63d398b58f58777b5315ce Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:34:19 +0900 Subject: [PATCH 08/17] =?UTF-8?q?docs:=20=F0=9F=93=9D=20Explain=20capture?= =?UTF-8?q?=20requirements=20for=20offline=20inspect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #76. --- README.ko.md | 3 +-- README.md | 3 +-- src/main/kotlin/dev/scouterctl/cli/Cli.kt | 5 ++++- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.ko.md b/README.ko.md index b28144f..4e8abdf 100644 --- a/README.ko.md +++ b/README.ko.md @@ -9,8 +9,7 @@ ## 주요 기능 - Collector·Agent의 상태, 설정, 실시간 지표와 트랜잭션·SQL·오류 프로파일 조회 -- 중단 후 이어서 받을 수 있는 캡처 파일 저장 -- 저장한 캡처의 검색·서비스별 요약·비교 +- 중단 후 이어서 받을 수 있는 캡처 파일 저장. 오프라인에서 저장한 캡처를 조회·검색·요약·비교하려면 수집할 때 `capture --include-profiles`를 사용합니다. [캡처 호환성](docs/capture-compatibility.md) - 스크립트와 다른 도구에서 사용할 수 있는 JSON·JSONL 출력 ## 시작하기 diff --git a/README.md b/README.md index e94ee59..82fd511 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,7 @@ Starting with diagnostics and data capture, the project aims to cover configurat ## Features - Inspect Collector and Agent status, settings and real-time metrics; query transactions, SQL and errors. -- Export resumable captures for offline investigation. -- Search, summarize and compare saved captures. +- Export resumable captures. Use `capture --include-profiles` when you plan to inspect, search, summarize or compare saved captures offline. See [capture compatibility](docs/capture-compatibility.md). - Use JSON/JSONL output in scripts and other tools. ## Quick start diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index b1814fb..f3b6a8c 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -36,7 +36,10 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut @Option(names = ["--before"], description = ["inspect compare: baseline v2/v3 capture directory"]) var before: Path? = null @Option(names = ["--after"], description = ["inspect compare: candidate v2/v3 capture directory"]) var after: Path? = null @Option(names = ["--resume"]) var resume: Boolean = false - @Option(names = ["--include-profiles"], description = ["Capture profiles in a v3 offline bundle"]) var includeProfiles: Boolean = false + @Option(names = ["--include-profiles"], description = [ + "Capture profiles in a v3 bundle for offline inspect.", + "Default capture writes an XLog-only v1 bundle.", + ]) var includeProfiles: Boolean = false @Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in xlog search, profile get, profile capture or inspect"]) var includeText: Boolean = false @Option(names = ["--limit"], description = ["inspect search/summary/compare: 1..1000 requests/services; default 20"]) var limit: Int? = null From a7af0229438e48060fa8061c580312c0a19cbb67 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:34:31 +0900 Subject: [PATCH 09/17] =?UTF-8?q?feat:=20=E2=9C=A8=20Show=20approved=20Age?= =?UTF-8?q?nt=20configuration=20values?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #64. --- docs/inspection.md | 2 +- .../adapters/collector/CollectorSource.kt | 25 ++++++++++++------ .../scouterctl/application/AgentInspection.kt | 7 ++++- .../dev/scouterctl/AgentInspectionTest.kt | 26 ++++++++++++------- src/test/kotlin/dev/scouterctl/WireFixture.kt | 14 +++++++++- 5 files changed, 54 insertions(+), 20 deletions(-) diff --git a/docs/inspection.md b/docs/inspection.md index 57ee2f0..86e4332 100644 --- a/docs/inspection.md +++ b/docs/inspection.md @@ -28,7 +28,7 @@ scouterctl agent config --object-hash 123456 --key net_collector_tcp_port Both commands go through the selected Collector. Status includes the Agent name, type, version and Collector-reported `alive` flag. When available, `jvm` contains Java/VM versions, vendor, OS name and architecture; arbitrary system properties and personal paths are excluded. `jvmAvailability` is `available`, `offline` when the Collector reports the Agent as down, or `unavailable` when no JVM data is returned. An unavailable JVM is not proof of a healthy application. A transport timeout returns exit 5 instead of a status result. -Configuration lists the Agent's runtime settings and defaults. Only selected numeric/boolean operational settings expose values; other current/default values are null with `redacted: true`, including when selected by `--key`. Neither command writes configuration. An unknown target, an offline Agent's configuration, unavailable configuration or a missing setting returns exit 7. Malformed responses return exit 6. The CLI remains on JDK 21 even when the Agent/application runs on Java 8. +Configuration lists the Agent's runtime settings and defaults. Visible settings include selected numeric and boolean controls, such as `trace_sql_parameter_max_length`, `profile_http_parameter_enabled` and `profile_http_querystring_enabled`, plus the text settings `net_collector_ip`, `obj_name` and `obj_type`. Other settings report null for current and default values with `redacted: true`, including when selected by `--key`. Neither command writes configuration. An unknown target, an offline Agent's configuration, unavailable configuration or a missing setting returns exit 7. Malformed responses return exit 6. The CLI remains on JDK 21 even when the Agent/application runs on Java 8. ## Real-time counters diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index 3d99f30..db67c46 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -4,6 +4,7 @@ import dev.scouterctl.core.* import dev.scouterctl.application.ObservationSource import dev.scouterctl.application.collectorSettingValueIsVisible import dev.scouterctl.application.agentSettingValueIsVisible +import dev.scouterctl.application.agentTextSettingValueIsVisible import scouter.io.DataInputX import scouter.io.DataOutputX @@ -135,7 +136,7 @@ class CollectorSource private constructor(private val socket: Socket, private va if (!agent.alive) throw OperationFailure(ErrorCode.NOT_FOUND, "Agent is not alive") val response = exchange(RequestCmd.LIST_CONFIGURE_WAS, MapPack().apply { put("objHash", objHash.toLong()) }) if (response.isEmpty()) throw OperationFailure(ErrorCode.NOT_FOUND, "Agent settings are unavailable") - AgentConfiguration(agent, parseSettings(response, ::agentSettingValueIsVisible)) + AgentConfiguration(agent, parseSettings(response, ::agentSettingValueIsVisible, ::agentTextSettingValueIsVisible)) } override fun counterHistory(query: CounterHistoryQuery): CounterHistory = collectorOperation { @@ -165,7 +166,11 @@ class CollectorSource private constructor(private val socket: Socket, private va return AgentIdentity(agent.objHash, agent.objName ?: "", agent.objType ?: "", agent.alive, agent.version ?: "unknown") } - private fun parseSettings(response: List, valueIsVisible: (String) -> Boolean): List { + private fun parseSettings( + response: List, + valueIsVisible: (String) -> Boolean, + textValueIsVisible: (String) -> Boolean = { false }, + ): List { val pack = response.singleOrNull() as? MapPack ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Expected one runtime settings response") fun items(key: String): ListValue = pack.get(key) as? ListValue @@ -177,10 +182,14 @@ class CollectorSource private constructor(private val socket: Socket, private va throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Inconsistent runtime settings lists") } val seen = mutableSetOf() - fun visibleValue(value: Value): String = when (value) { - is DecimalValue -> value.value.toString() - is BooleanValue -> value.value.toString() - else -> throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid operational setting value") + fun invalidValue(): Nothing = throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid operational setting value") + fun visibleValue(key: String, value: Value): String { + if (textValueIsVisible(key)) return (value as? TextValue)?.value ?: invalidValue() + return when (value) { + is DecimalValue -> value.value.toString() + is BooleanValue -> value.value.toString() + else -> invalidValue() + } } return (0 until keys.size()).map { index -> val key = (keys.get(index) as? TextValue)?.value @@ -189,8 +198,8 @@ class CollectorSource private constructor(private val socket: Socket, private va throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid or duplicate runtime setting key") } val visible = valueIsVisible(key) - RuntimeSetting(key, if (visible) visibleValue(values.get(index)) else null, - if (visible) visibleValue(defaults.get(index)) else null, !visible) + RuntimeSetting(key, if (visible) visibleValue(key, values.get(index)) else null, + if (visible) visibleValue(key, defaults.get(index)) else null, !visible) } } diff --git a/src/main/kotlin/dev/scouterctl/application/AgentInspection.kt b/src/main/kotlin/dev/scouterctl/application/AgentInspection.kt index 5e8425c..0dc2507 100644 --- a/src/main/kotlin/dev/scouterctl/application/AgentInspection.kt +++ b/src/main/kotlin/dev/scouterctl/application/AgentInspection.kt @@ -5,9 +5,14 @@ import dev.scouterctl.core.* private val visibleAgentSettings = setOf( "net_collector_tcp_port", "net_collector_udp_port", "xlog_sampling_enabled", "trace_sampling_enabled", "trace_interservice_enabled", "trace_auto_service_enabled", "profile_step_max_count", "profile_sql_max_count", + "trace_sql_parameter_max_length", "profile_http_parameter_enabled", "profile_http_querystring_enabled", ) -fun agentSettingValueIsVisible(key: String): Boolean = key in visibleAgentSettings +private val visibleAgentTextSettings = setOf("net_collector_ip", "obj_name", "obj_type") + +fun agentTextSettingValueIsVisible(key: String): Boolean = key in visibleAgentTextSettings + +fun agentSettingValueIsVisible(key: String): Boolean = key in visibleAgentSettings || agentTextSettingValueIsVisible(key) fun agentSettings(source: ObservationSource, objHash: Int, key: String?): AgentConfiguration { val configuration = source.agentSettings(objHash) diff --git a/src/test/kotlin/dev/scouterctl/AgentInspectionTest.kt b/src/test/kotlin/dev/scouterctl/AgentInspectionTest.kt index 046ee0b..ea6b16d 100644 --- a/src/test/kotlin/dev/scouterctl/AgentInspectionTest.kt +++ b/src/test/kotlin/dev/scouterctl/AgentInspectionTest.kt @@ -28,14 +28,21 @@ class AgentInspectionTest : FunSpec({ val result = cli.run(server, listOf("agent", "config", "--object-hash", "42")) result.exit shouldBe 0 val settings = result.data()["settings"]!!.jsonArray.map { it.jsonObject } - settings.map { it["key"]!!.jsonPrimitive.content } shouldBe listOf("db_dir", "net_collector_tcp_port", "smtp_password") - val port = settings.single { it["key"] == JsonPrimitive("net_collector_tcp_port") } - port["value"] shouldBe JsonPrimitive("6100") - port["redacted"] shouldBe JsonPrimitive(false) - settings.filter { it != port }.forEach { - it["value"] shouldBe JsonNull - it["defaultValue"] shouldBe JsonNull - it["redacted"] shouldBe JsonPrimitive(true) + val visible = mapOf( + "net_collector_tcp_port" to ("6100" to "6100"), + "trace_sql_parameter_max_length" to ("0" to "20"), + "profile_http_parameter_enabled" to ("true" to "false"), + "profile_http_querystring_enabled" to ("true" to "false"), + "net_collector_ip" to ("collector.example.test" to "127.0.0.1"), + "obj_name" to ("fixture" to ""), + "obj_type" to ("java" to ""), + ) + settings.map { it["key"]!!.jsonPrimitive.content } shouldBe (visible.keys + listOf("db_dir", "smtp_password")).sorted() + settings.forEach { setting -> + val expected = visible[setting["key"]!!.jsonPrimitive.content] + setting["value"] shouldBe (expected?.first?.let(::JsonPrimitive) ?: JsonNull) + setting["defaultValue"] shouldBe (expected?.second?.let(::JsonPrimitive) ?: JsonNull) + setting["redacted"] shouldBe JsonPrimitive(expected == null) } result.output shouldNotContain "private-" result.output shouldNotContain "/private/" @@ -64,7 +71,8 @@ class AgentInspectionTest : FunSpec({ AgentFailureCase(Fault.AGENT_DUPLICATE, "status", "42", 6), AgentFailureCase(Fault.AGENT_JVM_MALFORMED, "status", "42", 6), AgentFailureCase(Fault.CONFIG_MALFORMED, "config", "42", 6), - AgentFailureCase(Fault.CONFIG_INVALID_VISIBLE, "config", "42", 6))) { + AgentFailureCase(Fault.CONFIG_INVALID_VISIBLE, "config", "42", 6), + AgentFailureCase(Fault.CONFIG_INVALID_AGENT_TEXT, "config", "42", 6))) { WireFixture(fault).use { server -> val result = cli.run(server, listOf("agent", command, "--object-hash", hash)) result.exit shouldBe exit diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index 3ce4536..82f0743 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -29,7 +29,7 @@ enum class Fault { HISTORY_EMPTY, HISTORY_MALFORMED, HISTORY_TIME_TYPE, HISTORY_OUTSIDE, HISTORY_NAN, HISTORY_TARGET_DUPLICATE, COUNTER_MALFORMED, COUNTER_DUPLICATE, COUNTER_NAN, COUNTER_XML_DTD, COUNTER_EMPTY, COUNTER_CUSTOM, AGENT_OFFLINE, AGENT_NO_RESPONSE, AGENT_JVM_MALFORMED, AGENT_DUPLICATE, - NONE, STATUS_MALFORMED, CONFIG_MALFORMED, CONFIG_DUPLICATE, CONFIG_INVALID_VISIBLE, VERSION, AUTH, SESSION, MISSING_META, INVALID_META, STALLED_CURSOR, DISCONNECT, EMPTY_OBJECTS, + NONE, STATUS_MALFORMED, CONFIG_MALFORMED, CONFIG_DUPLICATE, CONFIG_INVALID_VISIBLE, CONFIG_INVALID_AGENT_TEXT, VERSION, AUTH, SESSION, MISSING_META, INVALID_META, STALLED_CURSOR, DISCONNECT, EMPTY_OBJECTS, STRUCTURAL_ONLY, MISSING_TEXT, EMPTY_TEXT, MANY_SQL, TEXT_WRONG_TYPE, TEXT_WRONG_HASH, TEXT_DUPLICATE, TEXT_PACK, TEXT_DISCONNECT, TEXT_SESSION, XLOG_MISSING, XLOG_WRONG_ID, PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, @@ -210,16 +210,28 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas newList("key").apply { add("smtp_password"); add(if (agent) "net_collector_tcp_port" else "net_tcp_listen_port"); add("db_dir") if (fault == Fault.CONFIG_DUPLICATE) add("db_dir") + if (agent) { + add("trace_sql_parameter_max_length"); add("profile_http_parameter_enabled") + add("profile_http_querystring_enabled"); add("net_collector_ip"); add("obj_name"); add("obj_type") + } } newList("value").apply { add("private-config-marker"); add(6100L); add("/private/collector/data") if (fault == Fault.CONFIG_DUPLICATE) add("duplicate-private-marker") if (fault == Fault.CONFIG_INVALID_VISIBLE) set(1, scouter.lang.value.TextValue("private-visible-marker")) + if (agent) { + add(0L); add(true); add(true); add("collector.example.test") + if (fault == Fault.CONFIG_INVALID_AGENT_TEXT) add(42L) else add("fixture") + add("java") + } } newList("default").apply { add("private-default-marker"); add(6100L) if (fault != Fault.CONFIG_MALFORMED) add("/private/default") if (fault == Fault.CONFIG_DUPLICATE) add("duplicate-private-default") + if (agent) { + add(20L); add(false); add(false); add("127.0.0.1"); add(""); add("") + } } }) } From 53d4c1b5ea2adf9d3d403b552b9ff9f353ebd3da Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Thu, 12 Nov 2026 02:29:04 +0900 Subject: [PATCH 10/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Explain=20invalid?= =?UTF-8?q?=20capture=20directory=20paths?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Report actionable path errors for transaction capture and offline inspection, including regular-file ancestors, while preserving atomic creation, existing data and integrity failures. Refs #75 --- docs/captures.md | 4 +- .../adapters/files/CaptureDirectory.kt | 50 +++++++++++++ .../scouterctl/adapters/files/CaptureStore.kt | 3 +- .../adapters/files/EvidenceStore.kt | 5 +- .../kotlin/dev/scouterctl/CapturePathTest.kt | 74 +++++++++++++++++++ 5 files changed, 130 insertions(+), 6 deletions(-) create mode 100644 src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt create mode 100644 src/test/kotlin/dev/scouterctl/CapturePathTest.kt diff --git a/docs/captures.md b/docs/captures.md index 026761b..cfabe71 100644 --- a/docs/captures.md +++ b/docs/captures.md @@ -29,7 +29,7 @@ scouterctl capture \ Add `--include-text` to `xlog search` to resolve service names and error text without downloading profiles. Error text can contain sensitive values, including bound values or values from existing database rows. -The destination directory must not exist for a new capture. If the export is interrupted or partially completes, add `--resume` to the same capture command. Each query covers one date and one object over `[from,to)`. Set `--timezone` to the Collector's actual time zone. +For a new transaction capture, choose an unused destination under an existing parent directory. An occupied destination or a missing or invalid parent returns `INVALID_ARGUMENT` (exit 2). If the export is interrupted or partially completes, add `--resume` to the same capture command. Each query covers one date and one object over `[from,to)`. Set `--timezone` to the Collector's actual time zone. The example saves profiles in a v3 bundle. Add `--include-text` to the capture command only when you intend to store potentially sensitive service, SQL and error text; the default page size is 25 and the maximum is 100. Resume v3 with the same flags and query. Existing v2 bundles remain readable but cannot be resumed by this version; see [Capture compatibility](capture-compatibility.md). Missing profiles are recorded and retried on resume; connection failures preserve earlier page checkpoints. See [Capture recovery](capture-recovery.md) for interruption handling and the limits of recovery. @@ -45,7 +45,7 @@ scouterctl inspect --directory captures/incident-001 \ --txid 9223372036854775806 ``` -Use `--include-text` when the capture was created with stored text and you want to display it. Requesting text that was not stored returns `2`. Inspect verifies all referenced pages and returns `8` with data for partial captures or unavailable profiles. A missing transaction returns `7`; damaged files return `10`. It supports v2 and v3, with no implicit v1 migration. +Use `--include-text` when the capture was created with stored text and you want to display it. Requesting text that was not stored returns `2`. Inspect verifies all referenced pages and returns `8` with data for partial captures or unavailable profiles. A missing transaction returns `7`. A missing capture directory or a path to a file returns `INVALID_ARGUMENT` (exit 2). Damaged capture files return `10`. It supports v2 and v3, with no implicit v1 migration. To find a transaction ID first, search the same bundle offline: diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt new file mode 100644 index 0000000..2c31595 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt @@ -0,0 +1,50 @@ +package dev.scouterctl.adapters.files + +import dev.scouterctl.core.ErrorCode +import dev.scouterctl.core.OperationFailure +import java.nio.file.FileAlreadyExistsException +import java.nio.file.FileSystemException +import java.nio.file.Files +import java.nio.file.NoSuchFileException +import java.nio.file.NotDirectoryException +import java.nio.file.Path +import java.nio.file.attribute.BasicFileAttributes + +private const val MISSING_PARENT = "Capture parent directory does not exist. Create it before starting a capture." + +internal fun createCaptureDirectory(directory: Path): Unit = fileOperation { + directory.toAbsolutePath().parent?.let { parent -> + requireDirectory(parent, MISSING_PARENT, "Capture parent path is not a directory. Choose an existing parent directory.") + } + + try { + Files.createDirectory(directory) + } catch (_: FileAlreadyExistsException) { + throw OperationFailure( + ErrorCode.INVALID_ARGUMENT, + "Capture destination already exists. Choose an unused path, or use --resume for a compatible capture." + ) + } catch (_: NoSuchFileException) { + throw OperationFailure(ErrorCode.INVALID_ARGUMENT, MISSING_PARENT) + } +} + +internal fun requireCaptureDirectory(directory: Path) = requireDirectory( + directory, + "Capture directory does not exist. Specify the path to an existing capture.", + "Capture path is not a directory. Specify the path to an existing capture." +) + +private fun requireDirectory(path: Path, missing: String, invalid: String) { + val attributes = try { + Files.readAttributes(path, BasicFileAttributes::class.java) + } catch (_: NoSuchFileException) { + throw OperationFailure(ErrorCode.INVALID_ARGUMENT, missing) + } catch (_: NotDirectoryException) { + throw OperationFailure(ErrorCode.INVALID_ARGUMENT, invalid) + } catch (error: FileSystemException) { + path.toAbsolutePath().parent?.let { requireDirectory(it, missing, invalid) } + throw error + } + if (!attributes.isDirectory) throw OperationFailure(ErrorCode.INVALID_ARGUMENT, invalid) +} diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt index 2910743..059680b 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt @@ -12,8 +12,9 @@ import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.security.MessageDigest class CaptureStore(private val directory: Path) : CaptureRepository { - override fun create() = fileOperation { Files.createDirectory(directory); Unit } + override fun create() = createCaptureDirectory(directory) override fun load(): CaptureManifest = fileOperation { + requireCaptureDirectory(directory) val content = Files.readString(directory.resolve("manifest.json")) requireCaptureSchema(content, "scouterctl.capture/v1") json.decodeFromString(content).toModel() diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt index 1b8f0e4..19f32ab 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt @@ -13,11 +13,10 @@ import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.security.MessageDigest class EvidenceStore(private val directory: Path) : EvidenceRepository { - override fun create(): Unit = fileOperation { - Files.createDirectory(directory) - } + override fun create() = createCaptureDirectory(directory) override fun load(): EvidenceManifest = fileOperation { + requireCaptureDirectory(directory) val content = read("manifest.json", 8_388_608) requireCaptureSchema(content, "scouterctl.capture/v2", "scouterctl.capture/v3") val manifest = json.decodeFromString(content).toModel() diff --git a/src/test/kotlin/dev/scouterctl/CapturePathTest.kt b/src/test/kotlin/dev/scouterctl/CapturePathTest.kt new file mode 100644 index 0000000..035afd4 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CapturePathTest.kt @@ -0,0 +1,74 @@ +package dev.scouterctl + +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import kotlinx.serialization.json.jsonObject +import kotlinx.serialization.json.jsonPrimitive +import java.nio.file.Files + +class CapturePathTest : FunSpec({ + test("inspect distinguishes missing and invalid paths from a damaged bundle") { + val work = tempdir().toPath() + Files.writeString(work.resolve("not-a-directory"), "ordinary file") + Files.createDirectory(work.resolve("empty-capture")) + val cli = ContractCli(work) + val inspect = listOf("inspect", "--txid", "1", "--directory") + + cli.run(inspect + "missing-capture", password = null).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture directory does not exist. Specify the path to an existing capture." + ) + for (path in listOf("not-a-directory", "not-a-directory/child", "not-a-directory/child/grandchild")) { + cli.run(inspect + path, password = null).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture path is not a directory. Specify the path to an existing capture." + ) + } + cli.run(inspect + "empty-capture", password = null).shouldFailWith( + 10, "IO_ERROR", "Capture integrity or record validation failed" + ) + } + + for (includeProfiles in listOf(false, true)) { + test("new capture reports occupied destinations and invalid parents with include-profiles=$includeProfiles") { + val work = tempdir().toPath() + val existing = Files.createDirectory(work.resolve("existing")) + val marker = Files.writeString(existing.resolve("marker"), "existing capture data") + val occupiedFile = Files.writeString(work.resolve("occupied-file"), "existing file") + val cli = ContractCli(work) + val args = listOf("capture") + evidenceWindow + if (includeProfiles) listOf("--include-profiles") else emptyList() + val occupied = "Capture destination already exists. Choose an unused path, or use --resume for a compatible capture." + + WireFixture().use { server -> + cli.run(server, args + listOf("--directory", "existing")).shouldFailWith(2, "INVALID_ARGUMENT", occupied) + cli.run(server, args + listOf("--directory", "occupied-file")).shouldFailWith(2, "INVALID_ARGUMENT", occupied) + cli.run(server, args + listOf("--directory", "missing-parent/capture")).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture parent directory does not exist. Create it before starting a capture." + ) + for (parent in listOf("occupied-file", "occupied-file/child", "occupied-file/child/grandchild")) { + cli.run(server, args + listOf("--directory", "$parent/capture")).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture parent path is not a directory. Choose an existing parent directory." + ) + cli.run(server, args + listOf("--resume", "--directory", parent)).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture path is not a directory. Specify the path to an existing capture." + ) + } + cli.run(server, args + listOf("--resume", "--directory", "missing-capture")).shouldFailWith( + 2, "INVALID_ARGUMENT", "Capture directory does not exist. Specify the path to an existing capture." + ) + } + + Files.list(existing).use { it.toList() } shouldBe listOf(marker) + Files.readString(marker) shouldBe "existing capture data" + Files.readString(occupiedFile) shouldBe "existing file" + Files.notExists(work.resolve("missing-parent")) shouldBe true + Files.notExists(work.resolve("missing-capture")) shouldBe true + } + } +}) + +private fun Outcome.shouldFailWith(expectedExit: Int, code: String, message: String) { + exit shouldBe expectedExit + val error = events.single().getValue("error").jsonObject + error.getValue("code").jsonPrimitive.content shouldBe code + error.getValue("message").jsonPrimitive.content shouldBe message +} From da33d93593db66fb2a43cb950c72b8179a3d0523 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Thu, 12 Nov 2026 10:15:29 +0900 Subject: [PATCH 11/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Retry=20denied=20Wi?= =?UTF-8?q?ndows=20file=20replacements?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retry transient Windows access-denied atomic replacements and distinguish local file errors from Collector transport errors. Preserve committed checkpoints and document recovery limits. Refs #58 --- QUICKSTART.ko.md | 2 + QUICKSTART.md | 2 + docs/capture-recovery.md | 6 ++ docs/cli-contract.md | 2 + .../adapters/collector/CollectorOperation.kt | 2 +- .../adapters/files/AtomicReplacement.kt | 25 ++++++++ .../scouterctl/adapters/files/CaptureStore.kt | 4 +- .../scouterctl/adapters/files/ContextStore.kt | 4 +- .../adapters/files/CounterCaptureStore.kt | 4 +- .../adapters/files/EvidenceStore.kt | 4 +- .../adapters/files/FileOperation.kt | 2 +- .../kotlin/dev/scouterctl/CaptureWriteTest.kt | 62 +++++++++++++++++++ .../dev/scouterctl/ConnectionDiagnostic.kt | 1 + .../scouterctl/ConnectionDiagnosticTest.kt | 4 +- .../WindowsAtomicReplacementTest.kt | 40 ++++++++++++ 15 files changed, 149 insertions(+), 15 deletions(-) create mode 100644 src/main/kotlin/dev/scouterctl/adapters/files/AtomicReplacement.kt create mode 100644 src/test/kotlin/dev/scouterctl/CaptureWriteTest.kt create mode 100644 src/test/kotlin/dev/scouterctl/WindowsAtomicReplacementTest.kt diff --git a/QUICKSTART.ko.md b/QUICKSTART.ko.md index 84a24fa..0d104be 100644 --- a/QUICKSTART.ko.md +++ b/QUICKSTART.ko.md @@ -55,6 +55,8 @@ scouterctl capture --object-hash 123456 --date 20261107 \ 중단되거나 일부만 저장됐으면 같은 명령에 `--resume`을 붙여 재개합니다. 서비스명·SQL·오류 원문을 저장하려는 경우에만 캡처할 때 `--include-text`를 추가하세요. 민감한 애플리케이션 정보가 포함될 수 있습니다. 자세한 사용법은 [캡처와 오프라인 분석(영문)](docs/captures.md)에 있습니다. +Windows에서는 일시적인 파일 교체 거부를 짧게 재시도합니다. 로컬 파일 오류가 계속되면 캡처를 사용하는 프로그램을 닫고 권한과 디스크 여유 공간을 확인하세요. 정상적으로 저장된 manifest가 있을 때만 재개하고, 없으면 실패한 디렉터리를 보존한 뒤 새 경로로 캡처합니다. 자세한 절차는 [캡처 복구 안내(영문)](docs/capture-recovery.md)를 참고하세요. + ## 4. Collector 없이 분석 ```sh diff --git a/QUICKSTART.md b/QUICKSTART.md index fd7e011..5581358 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -53,6 +53,8 @@ The capture result reports `storageDateVerification: "unverified"`. Check the su Repeat the same command with `--resume` after interruption or partial completion. Add `--include-text` at capture time only if you intend to store potentially sensitive service names, SQL and error text. See [capture and offline analysis](docs/captures.md). +Windows briefly retries temporary file replacement denial. If a local file error persists, close programs using the capture and check permissions and free disk space. Resume only when a valid committed manifest exists; otherwise preserve the failed directory and choose a new destination. See [capture recovery](docs/capture-recovery.md). + ## 4. Inspect without a Collector ```sh diff --git a/docs/capture-recovery.md b/docs/capture-recovery.md index f5b6eba..963acbf 100644 --- a/docs/capture-recovery.md +++ b/docs/capture-recovery.md @@ -20,6 +20,12 @@ A closed transport can report `IO_ERROR` (exit 10), whereas a refused connection Pages that were already committed are validated before resume. An incomplete page is fetched again, including profiles already read but not committed. Temporary files not referenced by the manifest are not evidence. Repeating resume on a complete capture preserves its data. +## Local file failures on Windows + +Windows can deny an atomic file replacement while another program holds the destination open. The CLI briefly retries access-denied replacements, preserving the committed file and the atomic replacement operation. Persistent denial still returns `IO_ERROR` (exit 10); other filesystem failures are not retried. An access-denied error can also mean insufficient permissions. + +The message distinguishes local file I/O from Collector transport I/O. Close programs using the capture and check its path, permissions and available disk space. If the directory contains a valid committed manifest, rerun the original command with `--resume` after correcting the cause. If the first manifest could not be committed, preserve that directory and start a new capture in an unused directory. Do not promote a `.partial` file to a manifest or remove referenced pages to force recovery. + ## Recovery limits Recovery preserves the records available in the fixed query window and the committed capture. It cannot recover telemetry that an agent or Collector lost during an outage, expired retention, deleted databases, or sampled requests. `executionComplete` does not certify complete upstream traffic; `upstreamCoverage` and `sampling` remain `unknown`. diff --git a/docs/cli-contract.md b/docs/cli-contract.md index 3e6cfbb..dee3e2b 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -46,6 +46,8 @@ The example message is illustrative. `error.code` is the machine-readable identi A mid-response EOF currently maps to `IO_ERROR` (10), whereas socket exceptions/timeouts map to `CONNECTION_FAILED` (5). Exit 8 therefore needs both the exit status and the envelope: it can carry available data or describe a failure without data. Treat unrecognized nonzero exits as failures rather than guessing a known category. +Local file failures and Collector transport I/O have distinct human-readable messages while retaining `IO_ERROR` (10). Integrity errors retain their specific messages. See [capture recovery](capture-recovery.md) before retrying a failed file write; a `.partial` file is not a committed checkpoint. + ## Streaming commands `xlog search` uses `scouterctl.stream/v1` on its first `start` event: diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorOperation.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorOperation.kt index 36a4fa0..b46248f 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorOperation.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorOperation.kt @@ -20,7 +20,7 @@ internal inline fun collectorOperation(block: () -> T): T = try { throw OperationFailure(ErrorCode.CONNECTION_FAILED, "Collector connection failed; check --host, --port and TCP access") } catch (_: IOException) { - throw OperationFailure(ErrorCode.IO_ERROR, "Transport or artifact I/O failed") + throw OperationFailure(ErrorCode.IO_ERROR, "Collector transport I/O failed; retry the query or check the Collector connection") } catch (_: IllegalArgumentException) { throw UnexpectedFailure(ErrorCode.INVALID_ARGUMENT, "Invalid argument or manifest") } catch (_: java.time.DateTimeException) { diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/AtomicReplacement.kt b/src/main/kotlin/dev/scouterctl/adapters/files/AtomicReplacement.kt new file mode 100644 index 0000000..df898bc --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/files/AtomicReplacement.kt @@ -0,0 +1,25 @@ +package dev.scouterctl.adapters.files + +import java.nio.file.AccessDeniedException +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.StandardCopyOption.ATOMIC_MOVE +import java.nio.file.StandardCopyOption.REPLACE_EXISTING + +internal fun replaceAtomically(partial: Path, target: Path) { + val attempts = if (System.getProperty("os.name").startsWith("Windows")) 5 else 1 + repeat(attempts) { attempt -> + try { + Files.move(partial, target, ATOMIC_MOVE, REPLACE_EXISTING) + return + } catch (failure: AccessDeniedException) { + if (attempt == attempts - 1) throw failure + try { + Thread.sleep(50) + } catch (_: InterruptedException) { + Thread.currentThread().interrupt() + throw failure + } + } + } +} diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt index 059680b..756b4a8 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureStore.kt @@ -7,8 +7,6 @@ import dev.scouterctl.application.CaptureRepository import dev.scouterctl.adapters.serialization.* import java.nio.file.Files import java.nio.file.Path -import java.nio.file.StandardCopyOption.ATOMIC_MOVE -import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.security.MessageDigest class CaptureStore(private val directory: Path) : CaptureRepository { @@ -39,7 +37,7 @@ class CaptureStore(private val directory: Path) : CaptureRepository { private fun atomicWrite(path: Path, content: String) { val partial = path.resolveSibling(path.fileName.toString() + ".partial") Files.writeString(partial, content) - Files.move(partial, path, ATOMIC_MOVE, REPLACE_EXISTING) + replaceAtomically(partial, path) } private fun digest(bytes: ByteArray): String = MessageDigest.getInstance("SHA-256") .digest(bytes).joinToString("") { "%02x".format(it) } diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/ContextStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/ContextStore.kt index a45a3ba..8096652 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/ContextStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/ContextStore.kt @@ -10,8 +10,6 @@ import java.nio.channels.FileChannel import java.nio.file.Files import java.nio.file.NoSuchFileException import java.nio.file.Path -import java.nio.file.StandardCopyOption.ATOMIC_MOVE -import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.nio.file.StandardOpenOption.CREATE import java.nio.file.StandardOpenOption.WRITE import java.nio.file.attribute.PosixFilePermissions @@ -44,7 +42,7 @@ class ContextStore(private val path: Path) : ContextRepository { val temporary = Files.createTempFile(parent, ".scouterctl-", ".tmp", *attributes) try { Files.writeString(temporary, json.encodeToString(updated.toDto()) + "\n") - Files.move(temporary, path, ATOMIC_MOVE, REPLACE_EXISTING) + replaceAtomically(temporary, path) } finally { Files.deleteIfExists(temporary) } return@fileOperation updated } diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/CounterCaptureStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/CounterCaptureStore.kt index 304bf8c..3bc1c67 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/CounterCaptureStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/CounterCaptureStore.kt @@ -10,8 +10,6 @@ import java.nio.channels.OverlappingFileLockException import java.nio.file.Files import java.nio.file.LinkOption.NOFOLLOW_LINKS import java.nio.file.Path -import java.nio.file.StandardCopyOption.ATOMIC_MOVE -import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.nio.file.StandardOpenOption.* import java.security.MessageDigest import java.util.Locale @@ -109,7 +107,7 @@ class CounterCaptureStore(private val directory: Path) : CounterCaptureRepositor Files.newOutputStream(partial, CREATE, TRUNCATE_EXISTING, WRITE, NOFOLLOW_LINKS).use { it.write(content.toByteArray(Charsets.UTF_8)) } - Files.move(partial, directory.resolve(name), ATOMIC_MOVE, REPLACE_EXISTING) + replaceAtomically(partial, directory.resolve(name)) } private fun digest(bytes: ByteArray): String = MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(Locale.ROOT, it) } diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt b/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt index 19f32ab..ca14040 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/EvidenceStore.kt @@ -8,8 +8,6 @@ import kotlinx.serialization.encodeToString import java.nio.file.Files import java.nio.file.LinkOption.NOFOLLOW_LINKS import java.nio.file.Path -import java.nio.file.StandardCopyOption.ATOMIC_MOVE -import java.nio.file.StandardCopyOption.REPLACE_EXISTING import java.security.MessageDigest class EvidenceStore(private val directory: Path) : EvidenceRepository { @@ -89,7 +87,7 @@ class EvidenceStore(private val directory: Path) : EvidenceRepository { java.nio.file.StandardOpenOption.TRUNCATE_EXISTING, java.nio.file.StandardOpenOption.WRITE).use { it.write(content.toByteArray(Charsets.UTF_8)) } - Files.move(partial, directory.resolve(name), ATOMIC_MOVE, REPLACE_EXISTING) + replaceAtomically(partial, directory.resolve(name)) } private fun digest(bytes: ByteArray): String = MessageDigest.getInstance("SHA-256") diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/FileOperation.kt b/src/main/kotlin/dev/scouterctl/adapters/files/FileOperation.kt index 0040416..914af9f 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/FileOperation.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/FileOperation.kt @@ -9,7 +9,7 @@ internal inline fun fileOperation(block: () -> T): T = try { } catch (error: OperationFailure) { throw error } catch (_: IOException) { - throw OperationFailure(ErrorCode.IO_ERROR, "Transport or artifact I/O failed") + throw OperationFailure(ErrorCode.IO_ERROR, "Local file I/O failed; check the path, permissions and available disk space") } catch (_: IllegalArgumentException) { throw UnexpectedFailure(ErrorCode.INVALID_ARGUMENT, "Invalid argument or manifest") } catch (_: java.time.DateTimeException) { diff --git a/src/test/kotlin/dev/scouterctl/CaptureWriteTest.kt b/src/test/kotlin/dev/scouterctl/CaptureWriteTest.kt new file mode 100644 index 0000000..231809f --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CaptureWriteTest.kt @@ -0,0 +1,62 @@ +package dev.scouterctl + +import com.sun.nio.file.ExtendedOpenOption.NOSHARE_DELETE +import dev.scouterctl.adapters.files.CaptureStore +import dev.scouterctl.adapters.files.EvidenceStore +import dev.scouterctl.core.ErrorCode +import dev.scouterctl.core.OperationFailure +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import java.nio.channels.FileChannel +import java.nio.file.Files +import java.nio.file.StandardOpenOption.READ +import java.nio.file.attribute.PosixFilePermissions + +class CaptureWriteTest : FunSpec({ + for (includeProfiles in listOf(false, true)) { + test("denied checkpoint replacement preserves committed bytes and identifies local I/O with include-profiles=$includeProfiles") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + val cli = ContractCli(work) + WireFixture().use { server -> + val args = listOf("capture", "--directory", directory.toString()) + evidenceWindow + + if (includeProfiles) listOf("--include-profiles") else emptyList() + cli.run(server, args).exit shouldBe 0 + } + val manifest = directory.resolve("manifest.json") + val partial = directory.resolve("manifest.json.partial") + val before = Files.readString(manifest) + Files.writeString(partial, "prior partial") + val save: () -> Unit = if (includeProfiles) { + val store = EvidenceStore(directory) + val next = store.load().copy(executionComplete = false) + val write = { store.save(next) } + write + } else { + val store = CaptureStore(directory) + val next = store.load().copy(executionComplete = false) + val write = { store.save(next) } + write + } + val checkFailure = { + val failure = shouldThrow { save() } + failure.code shouldBe ErrorCode.IO_ERROR + failure.message shouldBe "Local file I/O failed; check the path, permissions and available disk space" + Files.readString(manifest) shouldBe before + Files.exists(partial) shouldBe true + } + if (System.getProperty("os.name").startsWith("Windows")) { + FileChannel.open(manifest, READ, NOSHARE_DELETE).use { checkFailure() } + } else { + val permissions = Files.getPosixFilePermissions(directory) + Files.setPosixFilePermissions(directory, PosixFilePermissions.fromString("r-x------")) + try { checkFailure() } finally { Files.setPosixFilePermissions(directory, permissions) } + } + save() + Files.exists(partial) shouldBe false + Files.readString(manifest).contains("\"executionComplete\":false") shouldBe true + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/ConnectionDiagnostic.kt b/src/test/kotlin/dev/scouterctl/ConnectionDiagnostic.kt index ca60c47..74b872a 100644 --- a/src/test/kotlin/dev/scouterctl/ConnectionDiagnostic.kt +++ b/src/test/kotlin/dev/scouterctl/ConnectionDiagnostic.kt @@ -18,6 +18,7 @@ internal fun connectionDiagnostic(exit: Int?, elapsedMs: Long, output: String, s "Collector timeout; check TCP access and --timeout-ms; upstream cancellation is not guaranteed" -> "socket_timeout" "Collector connection failed; check --host, --port and TCP access" -> "socket_failure" "Transport or artifact I/O failed" -> "transport_or_artifact_io" + "Collector transport I/O failed; retry the query or check the Collector connection" -> "collector_transport_io" else -> "unclassified" } return buildJsonObject { diff --git a/src/test/kotlin/dev/scouterctl/ConnectionDiagnosticTest.kt b/src/test/kotlin/dev/scouterctl/ConnectionDiagnosticTest.kt index ff8e1f7..57b3bd3 100644 --- a/src/test/kotlin/dev/scouterctl/ConnectionDiagnosticTest.kt +++ b/src/test/kotlin/dev/scouterctl/ConnectionDiagnosticTest.kt @@ -36,7 +36,9 @@ class ConnectionDiagnosticTest : FunSpec({ val result = ContractCli(tempdir().toPath()).run(server, profile) result.exit shouldBe 10 val diagnostic = requireNotNull(result.diagnostic) - diagnostic["error"]!!.jsonObject["reason"] shouldBe JsonPrimitive("transport_or_artifact_io") + result.events.single()["error"]!!.jsonObject["message"] shouldBe + JsonPrimitive("Collector transport I/O failed; retry the query or check the Collector connection") + diagnostic["error"]!!.jsonObject["reason"] shouldBe JsonPrimitive("collector_transport_io") diagnostic["fixture"]!!.jsonObject["command"] shouldBe JsonPrimitive("dictionary") diagnostic.toString() shouldNotContain "secret-token" diagnostic.toString() shouldNotContain "private-address" diff --git a/src/test/kotlin/dev/scouterctl/WindowsAtomicReplacementTest.kt b/src/test/kotlin/dev/scouterctl/WindowsAtomicReplacementTest.kt new file mode 100644 index 0000000..7d60283 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/WindowsAtomicReplacementTest.kt @@ -0,0 +1,40 @@ +package dev.scouterctl + +import com.sun.nio.file.ExtendedOpenOption.NOSHARE_DELETE +import dev.scouterctl.adapters.files.replaceAtomically +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import java.io.IOException +import java.nio.channels.FileChannel +import java.nio.file.Files +import java.nio.file.StandardOpenOption.READ +import java.util.concurrent.CompletableFuture +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit.MILLISECONDS +import java.util.concurrent.TimeUnit.SECONDS + +class WindowsAtomicReplacementTest : FunSpec({ + if (System.getProperty("os.name").startsWith("Windows")) { + test("replacement succeeds after a temporary Windows handle releases the committed file") { + val work = tempdir().toPath() + val target = Files.writeString(work.resolve("manifest.json"), "committed") + val partial = Files.writeString(work.resolve("manifest.json.partial"), "replacement") + val released = CompletableFuture() + + FileChannel.open(target, READ, NOSHARE_DELETE).use { held -> + Executors.newSingleThreadScheduledExecutor().use { scheduler -> + scheduler.schedule({ + try { held.close(); released.complete(Unit) } + catch (failure: IOException) { released.completeExceptionally(failure) } + }, 100, MILLISECONDS) + replaceAtomically(partial, target) + released.get(5, SECONDS) + } + } + + Files.readString(target) shouldBe "replacement" + Files.exists(partial) shouldBe false + } + } +}) From 22c81fa3f176a18d825567bfd40293b915052cc3 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Thu, 12 Nov 2026 12:55:39 +0900 Subject: [PATCH 12/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Improve=20command?= =?UTF-8?q?=20help=20and=20option=20diagnostics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Provide command-specific help, clean redirected and NO_COLOR output, and safe option-name diagnostics while preserving CLI execution contracts. Refs #65 --- QUICKSTART.ko.md | 1 + QUICKSTART.md | 1 + docs/cli-contract.md | 3 + .../dev/scouterctl/bootstrap/Bootstrap.kt | 6 +- .../dev/scouterctl/cli/ArgumentFailure.kt | 21 ++++ src/main/kotlin/dev/scouterctl/cli/Cli.kt | 2 +- .../kotlin/dev/scouterctl/cli/CommandGuide.kt | 70 +++++++++++ .../kotlin/dev/scouterctl/cli/CommandHelp.kt | 36 ++++++ .../dev/scouterctl/cli/CommandInterpreter.kt | 38 +++--- .../kotlin/dev/scouterctl/CliUsabilityTest.kt | 112 ++++++++++++++++++ src/test/kotlin/dev/scouterctl/TerminalCli.kt | 9 +- 11 files changed, 272 insertions(+), 27 deletions(-) create mode 100644 src/main/kotlin/dev/scouterctl/cli/ArgumentFailure.kt create mode 100644 src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt create mode 100644 src/main/kotlin/dev/scouterctl/cli/CommandHelp.kt create mode 100644 src/test/kotlin/dev/scouterctl/CliUsabilityTest.kt diff --git a/QUICKSTART.ko.md b/QUICKSTART.ko.md index 0d104be..dd0d882 100644 --- a/QUICKSTART.ko.md +++ b/QUICKSTART.ko.md @@ -7,6 +7,7 @@ JDK 21과 [운영체제에 맞는 ZIP](docs/ko/installation.md)을 설치한 뒤 ```sh scouterctl --version scouterctl --help +scouterctl capture --help ``` 여러 줄로 적은 명령은 Bash 예시입니다. Windows에서는 각 명령을 한 줄로 입력하면 같은 옵션을 사용할 수 있습니다. diff --git a/QUICKSTART.md b/QUICKSTART.md index 5581358..a01e996 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -7,6 +7,7 @@ Install the [Linux x64 or Windows x64 command package](docs/installation.md) wit ```sh scouterctl --version scouterctl --help +scouterctl capture --help ``` ## 1. Connect to a Collector diff --git a/docs/cli-contract.md b/docs/cli-contract.md index dee3e2b..6f685e2 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -8,6 +8,7 @@ This guide describes the output consumed by scripts and other tools. Output sche - Command and argument errors also go to stdout as JSON. Do not assume stderr is the error API. Keep stdout and stderr separate when parsing. - Explicit interactive password input uses a terminal and a human-readable prompt on stderr. Automation should provide `SCOUTERCTL_PASSWORD` and use `--no-input`; it must not depend on prompt text. - `--help` and `--version` are plain-text stdout with exit 0, outside the JSON format. Help layout and diagnostic messages may change. Parse error codes rather than message text. +- `scouterctl COMMAND --help` shows that command's required and accepted options, combinations and an example (for example `scouterctl capture --help` or `scouterctl counter history --help`). A command-family help request such as `scouterctl counter --help` lists its commands. Help contains no ANSI escapes when output is redirected or `NO_COLOR` is set, including an empty value. - These formats apply after the JVM and CLI start successfully. A missing/incompatible JVM, process termination or an unwritable output stream can prevent a valid JSON response. Missing or malformed output must not be treated as successful execution. ## Single-result commands @@ -32,6 +33,8 @@ Completion never certifies traffic coverage, sampling, comparability or profile The example message is illustrative. `error.code` is the machine-readable identifier; `error.message` is a human-readable string. An error object contains no result data. During a stream, previously emitted records remain on stdout but are not a complete result. +Unknown options, invalid/missing option values and unsupported command-option combinations identify the offending option in `error.message` where available. Argument values are not echoed; error schema and exit codes are unchanged. + | Exit | Error code when an error object is emitted | Meaning | | --- | --- | --- | | 0 | None | Requested command completed; scope limitations still apply | diff --git a/src/main/kotlin/dev/scouterctl/bootstrap/Bootstrap.kt b/src/main/kotlin/dev/scouterctl/bootstrap/Bootstrap.kt index 01df50c..c19acd3 100644 --- a/src/main/kotlin/dev/scouterctl/bootstrap/Bootstrap.kt +++ b/src/main/kotlin/dev/scouterctl/bootstrap/Bootstrap.kt @@ -19,9 +19,11 @@ fun runCli(args: Array) { OnlineCommands(contexts, { endpoint, password -> CollectorSource.connect(endpoint, password, BuildInfo.version) }, ::CaptureStore, ::EvidenceStore, ::CounterCaptureStore, TerminalPasswordInput(), output), ) - val command = CommandLine(Cli(dispatcher, output)) + val cli = Cli(dispatcher, output) + val command = CommandLine(cli) + cli.configureHelp(command) command.commandSpec.version("scouterctl ${BuildInfo.version}") - command.setParameterExceptionHandler { _, _ -> output.failure(OperationFailure(ErrorCode.INVALID_ARGUMENT, "Invalid command arguments; use --help")) } + command.setParameterExceptionHandler { error, _ -> output.failure(argumentFailure(error)) } exitProcess(command.execute(*args)) } diff --git a/src/main/kotlin/dev/scouterctl/cli/ArgumentFailure.kt b/src/main/kotlin/dev/scouterctl/cli/ArgumentFailure.kt new file mode 100644 index 0000000..b99d2f3 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/cli/ArgumentFailure.kt @@ -0,0 +1,21 @@ +package dev.scouterctl.cli + +import dev.scouterctl.core.ErrorCode +import dev.scouterctl.core.OperationFailure +import picocli.CommandLine.Model.OptionSpec +import picocli.CommandLine.MissingParameterException +import picocli.CommandLine.ParameterException +import picocli.CommandLine.UnmatchedArgumentException + +internal fun argumentFailure(error: ParameterException): OperationFailure { + val option = (error.argSpec as? OptionSpec)?.longestName() + ?: (error as? MissingParameterException)?.missing?.filterIsInstance()?.firstOrNull()?.longestName() + val message = if (option != null) { + "Invalid or missing value for $option; use --help" + } else if (error is UnmatchedArgumentException) { + val unknown = error.unmatched.firstOrNull { it.substringBefore('=').matches(Regex("--?[A-Za-z][A-Za-z0-9-]*")) } + ?.substringBefore('=') + if (unknown == null) "Invalid command arguments; use --help" else "Unknown option $unknown; use --help" + } else "Invalid command arguments; use --help" + return OperationFailure(ErrorCode.INVALID_ARGUMENT, message) +} diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index f3b6a8c..4c8d1e7 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -16,7 +16,7 @@ import java.util.concurrent.Callable "Password: SCOUTERCTL_PASSWORD, or --interactive for hidden terminal input. Passwords are never saved.", "TCP has no native TLS; use a trusted path."]) class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOutput) : Callable { - @Parameters(arity = "1..2", paramLabel = "COMMAND") lateinit var words: List + @Parameters(arity = "1..2", paramLabel = "COMMAND") var words: List = emptyList() @Mixin val connection = ConnectionOptions() @Spec lateinit var spec: CommandSpec @Option(names = ["--name"], description = ["Name for context add/use"]) var name: String? = null diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt new file mode 100644 index 0000000..044b1f2 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt @@ -0,0 +1,70 @@ +package dev.scouterctl.cli + +internal data class CommandGuide( + val name: String, + val description: String, + val required: Set, + val optional: Set, + val example: String, + val notes: List = emptyList(), +) { + val allowed: Set get() = required + optional +} + +private val connectionOptions = setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input") +private val windowOptions = setOf("--from", "--to", "--timezone", "--object-hash") +private val xlogOptions = windowOptions + "--date" +private val pagingOptions = setOf("--page-size", "--max-pages") +private val offlineOptions = setOf("--include-text", "--no-input") +private val connectionNotes = listOf("Connection options override a saved context; otherwise host/port default to 127.0.0.1:6100.", + "Provide an account with --user or a saved context. Use SCOUTERCTL_PASSWORD, or --interactive for hidden terminal input.") +private const val windowExample = "--date 20261101 --from 2026-11-01T00:00:00Z --to 2026-11-01T00:01:00Z --timezone UTC --object-hash 42" + +internal val commandGuides = listOf( + CommandGuide("doctor", "Check Collector connectivity and authentication.", emptySet(), connectionOptions, + "doctor --host localhost --user reader --no-input", connectionNotes), + CommandGuide("auth login", "Check credentials without saving a password.", emptySet(), connectionOptions, + "auth login --host localhost --user reader --interactive", connectionNotes), + CommandGuide("context add", "Save a named connection without its password.", setOf("--name", "--user"), + setOf("--host", "--port", "--no-input"), "context add --name local --host localhost --user reader"), + CommandGuide("context list", "List saved connections and the selected context.", emptySet(), setOf("--no-input"), "context list"), + CommandGuide("context use", "Select a saved connection.", setOf("--name"), setOf("--no-input"), "context use --name local"), + CommandGuide("collector status", "Read Collector status.", emptySet(), connectionOptions, + "collector status --user reader --no-input", connectionNotes), + CommandGuide("collector config", "Read safe Collector settings; optionally select an exact key.", emptySet(), connectionOptions + "--key", + "collector config --user reader --no-input", connectionNotes), + CommandGuide("agent status", "Read Agent JVM and runtime status.", setOf("--object-hash"), connectionOptions, + "agent status --object-hash 42 --user reader --no-input", connectionNotes), + CommandGuide("agent config", "Read safe Agent settings; optionally select an exact key.", setOf("--object-hash"), connectionOptions + "--key", + "agent config --object-hash 42 --user reader --no-input", connectionNotes), + CommandGuide("object list", "List live objects, or recorded objects for --date.", emptySet(), connectionOptions + "--date", + "object list --user reader --no-input", connectionNotes), + CommandGuide("counter list", "List available real-time counters for an object.", setOf("--object-hash"), connectionOptions, + "counter list --object-hash 42 --user reader --no-input", connectionNotes), + CommandGuide("counter get", "Read one exact real-time counter.", setOf("--object-hash", "--counter"), connectionOptions, + "counter get --object-hash 42 --counter Cpu --user reader --no-input", connectionNotes), + CommandGuide("counter history", "Stream stored counter values for a dated window.", xlogOptions + "--counter", connectionOptions, + "counter history $windowExample --counter Cpu --user reader --no-input", connectionNotes), + CommandGuide("counter export", "Export stored counters in resumable chunks.", windowOptions + setOf("--counter", "--directory"), + connectionOptions + setOf("--resume", "--chunk-minutes", "--max-chunks"), + "counter export --from 2026-11-01T00:00:00Z --to 2026-11-01T00:01:00Z --timezone UTC --object-hash 42 --counter Cpu --directory counters --user reader --no-input", + connectionNotes + "Use the same query and directory with --resume; chunk boundaries split at Collector midnight."), + CommandGuide("xlog search", "Stream XLogs for a storage date and time window.", xlogOptions, connectionOptions + pagingOptions + "--include-text", + "xlog search $windowExample --user reader --no-input", connectionNotes), + CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + "--include-text", + "profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes), + CommandGuide("capture", "Save a resumable XLog capture, optionally including profiles.", xlogOptions + "--directory", + connectionOptions + pagingOptions + setOf("--resume", "--include-profiles", "--include-text"), + "capture $windowExample --directory capture --user reader --no-input", + connectionNotes + listOf("--include-text requires --include-profiles. Text can contain sensitive values.", + "Use the same query and directory with --resume. Profile capture defaults to page size 25, maximum 100.")), + CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions, + "inspect --directory capture --txid 123"), + CommandGuide("inspect search", "Search captured requests offline.", setOf("--directory"), offlineOptions + + setOf("--limit", "--min-elapsed-ms", "--errors-only", "--service-hash", "--service-name"), + "inspect search --directory capture --errors-only", listOf("--service-name requires --include-text and stored dictionary text.")), + CommandGuide("inspect summary", "Summarize captured services offline.", setOf("--directory"), offlineOptions + setOf("--limit", "--sort"), + "inspect summary --directory capture --sort elapsed"), + CommandGuide("inspect compare", "Compare baseline and candidate captures offline.", setOf("--before", "--after"), offlineOptions + "--limit", + "inspect compare --before baseline --after candidate"), +).associateBy { it.name } diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandHelp.kt b/src/main/kotlin/dev/scouterctl/cli/CommandHelp.kt new file mode 100644 index 0000000..0ec4a5e --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/cli/CommandHelp.kt @@ -0,0 +1,36 @@ +package dev.scouterctl.cli + +import picocli.CommandLine +import picocli.CommandLine.Help +import picocli.CommandLine.Model.CommandSpec + +internal fun Cli.configureHelp(command: CommandLine) { + val ansi = if (System.console() == null || System.getenv("NO_COLOR") != null) Help.Ansi.OFF else Help.Ansi.AUTO + command.setColorScheme(Help.defaultColorScheme(ansi)) + command.setHelpFactory { root, colors -> Help(helpSpec(root, words.joinToString(" ")), colors) } +} + +private fun helpSpec(root: CommandSpec, name: String): CommandSpec { + val guide = commandGuides[name] + val children = commandGuides.values.filter { it.name.startsWith("$name ") } + val help = CommandSpec.create().name(if (guide == null) "scouterctl" else "scouterctl $name") + val usage = help.usageMessage().optionListHeading("Options:%n").sortOptions(false) + if (guide != null) { + val footer = listOf(" scouterctl ${guide.example}") + if (children.isEmpty()) emptyList() else + listOf("", "Subcommands:") + children.map { " ${it.name.padEnd(18)} ${it.description}" } + usage.description(guide.description, *guide.notes.toTypedArray()) + .footerHeading("%nExample:%n").footer(*footer.toTypedArray()) + root.options().filter { it.usageHelp() || it.versionHelp() || it.longestName() in guide.allowed }.forEach { option -> + help.addOption(option.toBuilder().required(option.longestName() in guide.required).build()) + } + } else { + val entries = if (name.isEmpty()) commandGuides.values else children + if (entries.isEmpty()) return root + usage.customSynopsis("scouterctl ${if (name.isEmpty()) "COMMAND" else "$name COMMAND"} [OPTIONS]") + .description(*(if (name.isEmpty()) root.usageMessage().description() else arrayOf("Scouter $name commands."))) + .footerHeading("%nCommands:%n").footer(*entries.map { " ${it.name.padEnd(18)} ${it.description}" }.toTypedArray(), + "", "Use scouterctl COMMAND --help for required options, accepted options and an example.") + root.options().filter { it.usageHelp() || it.versionHelp() }.forEach { help.addOption(it.toBuilder().build()) } + } + return help +} diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt index df30597..654e2c5 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt @@ -11,12 +11,11 @@ internal fun Cli.interpret(): CommandRequest { if (command in setOf("context add", "context list", "context use")) return contextRequest(command) if (name != null) invalid("--name requires context add/use") if (command == "inspect compare") { - allowedOptions(setOf("--before", "--after", "--limit", "--include-text", "--no-input")) + allowedOptions() return OfflineRequest.Compare(required(before, "--before"), required(after, "--after"), ComparisonOptions(limit ?: 20, includeText)) } if (command == "counter export") { - allowedOptions(setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input", - "--object-hash", "--counter", "--from", "--to", "--timezone", "--directory", "--resume", "--chunk-minutes", "--max-chunks")) + allowedOptions() val budget = maxChunks ?: 24 if (budget !in 1..10000) invalid("Use max-chunks 1..10000") val query = CounterExportQuery(Instant.parse(required(from, "--from")).toEpochMilli(), @@ -27,24 +26,20 @@ internal fun Cli.interpret(): CommandRequest { } if (chunkMinutes != null || maxChunks != null) invalid("--chunk-minutes and --max-chunks require counter export") if (command == "counter history") { - allowedOptions(setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input", - "--object-hash", "--counter", "--date", "--from", "--to", "--timezone")) + allowedOptions() return OnlineRequest.CounterHistory(CounterHistoryQuery(required(date, "--date"), Instant.parse(required(from, "--from")).toEpochMilli(), Instant.parse(required(to, "--to")).toEpochMilli(), required(timezone, "--timezone"), required(objHash, "--object-hash"), required(counterName, "--counter")).also { it.validate() }) } if (command in setOf("counter list", "counter get")) { - allowedOptions(setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input", "--object-hash") + - if (command == "counter get") setOf("--counter") else emptySet()) + allowedOptions() val counter = if (command == "counter get") required(counterName, "--counter") else null if (counter != null && (counter.isBlank() || counter.length > 256 || counter.any(Char::isISOControl))) invalid("Invalid counter name") return OnlineRequest.Counters(required(objHash, "--object-hash"), counter) } if (counterName != null) invalid("--counter requires counter get, counter history or counter export") if (command in setOf("collector status", "collector config", "agent status", "agent config")) { - allowedOptions(setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input") + - (if (command.endsWith("config")) setOf("--key") else emptySet()) + - if (command.startsWith("agent ")) setOf("--object-hash") else emptySet()) + allowedOptions() settingKey?.let { if (!it.matches(Regex("[A-Za-z_][A-Za-z0-9_.-]{0,255}"))) invalid("Use a setting name for --key") } return when (command) { "collector status" -> OnlineRequest.CollectorStatus @@ -56,7 +51,7 @@ internal fun Cli.interpret(): CommandRequest { if (settingKey != null) invalid("--key requires collector config or agent config") if (before != null || after != null) invalid("--before and --after require inspect compare") if (command == "inspect summary") { - allowedOptions(setOf("--directory", "--limit", "--sort", "--include-text", "--no-input")) + allowedOptions() return OfflineRequest.Summary(required(directory, "--directory"), ServiceSummaryOptions(limit ?: 20, ServiceSort.parse(summarySort ?: "errors"), includeText)) } @@ -77,12 +72,7 @@ internal fun Cli.interpret(): CommandRequest { } private fun Cli.contextRequest(command: String): ContextRequest { - val allowed = when (command) { - "context add" -> setOf("--name", "--host", "--port", "--user", "--no-input") - "context use" -> setOf("--name", "--no-input") - else -> setOf("--no-input") - } - allowedOptions(allowed) + allowedOptions() return when (command) { "context add" -> ContextRequest.Add(name, connection.host ?: "127.0.0.1", connection.port ?: 6100, connection.user) "context use" -> ContextRequest.Use(name) @@ -94,10 +84,13 @@ private fun Cli.validateGeneralOptions(command: String) { val supported = setOf("doctor", "auth login", "object list", "xlog search", "profile get", "capture", "inspect", "inspect search") if (command !in supported) invalid("Unknown command") if (command == "auth login") { - allowedOptions(setOf("--host", "--port", "--user", "--context", "--timeout-ms", "--interactive", "--no-input")) + allowedOptions() } if (command != "inspect search" && (limit != null || minElapsedMs != null || errorsOnly || serviceHash != null || serviceName != null)) { - invalid("Search filters require inspect search") + val option = spec.commandLine().parseResult.matchedOptions().first { + it.longestName() in setOf("--limit", "--min-elapsed-ms", "--errors-only", "--service-hash", "--service-name") + }.longestName() + invalid("$option requires inspect search") } if (includeProfiles && command != "capture") invalid("--include-profiles requires capture") if (includeText && command !in setOf("xlog search", "profile get", "inspect", "inspect search") && !(command == "capture" && includeProfiles)) { @@ -135,9 +128,10 @@ private fun Cli.query(): Query = Query(required(date, "--date"), Instant.parse(required(from, "--from")).toEpochMilli(), Instant.parse(required(to, "--to")).toEpochMilli(), required(timezone, "--timezone"), required(objHash, "--object-hash"), pageSize ?: if (includeProfiles) 25 else 1000).also { it.validate() } -private fun Cli.allowedOptions(allowed: Set) { - if (spec.commandLine().parseResult.matchedOptions().any { it.longestName() !in allowed }) { - invalid("Option is not supported by this command; use --help") +private fun Cli.allowedOptions() { + val allowed = commandGuides.getValue(words.joinToString(" ")).allowed + spec.commandLine().parseResult.matchedOptions().firstOrNull { it.longestName() !in allowed }?.let { + invalid("Option ${it.longestName()} is not supported by this command; use --help") } } private fun invalid(message: String): Nothing = throw OperationFailure(ErrorCode.INVALID_ARGUMENT, message) diff --git a/src/test/kotlin/dev/scouterctl/CliUsabilityTest.kt b/src/test/kotlin/dev/scouterctl/CliUsabilityTest.kt new file mode 100644 index 0000000..fa416ee --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CliUsabilityTest.kt @@ -0,0 +1,112 @@ +package dev.scouterctl + +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldContain +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.jsonObject +import kotlinx.serialization.json.jsonPrimitive +import java.nio.file.Files +import java.nio.file.Path +import java.util.concurrent.TimeUnit + +class CliUsabilityTest : FunSpec({ + for (noColor in listOf(false, true)) { + test("redirected help contains no ANSI even when color is forced and NO_COLOR is $noColor") { + val result = helpOutput(tempdir().toPath(), listOf("--help"), noColor) + result.exit shouldBe 0 + result.output shouldContain "scouterctl" + result.output shouldNotContain "\u001b" + result.stderr shouldBe "" + } + } + + for (noColor in listOf(null, "")) { + test("terminal help honors NO_COLOR presence even with an empty value: $noColor") { + TerminalCli(tempdir().toPath(), listOf("capture", "--help"), + command = cliJavaCommand(jvmOptions = listOf("-Dpicocli.ansi=true")), noColor = noColor).use { terminal -> + terminal.finish().exit shouldBe 0 + terminal.rawText().contains("\u001b[1m") shouldBe (noColor == null) + } + } + } + + data class HelpCase(val words: List, val required: List, val excluded: String) + for (case in listOf( + HelpCase(listOf("capture"), listOf("--directory", "--date", "--timezone", "--object-hash"), "--key"), + HelpCase(listOf("counter", "history"), listOf("--counter", "--date", "--object-hash"), "--directory"), + HelpCase(listOf("inspect", "compare"), listOf("--before", "--after"), "--host"), + )) { + test("${case.words.joinToString(" ")} help identifies its required options and excludes unrelated options") { + val result = helpOutput(tempdir().toPath(), case.words + "--help") + result.exit shouldBe 0 + result.stderr shouldBe "" + val synopsis = result.output.substringBefore("Options:") + synopsis shouldContain "scouterctl ${case.words.joinToString(" ")}" + case.required.forEach { option -> + synopsis shouldContain "$option=" + synopsis shouldNotContain "[$option=" + } + result.output shouldNotContain case.excluded + } + } + + test("inspect help also exposes its offline subcommands") { + val result = helpOutput(tempdir().toPath(), listOf("inspect", "--help")) + result.exit shouldBe 0 + for (subcommand in listOf("search", "summary", "compare")) result.output shouldContain "inspect $subcommand" + result.output shouldNotContain "--host" + } + + test("counter family help lists its commands without unrelated command options") { + val result = helpOutput(tempdir().toPath(), listOf("counter", "--help")) + result.exit shouldBe 0 + result.output shouldContain "scouterctl counter COMMAND" + for (subcommand in listOf("list", "get", "history", "export")) result.output shouldContain "counter $subcommand" + result.output shouldNotContain "--directory" + result.output shouldNotContain "agent status" + } + + test("unmatched positional values return a safe JSON error") { + val result = runOfflineCli(tempdir().toPath(), listOf("doctor", "unexpected", "secret-value")) + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["code"]!!.jsonPrimitive.content shouldBe "INVALID_ARGUMENT" + result.output shouldNotContain "secret-value" + result.stderr shouldBe "" + } + + for ((args, option) in listOf( + listOf("doctor", "--unknown=secret-value") to "--unknown", + listOf("doctor", "-z", "secret-value") to "-z", + listOf("doctor", "--port", "secret-value") to "--port", + listOf("doctor", "--port") to "--port", + listOf("counter", "list", "--directory", "secret-value") to "--directory", + )) { + test("invalid ${args.joinToString(" ")} identifies the option without echoing values") { + val result = runOfflineCli(tempdir().toPath(), args) + result.exit shouldBe 2 + result.stderr shouldBe "" + val error = result.events.single()["error"]!!.jsonObject + error["code"]!!.jsonPrimitive.content shouldBe "INVALID_ARGUMENT" + error["message"]!!.jsonPrimitive.content shouldContain option + result.output shouldNotContain "secret-value" + } + } +}) + +private data class HelpOutput(val exit: Int, val output: String, val stderr: String) + +private fun helpOutput(work: Path, args: List, noColor: Boolean = false): HelpOutput { + val stdout = work.resolve("stdout") + val stderr = work.resolve("stderr") + val process = ProcessBuilder(cliJavaCommand(jvmOptions = listOf("-Dpicocli.ansi=true")) + args) + .apply { + environment()["TERM"] = "xterm" + if (noColor) environment()["NO_COLOR"] = "1" else environment().remove("NO_COLOR") + }.redirectOutput(stdout.toFile()).redirectError(stderr.toFile()).start() + try { + check(process.waitFor(15, TimeUnit.SECONDS)) { "Help did not terminate" } + return HelpOutput(process.exitValue(), Files.readString(stdout), Files.readString(stderr)) + } finally { process.destroyForcibly() } +} diff --git a/src/test/kotlin/dev/scouterctl/TerminalCli.kt b/src/test/kotlin/dev/scouterctl/TerminalCli.kt index 1b8c161..5f10a25 100644 --- a/src/test/kotlin/dev/scouterctl/TerminalCli.kt +++ b/src/test/kotlin/dev/scouterctl/TerminalCli.kt @@ -10,11 +10,14 @@ import java.nio.file.Path import java.util.concurrent.Executors import java.util.concurrent.TimeUnit -internal class TerminalCli(work: Path, args: List, password: String? = null, command: List? = null) : AutoCloseable { +internal class TerminalCli(work: Path, args: List, password: String? = null, command: List? = null, + noColor: String? = null) : AutoCloseable { private val environment = System.getenv().toMutableMap().apply { remove("SCOUTERCTL_PASSWORD") put("SCOUTERCTL_CONFIG", work.resolve("config.json").toString()) put("TERM", "xterm") + remove("NO_COLOR") + if (noColor != null) put("NO_COLOR", noColor) if (password != null) put("SCOUTERCTL_PASSWORD", password) if (command != null) withCliCoverage() } @@ -36,7 +39,9 @@ internal class TerminalCli(work: Path, args: List, password: String? = n } } - fun text(): String = synchronized(bytes) { bytes.toString(Charsets.UTF_8) } + fun rawText(): String = synchronized(bytes) { bytes.toString(Charsets.UTF_8) } + + fun text(): String = rawText() .replace(Regex("\u001B\\[[0-?]*[ -/]*[@-~]"), "") .replace(Regex("\u001B\\][^\u0007]*(?:\u0007|\u001B\\\\)"), "") .replace("\r", "") From defecb25d3977a543d7276babe0e3cd2b7ef3fc0 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:36:08 +0900 Subject: [PATCH 13/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Avoid=20duplicate?= =?UTF-8?q?=20profile=20capture=20resume=20scans?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Collect committed transaction IDs during the existing validated traversal, preserving integrity and recovery while removing the extra prefix scan. Add reproducible local I/O measurements and document remaining scaling limits. Refs #74 --- build.gradle.kts | 12 +++ docs/capture-resume-performance.md | 37 ++++++++ docs/capture-scale-validation.md | 2 + .../scouterctl/application/EvidenceCapture.kt | 5 +- .../dev/scouterctl/EvidenceResumeTest.kt | 55 +++++++++++ .../dev/scouterctl/ResumeReadBenchmark.kt | 93 +++++++++++++++++++ .../dev/scouterctl/ResumeReadMeasurement.kt | 36 +++++++ 7 files changed, 237 insertions(+), 3 deletions(-) create mode 100644 docs/capture-resume-performance.md create mode 100644 src/test/kotlin/dev/scouterctl/EvidenceResumeTest.kt create mode 100644 src/test/kotlin/dev/scouterctl/ResumeReadBenchmark.kt create mode 100644 src/test/kotlin/dev/scouterctl/ResumeReadMeasurement.kt diff --git a/build.gradle.kts b/build.gradle.kts index 48d30d3..d9345e9 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -183,6 +183,18 @@ listOf("scaleFixtures" to "generate", "scaleCheck" to "run").forEach { (taskName } } +tasks.register("resumeReadCheck") { + group = "verification" + description = "Measure synthetic profile-capture resume reads and elapsed time (opt-in)." + dependsOn(tasks.testClasses) + val externalJar = providers.gradleProperty("cliJar") + classpath = if (externalJar.isPresent) files(file(externalJar.get())) + sourceSets.test.get().runtimeClasspath + else sourceSets.test.get().runtimeClasspath + mainClass.set("dev.scouterctl.ResumeReadBenchmarkKt") + maxHeapSize = "512m" + args(providers.gradleProperty("resumeReadResults").getOrElse("build/resume-read-results")) +} + // Opt-in fixture for matching legacy Collector/Agent releases on Java 8. val legacyFixture = sourceSets.create("legacyFixture") configurations.named(legacyFixture.implementationConfigurationName) { diff --git a/docs/capture-resume-performance.md b/docs/capture-resume-performance.md new file mode 100644 index 0000000..070749c --- /dev/null +++ b/docs/capture-resume-performance.md @@ -0,0 +1,37 @@ +# Profile capture resume work + +Profile capture (`capture --include-profiles`) validates every committed page before recovering profiles or appending a new page. It checks SHA-256, JSON/record structure, object/time scope, profile identity and transaction uniqueness. The resume path collects existing transaction IDs during this same traversal; it does not reread all pages solely to rebuild that set. A duplicate incoming transaction is still rejected before its profile is fetched or its page is committed. + +Pages with unavailable profiles require an additional read for recovery and can be rewritten with new checksums. Already execution-complete captures still validate their committed pages and retry unavailable profiles before returning. Unreferenced or partial files are not committed pages. Capture formats, error codes and CLI budgets are unchanged. + +## Reproduce the measurement + +With JDK 21, run the opt-in application-level measurement: + +```sh +./gradlew --no-daemon resumeReadCheck +``` + +Use `gradlew.bat` on Windows. Results require a new directory; `-PresumeReadResults=PATH` selects one. Preserve an earlier result directory before choosing a new path for another trial. This task is separate from the default tests and from offline `inspect compare` scale validation. + +To compare an earlier implementation, preserve its fat JAR and run the current measurement harness with that JAR first on the runtime classpath: + +```sh +./gradlew --no-daemon resumeReadCheck -PcliJar=PATH/TO/BASELINE-all.jar -PresumeReadResults=build/resume-read-before +``` + +Use separate fresh JVM runs for each trial and compare the recorded implementation class hashes. Supplying a baseline JAR changes the measured application/store implementation; it does not replace the current synthetic fixture or measurement code. + +The fixture uses the real file-backed `EvidenceStore` and a synthetic observation source, without a Collector, credentials or simulated network delay. It captures 120 pages in three successive 40-page application invocations, then resumes the execution-complete capture once. Both 2 and 100 records per page are measured, with 20 small structural SQL steps per profile. Previously committed profiles must not be fetched again. This is local application/filesystem work, not production Collector throughput or packaged CLI startup time. + +`report.json` records OS, architecture, JDK, CPU count, heap limit, implementation class hashes and each invocation's existing page/record count, page bytes, read bytes, full-file scan-count histogram, elapsed capture time, preparation time to the first new XLog page request, added pages and profile requests. A completed capture makes no new page request, so its preparation time is null. `reads-*.jfr` retains the raw [JDK Flight Recorder](https://docs.oracle.com/en/java/javase/21/docs/api/jdk.jfr/jdk/jfr/Recording.html) events. + +JFR `jdk.FileRead` events use a zero duration threshold and are filtered to each fixture's page paths. The measurement sums positive read bytes per page and divides by that page's byte size; it does not count low-level read calls as whole scans. Capture timing excludes recorder start/dump/stop and report processing, but includes manifest I/O, page validation, synthetic profile creation, page writes and checkpointing. Recorder overhead within the measured operation remains. Preparation ends at the first synthetic page request and includes local validation/recovery. OS cache, JIT, GC and host contention are not controlled; timings are observations, not test thresholds or a universal speedup claim. No maximum-size memory or production crossover claim follows from these fixtures. + +## Remaining bounds and tradeoffs + +For an ordinary incomplete capture with no unavailable profiles, the existing-page scan counts are 0, 40 and 80 over the three capture invocations, rather than 0, 80 and 160 before the change. The subsequent completed resume scans all 120 pages once in both versions. Extra reads for unavailable-profile recovery are outside this ordinary-case count. + +Combining the passes halves these prefix scans; it does not eliminate their cumulative growth. With B new pages per invocation and P = mB total pages, ordinary repeated resumes scan Bm(m-1)/2 = (P²/B - P)/2 existing pages. Each individual resume remains linear in the committed prefix. Transaction-ID membership still requires memory proportional to committed records. Collecting through the existing validated callback retains the application's ID set while the store's uniqueness set is live, increasing simultaneous set storage; peak heap was not measured here. + +The CLI still allows `--max-pages` 1..40 and profile page sizes up to 100, with the existing capture limit of 10,000 pages. A bounded single-process continuation could retain IDs between batches and avoid repeated prefix validation, but it would need an explicit total-work budget, cancellation/checkpoint behavior and rules for on-disk changes between batches. Changing the current per-invocation budget contract is outside this fix. Persisting an unchecked ID cache or skipping validation on later invocations would weaken integrity guarantees and is not a substitute. diff --git a/docs/capture-scale-validation.md b/docs/capture-scale-validation.md index ef4f598..7bff012 100644 --- a/docs/capture-scale-validation.md +++ b/docs/capture-scale-validation.md @@ -1,5 +1,7 @@ # Capture scale validation +For synthetic profile-capture resume reads and preparation timing, see [profile capture resume work](capture-resume-performance.md). The workflow below measures offline comparison, a separate workload. + Run the opt-in **Capture scale validation** GitHub Actions workflow on a branch to measure offline `inspect compare`. It builds one CLI JAR and one synthetic capture archive, then verifies and uses the same bytes on Linux, Windows and macOS. It does not publish a release or run on every pull request. The macOS measurement job is outside the supported release-platform scope. You can also enable `scale_validation` when manually running **CI and release**, including on a feature branch. For a local Linux, macOS or Windows checkout with JDK 21: diff --git a/src/main/kotlin/dev/scouterctl/application/EvidenceCapture.kt b/src/main/kotlin/dev/scouterctl/application/EvidenceCapture.kt index ed44667..52b1386 100644 --- a/src/main/kotlin/dev/scouterctl/application/EvidenceCapture.kt +++ b/src/main/kotlin/dev/scouterctl/application/EvidenceCapture.kt @@ -6,6 +6,7 @@ fun captureEvidence(source: ObservationSource, query: Query, request: CaptureOpt query.validate() if (query.pageSize > 100) throw OperationFailure(ErrorCode.INVALID_ARGUMENT, "Profile capture page-size must be 1..100") val reader = EvidenceReader(source, query.date, request.includeText) + val ids = mutableSetOf() var manifest = if (request.resume) store.load().also { if (it.schemaVersion != "scouterctl.capture/v3") { throw OperationFailure(ErrorCode.INVALID_ARGUMENT, "Resume requires a v3 profile capture; preserve this capture and start a new directory") @@ -13,7 +14,7 @@ fun captureEvidence(source: ObservationSource, query: Query, request: CaptureOpt if (it.source != source.info || it.query != query || it.endpoint != request.endpoint || it.textIncluded != request.includeText) { throw OperationFailure(ErrorCode.INVALID_ARGUMENT, "Resume source, query or text policy differs from manifest") } - store.validate(it) + store.forEachRecord(it) { record -> ids.add(record.xlog.txid) } } else { store.create() EvidenceManifest(source = source.info, endpoint = request.endpoint, query = query, textIncluded = request.includeText) @@ -35,8 +36,6 @@ fun captureEvidence(source: ObservationSource, query: Query, request: CaptureOpt store.save(manifest) return manifest } - val ids = mutableSetOf() - manifest.pages.indices.forEach { index -> store.readPage(manifest, index).forEach { ids.add(it.xlog.txid) } } repeat(request.maxPages) { if (manifest.pages.size >= 10000) throw OperationFailure(ErrorCode.PARTIAL_RESULT, "Evidence capture limit of 10000 pages reached") val page = source.page(query, manifest.cursor) diff --git a/src/test/kotlin/dev/scouterctl/EvidenceResumeTest.kt b/src/test/kotlin/dev/scouterctl/EvidenceResumeTest.kt new file mode 100644 index 0000000..52be721 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/EvidenceResumeTest.kt @@ -0,0 +1,55 @@ +package dev.scouterctl + +import dev.scouterctl.adapters.files.EvidenceStore +import dev.scouterctl.application.CaptureOptions +import dev.scouterctl.application.captureEvidence +import dev.scouterctl.core.* +import io.kotest.core.spec.style.FunSpec +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import java.nio.file.Files + +class EvidenceResumeTest : FunSpec({ + test("issue 74 resume reads each committed page once while retaining IDs across page budgets") { + val directory = tempdir().toPath().resolve("capture") + val store = EvidenceStore(directory) + val profileIds = mutableListOf() + val source = MemorySource({ cursor -> + val first = cursor.txid.toLong() + 1 + val last = first + 1 + XLogPage(listOf(portRow(first), portRow(last)), Cursor(portQuery.from + last, last.toString()), last < 12) + }, { id -> profileIds += id; ProfileRecord(id.toString(), emptyList()) }) + var manifest = captureEvidence(source, portQuery, CaptureOptions(false, 2, "memory"), store) + + repeat(2) { + val expectedBytes = manifest.pages.associate { directory.resolve(it.name) to Files.size(directory.resolve(it.name)) } + val measured = measureResumeReads(directory) { + captureEvidence(source, portQuery, CaptureOptions(true, 2, "memory"), store) + } + measured.bytes shouldBe expectedBytes + manifest = measured.result + } + + manifest.complete shouldBe true + profileIds shouldBe (1L..12L).toList() + manifest.pages.flatMapIndexed { index, _ -> store.readPage(manifest, index) }.map { it.xlog.txid } shouldBe + (1L..12L).map(Long::toString) + } + + test("resume rejects an incoming transaction already committed before fetching or appending its profile") { + val store = EvidenceStore(tempdir().toPath().resolve("capture")) + var profiles = 0 + val source = MemorySource({ XLogPage(listOf(portRow(1)), Cursor(portQuery.from + 1, "1"), true) }, + { id -> profiles++; ProfileRecord(id.toString(), emptyList()) }) + val checkpoint = captureEvidence(source, portQuery, CaptureOptions(false, 1, "memory"), store) + + shouldThrow { + captureEvidence(source, portQuery, CaptureOptions(true, 1, "memory"), store) + }.code shouldBe ErrorCode.PROTOCOL_ERROR + + profiles shouldBe 1 + store.load().pages shouldBe checkpoint.pages + store.load().cursor shouldBe checkpoint.cursor + } +}) diff --git a/src/test/kotlin/dev/scouterctl/ResumeReadBenchmark.kt b/src/test/kotlin/dev/scouterctl/ResumeReadBenchmark.kt new file mode 100644 index 0000000..ebcf080 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/ResumeReadBenchmark.kt @@ -0,0 +1,93 @@ +package dev.scouterctl + +import dev.scouterctl.adapters.files.EvidenceStore +import dev.scouterctl.adapters.serialization.json +import dev.scouterctl.application.CaptureOptions +import dev.scouterctl.application.captureEvidence +import dev.scouterctl.core.* +import kotlinx.serialization.Serializable +import kotlinx.serialization.encodeToString +import java.nio.file.Files +import java.nio.file.Path +import java.security.MessageDigest + +@Serializable +private data class ResumeReadSample( + val pageSize: Int, val invocation: Int, val existingPages: Int, val existingRecords: Int, + val existingPageBytes: Long, val existingReadBytes: Long, val scanCounts: Map, + val preparationNanos: Long?, val captureNanos: Long, val addedPages: Int, val profileRequests: Int, + val executionComplete: Boolean, +) + +@Serializable +private data class ResumeReadReport( + val os: String, val architecture: String, val javaVersion: String, val processors: Int, + val maximumHeapBytes: Long, val classSha256: Map, val samples: List, +) + +fun main(args: Array) { + require(args.size == 1) { "Use a new result directory" } + val root = Files.createDirectory(Path.of(args.single()).toAbsolutePath()) + val samples = mutableListOf() + for (pageSize in listOf(2, 100)) { + val directory = root.resolve("records-per-page-$pageSize") + val fixture = ResumeBenchmarkFixture(pageSize) + val store = EvidenceStore(directory) + var previous: EvidenceManifest? = null + repeat(4) { index -> + val pages = previous?.pages.orEmpty() + val sizes = pages.associate { directory.resolve(it.name) to Files.size(directory.resolve(it.name)) } + val profilesBefore = fixture.profileRequests + fixture.firstPageNanos = null + var started = 0L + val measurement = measureResumeReads(directory) { + started = System.nanoTime() + captureEvidence(fixture.source, fixture.query, CaptureOptions(index > 0, 40, "memory"), store) + } + val result = measurement.result + val reads = sizes.map { (file, size) -> + val bytes = measurement.bytes[file] ?: 0L + check(bytes > 0 && bytes % size == 0L) { "Incomplete file-read measurement for $file" } + (bytes / size).toString() + }.groupingBy { it }.eachCount() + val added = result.pages.size - pages.size + val profiles = fixture.profileRequests - profilesBefore + check(profiles == added * pageSize) { "Committed profiles were fetched again" } + check(result.pages.size == minOf((index + 1) * 40, 120)) + check(result.pages.sumOf { it.records } == result.pages.size * pageSize) + check(result.executionComplete == (index >= 2)) + samples += ResumeReadSample(pageSize, index + 1, pages.size, pages.sumOf { it.records }, + sizes.values.sum(), sizes.keys.sumOf { measurement.bytes[it] ?: 0 }, reads, + fixture.firstPageNanos?.minus(started), measurement.elapsedNanos, added, profiles, result.executionComplete) + Files.move(directory.resolve("reads.jfr"), root.resolve("reads-$pageSize-${index + 1}.jfr")) + previous = result + } + check(fixture.profileRequests == 120 * pageSize) + store.validate(checkNotNull(previous)) + } + val classes = listOf("application/EvidenceCaptureKt", "adapters/files/EvidenceStore").associateWith { name -> + val bytes = checkNotNull(EvidenceStore::class.java.getResourceAsStream("/dev/scouterctl/$name.class")) + .use { it.readAllBytes() } + MessageDigest.getInstance("SHA-256").digest(bytes).joinToString("") { "%02x".format(it) } + } + val report = ResumeReadReport(System.getProperty("os.name") + " " + System.getProperty("os.version"), + System.getProperty("os.arch"), System.getProperty("java.runtime.version"), Runtime.getRuntime().availableProcessors(), + Runtime.getRuntime().maxMemory(), classes, samples) + Files.writeString(root.resolve("report.json"), json.encodeToString(report)) + println("Recorded ${samples.size} capture invocations in ${root.resolve("report.json")}") +} + +private class ResumeBenchmarkFixture(pageSize: Int) { + val query = portQuery.copy(to = portQuery.from + 3_600_000, pageSize = pageSize) + var profileRequests = 0 + var firstPageNanos: Long? = null + val source = MemorySource({ cursor -> + if (firstPageNanos == null) firstPageNanos = System.nanoTime() + val first = cursor.txid.toLong() + 1 + val last = first + pageSize - 1 + XLogPage((first..last).map { portRow(it) }, Cursor(query.from + last, last.toString()), last < 120L * pageSize) + }, { id -> + profileRequests++ + ProfileRecord(id.toString(), List(20) { order -> ProfileStep("sql", order, SqlProfile(123, 1, 0)) }) + }) +} diff --git a/src/test/kotlin/dev/scouterctl/ResumeReadMeasurement.kt b/src/test/kotlin/dev/scouterctl/ResumeReadMeasurement.kt new file mode 100644 index 0000000..974af4b --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/ResumeReadMeasurement.kt @@ -0,0 +1,36 @@ +package dev.scouterctl + +import jdk.jfr.Recording +import jdk.jfr.consumer.RecordingFile +import java.nio.file.Path +import java.time.Duration + +internal data class ResumeReadMeasurement(val result: T, val bytes: Map, val elapsedNanos: Long) + +internal fun measureResumeReads(directory: Path, operation: () -> T): ResumeReadMeasurement { + val recordingFile = directory.resolve("reads.jfr") + val measuredDirectory = directory.toAbsolutePath().normalize() + val elapsed: Long + val result: T + Recording().use { recording -> + recording.setToDisk(false) + recording.enable("jdk.FileRead").withThreshold(Duration.ZERO).withoutStackTrace() + recording.start() + val started = System.nanoTime() + result = operation() + elapsed = System.nanoTime() - started + recording.dump(recordingFile) + recording.stop() + } + val bytes = mutableMapOf() + RecordingFile(recordingFile).use { events -> + while (events.hasMoreEvents()) { + val event = events.readEvent() + if (event.eventType.name != "jdk.FileRead" || event.getLong("bytesRead") <= 0) continue + val path = Path.of(event.getString("path")).toAbsolutePath().normalize() + if (path.parent != measuredDirectory || !path.fileName.toString().startsWith("page-")) continue + bytes.merge(path, event.getLong("bytesRead"), Long::plus) + } + } + return ResumeReadMeasurement(result, bytes, elapsed) +} From b1dd2fae9a0933e51984dffee9f6ddd5daaffce0 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Thu, 12 Nov 2026 15:05:35 +0900 Subject: [PATCH 14/17] =?UTF-8?q?feat:=20=E2=9C=A8=20Group=20profile=20SQL?= =?UTF-8?q?=20steps=20by=20hash?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why Profiles with repeated SQL can produce hundreds of kilobytes of JSON. Add the single aggregate option retained by the issue's triage. Refs #62 ## Changes `profile get --group-sql` and offline `inspect --group-sql` return SQL hash groups with execution count, total/max elapsed milliseconds and error count. Statements still require `--include-text`; unresolved or conflicting dictionary text is explicit. The summary includes decoded-step counts, lossless transaction IDs and original completeness metadata. English and Korean guides explain the output and limits. In a synthetic wire fixture with 3,000 repeated SQL steps, live output with text fell from 667,803 bytes to 501 bytes. Default individual-step output remains available. ## Validation - Full Linux `coverageCheck verifyReleaseInputs`: passed; 266 executable test cases, line coverage 95.38%, branch coverage 80.95% (90%/80% minimums). - Native Windows JDK 21: 10 new packaged-CLI scenarios and 6 aggregate tests passed. - Actual CLI comparisons against the previous build: byte-identical default live/inspect output and capture files; previous strict reader accepts new captures. Verified text redaction, unchanged capture bytes after inspection, late-page corruption rejection, truncation, unavailable profiles and command-option validation. - Baseline and deliberate aggregation mutations demonstrated that the new tests detect missing functionality, overflow and incorrect redacted statement output. Validation used synthetic local TCP fixtures; no external Collector/Agent deployment was tested. ## Compatibility and risks The opt-in payload declares `scouterctl.profile-sql-summary/v1`; existing result envelopes, default outputs, capture formats and exit/completeness semantics are preserved. Grouped output omits individual chronology and non-SQL details. Size grows with distinct SQL hashes; this does not reduce upstream transfer/decoding or prove trace completeness. Duration totals count decoded steps and are not transaction wall time. --- docs/README.md | 1 + docs/cli-contract.md | 2 +- docs/ko/README.md | 1 + docs/ko/profile-sql-summary.md | 32 +++++ docs/profile-completeness.md | 2 + docs/profile-sql-summary.md | 44 +++++++ .../serialization/ProfileSqlSummaryJson.kt | 31 +++++ .../scouterctl/application/SummarizeSql.kt | 15 +++ src/main/kotlin/dev/scouterctl/cli/Cli.kt | 2 + .../kotlin/dev/scouterctl/cli/CommandGuide.kt | 8 +- .../dev/scouterctl/cli/CommandInterpreter.kt | 7 +- .../dev/scouterctl/cli/CommandRequest.kt | 4 +- .../dev/scouterctl/cli/OfflineCommands.kt | 4 +- .../dev/scouterctl/cli/OnlineCommands.kt | 3 +- .../dev/scouterctl/core/ProfileSqlSummary.kt | 11 ++ .../dev/scouterctl/ProfileSqlGroupingTest.kt | 109 ++++++++++++++++++ .../kotlin/dev/scouterctl/SqlSummaryTest.kt | 76 ++++++++++++ src/test/kotlin/dev/scouterctl/WireFixture.kt | 10 +- 18 files changed, 349 insertions(+), 13 deletions(-) create mode 100644 docs/ko/profile-sql-summary.md create mode 100644 docs/profile-sql-summary.md create mode 100644 src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt create mode 100644 src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt create mode 100644 src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt create mode 100644 src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt create mode 100644 src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt diff --git a/docs/README.md b/docs/README.md index f2e4997..56b9583 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,6 +23,7 @@ See [command and output examples](examples.md) for actual response shapes. - [Capture formats and older files](capture-compatibility.md) - [Interrupted capture recovery](capture-recovery.md) - [Profile completeness](profile-completeness.md) +- [Grouped profile SQL output](profile-sql-summary.md) - [Security](../SECURITY.md) ## Development and maintenance diff --git a/docs/cli-contract.md b/docs/cli-contract.md index 6f685e2..41eda1c 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -84,4 +84,4 @@ A consumer should accept the stream schema, process records, then require both t - Removing/renaming existing fields, changing their type, units or meaning, or changing established event semantics requires a new affected output schema version and documented migration. Existing error codes and exit meanings must not be silently repurposed. Enum/status additions that require new interpretation also need an explicit compatibility decision rather than being assumed safe. - New commands and additive fields can be introduced without changing existing envelopes. Changing existing command syntax or defaults in a way that breaks scripts requires release notes and a migration path. This document does not promise indefinite support for every old release. -Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes. +Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, [grouped SQL output](profile-sql-summary.md) for the optional profile summary schema, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes. diff --git a/docs/ko/README.md b/docs/ko/README.md index c7e28eb..3479bf2 100644 --- a/docs/ko/README.md +++ b/docs/ko/README.md @@ -12,6 +12,7 @@ - [설치·업데이트·이전 버전으로 전환·제거](installation.md) - [빠른 시작](../../QUICKSTART.ko.md) - [연결과 비밀번호](connections.md) +- [프로파일 SQL 집계](profile-sql-summary.md) - [기여 안내](../../CONTRIBUTING.ko.md) · [보안 안내](../../SECURITY.ko.md) ## 상세 사용법 diff --git a/docs/ko/profile-sql-summary.md b/docs/ko/profile-sql-summary.md new file mode 100644 index 0000000..abbdf94 --- /dev/null +++ b/docs/ko/profile-sql-summary.md @@ -0,0 +1,32 @@ +# 프로파일 SQL 집계 + +[English](../profile-sql-summary.md) | 한국어 + +SQL이 반복되어 프로파일 출력이 커지면 `--group-sql`로 SQL 해시별 집계를 조회할 수 있습니다. + +```sh +scouterctl profile get --date YYYYMMDD --txid TXID --group-sql +scouterctl inspect --directory CAPTURE --txid TXID --group-sql +``` + +이 옵션은 `profile get`과 `inspect`에서만 사용할 수 있습니다. SQL 문장을 보려면 `--include-text`도 지정하세요. 오프라인 조회에서는 텍스트를 포함해 저장한 캡처가 필요합니다. 옵션을 생략하면 개별 스텝을 반환합니다. 집계 조회는 캡처 파일이나 저장 형식을 바꾸지 않습니다. + +온라인 결과의 `data`, 오프라인 결과의 `data.record.profile`이 `scouterctl.profile-sql-summary/v1` 집계 객체가 됩니다. 바깥쪽 `scouterctl.result/v1` 응답과 오프라인 캡처·XLog·누락 정보는 유지합니다. 사용할 수 없는 오프라인 프로파일은 `null`입니다. + +| 필드 | 의미 | +| --- | --- | +| `txid` | 부호를 포함한 10진 문자열 트랜잭션 ID | +| `totalSteps`, `sqlSteps`, `nonSqlSteps` | 디코딩된 전체·SQL·나머지 스텝 수 | +| `groups[].hash`, `count` | SQL 해시와 해당 SQL 스텝 수 | +| `elapsedTotalMs`, `maxElapsedMs` | 그룹의 경과 시간 합계와 최댓값, 밀리초 | +| `errors` | 오류 해시가 0이 아닌 SQL 스텝 수 | +| `textIncluded`, `completeness` | 요청에 따른 텍스트 포함 여부와 원래 프로파일의 잘림 상태 | +| `basis` | `decoded-profile-steps`: 디코딩된 스텝을 기준으로 집계 | + +서로 다른 SQL 스텝 종류도 해시가 같으면 한 그룹에 포함합니다. 경과 시간 합계 내림차순, 같은 합계에서는 부호 있는 해시 오름차순으로 정렬합니다. 합계는 64비트 정수이며 원래 음수 값도 유지합니다. 실행 수는 영향받은 행 수가 아니며, 스텝 시간이 겹칠 수 있으므로 합계가 트랜잭션 전체 시간과 같다고 해석하면 안 됩니다. + +`--include-text`를 지정하면 `statement`에 사전의 `hash`, `text`, `status`를 반환합니다. 텍스트가 없거나 다른 해시의 사전 항목만 있으면 미해결 상태입니다. 같은 해시에 서로 다른 문장이 확인되면 `statementConflict: true`와 미해결 텍스트를 반환합니다. 텍스트 옵션을 생략하면 `statement: null`이며 충돌 정보도 공개하지 않습니다. 집계는 오류 텍스트·바인드 값·스텝 순서·SQL 외 스텝 상세를 포함하지 않습니다. 상세 분석에는 기본 스텝 출력을 사용하세요. + +출력 크기는 서로 다른 SQL 해시 수에 비례합니다. 반복 SQL에 효과가 크지만 Collector 전송량이나 디코딩 작업을 줄이지 않으며, 스텝·그룹 수에 새 제한을 두지 않습니다. 누락된 스텝을 복원하지도 않습니다. + +[기존 프로파일 완전성·종료 코드](../profile-completeness.md)를 유지합니다. 명시적인 잘림은 가능한 집계와 함께 `completeness: "truncated"`, `executionComplete: false`, 종료 코드 8을 반환합니다. `unknown`은 완전함을 보장하지 않습니다. 불완전한 캡처와 사용할 수 없는 프로파일도 기존의 종료 코드 8을 유지합니다. 오프라인 집계 역시 선택한 트랜잭션 뒤의 페이지까지 캡처 전체 무결성을 검사하며, 손상되면 결과 대신 오류를 반환합니다. [영문 가이드](../profile-sql-summary.md)에서 전체 JSON 예시를 볼 수 있습니다. diff --git a/docs/profile-completeness.md b/docs/profile-completeness.md index 7d1aea6..d4f2ba4 100644 --- a/docs/profile-completeness.md +++ b/docs/profile-completeness.md @@ -16,6 +16,8 @@ scouterctl profile get --date YYYYMMDD --txid TXID scouterctl inspect --directory CAPTURE --txid TXID ``` +Add [`--group-sql`](profile-sql-summary.md) to aggregate decoded SQL steps by hash. The summary preserves these completeness and exit semantics. + ## Capture compatibility and limits The completeness value is stored with profiles in newly written v3 captures and survives text redaction during inspection. Existing captures without the field read as `unknown`; their former control-step text/code was not stored, so it cannot reliably be reconstructed. Reading an old capture does not rewrite it. diff --git a/docs/profile-sql-summary.md b/docs/profile-sql-summary.md new file mode 100644 index 0000000..10bced9 --- /dev/null +++ b/docs/profile-sql-summary.md @@ -0,0 +1,44 @@ +# Group profile SQL steps + +English | [한국어](ko/profile-sql-summary.md) + +Use `--group-sql` when repeated SQL makes a profile too large to read: + +```sh +scouterctl profile get --date YYYYMMDD --txid TXID --group-sql +scouterctl inspect --directory CAPTURE --txid TXID --group-sql +``` + +The option is accepted only by `profile get` and `inspect`. Add `--include-text` to include resolved SQL statements; offline inspection also requires a capture that stored text. Without `--group-sql`, both commands return their usual individual steps. Capture files and their schemas are unchanged, and inspection never rewrites them. + +## Output + +Online `data`, or offline `data.record.profile`, contains a `scouterctl.profile-sql-summary/v1` object instead of individual profile steps. The outer result envelope remains `scouterctl.result/v1`. Offline capture, XLog and availability metadata keep their existing fields. An unavailable offline profile remains `null`. + +Example summary with text omitted: + +```json +{ + "txid": "9223372036854775804", + "completeness": "unknown", + "textIncluded": false, + "totalSteps": 3004, + "sqlSteps": 3000, + "nonSqlSteps": 4, + "groups": [ + {"hash":123,"statement":null,"statementConflict":false,"count":3000,"elapsedTotalMs":6000,"maxElapsedMs":2,"errors":30} + ], + "schemaVersion": "scouterctl.profile-sql-summary/v1", + "basis": "decoded-profile-steps" +} +``` + +Each group represents one signed SQL hash across all decoded SQL step variants. Groups are sorted by descending `elapsedTotalMs`, then ascending signed hash. `count` counts SQL steps, `elapsedTotalMs` sums their elapsed milliseconds using a 64-bit integer, `maxElapsedMs` is their largest elapsed value, and `errors` counts steps with a nonzero error hash. Original signed duration values are retained. These are decoded-step statistics, not affected-row counts or transaction wall time; overlapping steps can make their sum exceed transaction elapsed time. `txid` remains a decimal string. + +With `--include-text`, `statement` has the usual dictionary `hash`, `text` and `status`. Missing text is unresolved. If multiple distinct resolved statements share a hash, the group has `statementConflict: true` and unresolved text rather than choosing one. Dictionary entries for a different hash are ignored. Without `--include-text`, statements are `null` and text conflicts are not reported. Error text, bind values, individual step order and non-SQL step details are not part of the summary; request the default profile output when you need step details. + +## Limits and completion + +Output grows with the number of distinct SQL hashes. Repeated statements benefit most; this mode imposes no step or group cap and does not reduce Collector transfer or profile decoding. It cannot restore missing steps or establish that a trace is complete. + +The original [profile completeness](profile-completeness.md) and exit behavior apply: known truncation returns available groups with `completeness: "truncated"`, `executionComplete: false` and exit 8. `unknown` does not certify a complete trace. Offline inspection still validates the entire capture before returning a summary, including pages after the selected transaction. Incomplete captures and unavailable profiles retain their existing exit 8 behavior; corrupt capture data remains an error. diff --git a/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt b/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt new file mode 100644 index 0000000..1bc9288 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt @@ -0,0 +1,31 @@ +package dev.scouterctl.adapters.serialization + +import dev.scouterctl.core.* +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.* + +@Serializable +data class SqlAggregateDto( + val hash: Int, val statement: DictionaryTextDto?, val statementConflict: Boolean, + val count: Int, val elapsedTotalMs: Long, val maxElapsedMs: Int, val errors: Int, +) + +@Serializable +data class ProfileSqlSummaryDto( + val txid: String, val completeness: ProfileCompletenessDto, val textIncluded: Boolean, + val totalSteps: Int, val sqlSteps: Int, val nonSqlSteps: Int, val groups: List, + val schemaVersion: String = "scouterctl.profile-sql-summary/v1", val basis: String = "decoded-profile-steps", +) + +fun ProfileSqlSummary.toDto(): ProfileSqlSummaryDto = ProfileSqlSummaryDto(txid, + when (completeness) { ProfileCompleteness.UNKNOWN -> ProfileCompletenessDto.UNKNOWN; ProfileCompleteness.TRUNCATED -> ProfileCompletenessDto.TRUNCATED }, + textIncluded, totalSteps, sqlSteps, nonSqlSteps, groups.map { + SqlAggregateDto(it.hash, it.statement?.toDto(), it.statementConflict, it.count, it.elapsedTotalMs, it.maxElapsedMs, it.errors) + }) + +fun Inspection.toSqlSummaryJson(summary: ProfileSqlSummary?): JsonObject { + val base = json.encodeToJsonElement(copy(record = record.copy(profile = null)).toDto()).jsonObject + val record = base.getValue("record").jsonObject + val grouped = summary?.let { json.encodeToJsonElement(it.toDto()) } ?: JsonNull + return JsonObject(base + ("record" to JsonObject(record + ("profile" to grouped)))) +} diff --git a/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt b/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt new file mode 100644 index 0000000..47cd784 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt @@ -0,0 +1,15 @@ +package dev.scouterctl.application + +import dev.scouterctl.core.* + +fun summarizeSql(profile: ProfileRecord, includeText: Boolean): ProfileSqlSummary { + val sql = profile.steps.mapNotNull { it.sql } + val groups = sql.groupBy { it.hash }.map { (hash, steps) -> + val texts = if (includeText) steps.mapNotNull { it.statement?.takeIf { text -> text.hash == hash }?.text }.distinct() + else emptyList() + SqlAggregate(hash, if (includeText) DictionaryText(hash, texts.singleOrNull()) else null, texts.size > 1, + steps.size, steps.sumOf { it.elapsedMs.toLong() }, steps.maxOf { it.elapsedMs }, steps.count { it.errorHash != 0 }) + }.sortedWith(compareByDescending { it.elapsedTotalMs }.thenBy { it.hash }) + return ProfileSqlSummary(profile.txid, profile.completeness, includeText, profile.steps.size, sql.size, + profile.steps.size - sql.size, groups) +} diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index 4c8d1e7..4fde5a4 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -42,6 +42,8 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut ]) var includeProfiles: Boolean = false @Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in xlog search, profile get, profile capture or inspect"]) var includeText: Boolean = false + @Option(names = ["--group-sql"], description = ["profile get / inspect: aggregate decoded SQL steps by hash instead of returning steps"]) + var groupSql: Boolean = false @Option(names = ["--limit"], description = ["inspect search/summary/compare: 1..1000 requests/services; default 20"]) var limit: Int? = null @Option(names = ["--sort"], description = ["inspect summary: errors, elapsed (maximum), requests; default errors"]) var summarySort: String? = null @Option(names = ["--min-elapsed-ms"], description = ["inspect search: inclusive minimum request duration"]) var minElapsedMs: Int? = null diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt index 044b1f2..a756791 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt @@ -51,15 +51,15 @@ internal val commandGuides = listOf( connectionNotes + "Use the same query and directory with --resume; chunk boundaries split at Collector midnight."), CommandGuide("xlog search", "Stream XLogs for a storage date and time window.", xlogOptions, connectionOptions + pagingOptions + "--include-text", "xlog search $windowExample --user reader --no-input", connectionNotes), - CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + "--include-text", - "profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes), + CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + setOf("--include-text", "--group-sql"), + "profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes + "Use --group-sql for SQL hash aggregates; --include-text is still required for statements."), CommandGuide("capture", "Save a resumable XLog capture, optionally including profiles.", xlogOptions + "--directory", connectionOptions + pagingOptions + setOf("--resume", "--include-profiles", "--include-text"), "capture $windowExample --directory capture --user reader --no-input", connectionNotes + listOf("--include-text requires --include-profiles. Text can contain sensitive values.", "Use the same query and directory with --resume. Profile capture defaults to page size 25, maximum 100.")), - CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions, - "inspect --directory capture --txid 123"), + CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions + "--group-sql", + "inspect --directory capture --txid 123", listOf("Use --group-sql for SQL hash aggregates; --include-text requires stored text.")), CommandGuide("inspect search", "Search captured requests offline.", setOf("--directory"), offlineOptions + setOf("--limit", "--min-elapsed-ms", "--errors-only", "--service-hash", "--service-name"), "inspect search --directory capture --errors-only", listOf("--service-name requires --include-text and stored dictionary text.")), diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt index 654e2c5..aa484fa 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt @@ -8,6 +8,9 @@ import java.time.format.DateTimeFormatter internal fun Cli.interpret(): CommandRequest { val command = words.joinToString(" ") + if (spec.commandLine().parseResult.hasMatchedOption("--group-sql") && command !in setOf("profile get", "inspect")) { + invalid("--group-sql requires profile get or inspect") + } if (command in setOf("context add", "context list", "context use")) return contextRequest(command) if (name != null) invalid("--name requires context add/use") if (command == "inspect compare") { @@ -65,7 +68,7 @@ internal fun Cli.interpret(): CommandRequest { } "inspect" -> { if (resume) invalid("--resume requires capture") - OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText) + OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText, groupSql) } else -> onlineRequest(command) } @@ -110,7 +113,7 @@ private fun Cli.onlineRequest(command: String): OnlineRequest { return when (command) { "doctor", "auth login" -> OnlineRequest.Doctor(command == "auth login") "object list" -> OnlineRequest.Objects(date?.also(::validateStorageDate)) - "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText) + "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText, groupSql) "xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages, includeText) "capture" -> OnlineRequest.Capture( required(query, "query"), required(directory, "--directory"), resume, maxPages, includeProfiles, includeText, diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt index 29519c9..bc82b40 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt @@ -11,7 +11,7 @@ sealed interface ContextRequest : CommandRequest { data class Use(val name: String?) : ContextRequest } sealed interface OfflineRequest : CommandRequest { - data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean) : OfflineRequest + data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OfflineRequest data class Search(val directory: Path, val filter: EvidenceFilter, val includeText: Boolean) : OfflineRequest data class Summary(val directory: Path, val options: ServiceSummaryOptions) : OfflineRequest data class Compare(val before: Path, val after: Path, val options: ComparisonOptions) : OfflineRequest @@ -26,7 +26,7 @@ sealed interface OnlineRequest : CommandRequest { data class CounterHistory(val query: CounterHistoryQuery) : OnlineRequest data class Counters(val objHash: Int, val name: String?) : OnlineRequest data class Objects(val date: String?) : OnlineRequest - data class Profile(val date: String, val txid: Long, val includeText: Boolean) : OnlineRequest + data class Profile(val date: String, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OnlineRequest data class Search(val query: Query, val maxPages: Int, val includeText: Boolean = false) : OnlineRequest data class Capture(val query: Query, val directory: Path, val resume: Boolean, val maxPages: Int, val includeProfiles: Boolean, val includeText: Boolean) : OnlineRequest diff --git a/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt index 963d318..72c50cf 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt @@ -16,7 +16,9 @@ class OfflineCommands(private val repository: (Path) -> EvidenceRepository, priv } private fun inspect(request: OfflineRequest.Inspect): Int { val result = inspectCapture(repository(request.directory), request.txid, request.includeText) - return output.result(json.encodeToJsonElement(result.toDto()), result.captureExecutionComplete && result.unavailableProfiles == 0 && + val data = if (request.groupSql) result.toSqlSummaryJson(result.record.profile?.let { summarizeSql(it, request.includeText) }) + else json.encodeToJsonElement(result.toDto()) + return output.result(data, result.captureExecutionComplete && result.unavailableProfiles == 0 && result.record.profile?.completeness != ProfileCompleteness.TRUNCATED) } private fun search(request: OfflineRequest.Search): Int { diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 053ae70..670f292 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -49,7 +49,8 @@ class OnlineCommands( } private fun profile(source: ObservationSource, request: OnlineRequest.Profile): Int { val profile = source.profile(request.date, request.txid, request.includeText, includeStepDetails = true) - return output.result(profile.toLiveJson(), profile.completeness != ProfileCompleteness.TRUNCATED) + val data = if (request.groupSql) json.encodeToJsonElement(summarizeSql(profile, request.includeText).toDto()) else profile.toLiveJson() + return output.result(data, profile.completeness != ProfileCompleteness.TRUNCATED) } private fun capture(source: ObservationSource, endpoint: Endpoint, request: OnlineRequest.Capture): Int { diff --git a/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt b/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt new file mode 100644 index 0000000..93e4719 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt @@ -0,0 +1,11 @@ +package dev.scouterctl.core + +data class SqlAggregate( + val hash: Int, val statement: DictionaryText?, val statementConflict: Boolean, + val count: Int, val elapsedTotalMs: Long, val maxElapsedMs: Int, val errors: Int, +) + +data class ProfileSqlSummary( + val txid: String, val completeness: ProfileCompleteness, val textIncluded: Boolean, + val totalSteps: Int, val sqlSteps: Int, val nonSqlSteps: Int, val groups: List, +) diff --git a/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt b/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt new file mode 100644 index 0000000..181285d --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt @@ -0,0 +1,109 @@ +package dev.scouterctl + +import dev.scouterctl.adapters.files.EvidenceStore +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.* +import java.nio.file.Files + +class ProfileSqlGroupingTest : FunSpec({ + for (text in listOf(false, true)) { + test("profile get groups 3000 repeated SQL steps with text policy $text") { + WireFixture(Fault.PROFILE_REPEATED_SQL).use { server -> + val args = listOf("profile", "get", "--date", "20261101", "--txid", WireFixture.ids.first().toString(), "--group-sql") + + if (text) listOf("--include-text") else emptyList() + val result = ContractCli(tempdir().toPath()).run(server, args) + result.exit shouldBe 0 + result.stderr shouldBe "" + val summary = result.data() + summary["schemaVersion"] shouldBe JsonPrimitive("scouterctl.profile-sql-summary/v1") + summary["txid"] shouldBe JsonPrimitive(WireFixture.ids.first().toString()) + summary["totalSteps"] shouldBe JsonPrimitive(3004) + summary["sqlSteps"] shouldBe JsonPrimitive(3000) + summary["nonSqlSteps"] shouldBe JsonPrimitive(4) + summary["completeness"] shouldBe JsonPrimitive("unknown") + summary.containsKey("steps") shouldBe false + val group = summary["groups"]!!.jsonArray.single().jsonObject + group["hash"] shouldBe JsonPrimitive(123) + group["count"] shouldBe JsonPrimitive(3000) + group["elapsedTotalMs"] shouldBe JsonPrimitive(6000) + group["maxElapsedMs"] shouldBe JsonPrimitive(2) + group["errors"] shouldBe JsonPrimitive(30) + if (text) group["statement"]!!.jsonObject["text"] shouldBe JsonPrimitive("select marker from fixture where id = ?") + else group["statement"] shouldBe JsonNull + result.output shouldNotContain "secret-token" + result.output shouldNotContain "fixture-password" + } + } + } + + test("grouped offline inspection redacts by request and never rewrites stored steps") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + WireFixture().use { server -> + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--include-text", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 0 + } + val before = Files.list(directory).use { files -> files.toList().associate { it.fileName.toString() to Files.readString(it) } } + val args = listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids.first().toString(), "--group-sql") + for (text in listOf(false, true)) { + val result = runOfflineCli(work, args + if (text) listOf("--include-text") else emptyList()) + result.exit shouldBe 0 + val inspection = result.data() + inspection["textIncluded"] shouldBe JsonPrimitive(text) + val summary = inspection["record"]!!.jsonObject["profile"]!!.jsonObject + summary["schemaVersion"] shouldBe JsonPrimitive("scouterctl.profile-sql-summary/v1") + summary["groups"]!!.jsonArray.map { it.jsonObject["hash"]!!.jsonPrimitive.int } shouldBe listOf(456, 123) + if (!text) result.output shouldNotContain "select marker" + else summary["groups"]!!.jsonArray.last().jsonObject["statement"]!!.jsonObject["text"] shouldBe + JsonPrimitive("select marker from fixture where id = ?") + } + Files.list(directory).use { files -> files.toList().associate { it.fileName.toString() to Files.readString(it) } } shouldBe before + val raw = runOfflineCli(work, args.dropLast(1)) + raw.data()["record"]!!.jsonObject["profile"]!!.jsonObject["steps"]!!.jsonArray.size shouldBe 3 + + val manifest = EvidenceStore(directory).load() + Files.writeString(directory.resolve(manifest.pages.last().name), "damaged later page") + runOfflineCli(work, args).exit shouldBe 10 + } + + for (offline in listOf(false, true)) { + test("grouped ${if (offline) "inspect" else "profile"} preserves explicit profile truncation") { + val work = tempdir().toPath() + WireFixture(Fault.PROFILE_TRUNCATED).use { server -> + val result = if (offline) { + val directory = work.resolve("capture") + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 0 + runOfflineCli(work, listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids.first().toString(), "--group-sql")) + } else ContractCli(work).run(server, listOf("profile", "get", "--date", "20261101", "--txid", WireFixture.ids.first().toString(), "--group-sql")) + result.exit shouldBe 8 + result.events.single()["executionComplete"] shouldBe JsonPrimitive(false) + val profile = if (offline) result.data()["record"]!!.jsonObject["profile"]!!.jsonObject else result.data() + profile["completeness"] shouldBe JsonPrimitive("truncated") + profile["groups"]!!.jsonArray.single().jsonObject["count"] shouldBe JsonPrimitive(1) + } + } + } + + test("grouped inspect retains unavailable profiles and refuses text that was not stored") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + WireFixture(Fault.PROFILE_MISSING).use { server -> + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 8 + } + val args = listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids[1].toString(), "--group-sql") + val result = runOfflineCli(work, args) + result.exit shouldBe 8 + result.data()["record"]!!.jsonObject["profile"] shouldBe JsonNull + runOfflineCli(work, args + "--include-text").exit shouldBe 2 + } + + for (command in listOf(listOf("doctor"), listOf("capture"), listOf("inspect", "search"), listOf("context", "list"))) { + test("${command.joinToString(" ")} rejects the SQL grouping option before execution") { + val result = runOfflineCli(tempdir().toPath(), command + "--group-sql") + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["message"]!!.jsonPrimitive.content.contains("--group-sql") shouldBe true + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt b/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt new file mode 100644 index 0000000..f8bd3a7 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt @@ -0,0 +1,76 @@ +package dev.scouterctl + +import dev.scouterctl.application.summarizeSql +import dev.scouterctl.core.* +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe + +class SqlSummaryTest : FunSpec({ + test("SQL aggregates preserve signed hashes, wide duration totals, errors and profile scope") { + val profile = ProfileRecord("9223372036854775806", listOf( + ProfileStep("SQL3", 0, SqlProfile(10, Int.MAX_VALUE, 0)), + ProfileStep("SQL", 1, SqlProfile(-7, 20, -99)), + ProfileStep("SQL3", 2, SqlProfile(10, Int.MAX_VALUE, 7)), + ProfileStep("SQL", 3, SqlProfile(-7, -3, 0)), + ProfileStep("SOCKET", 4), + ), completeness = ProfileCompleteness.TRUNCATED) + + val summary = summarizeSql(profile, false) + + summary.txid shouldBe profile.txid + summary.completeness shouldBe ProfileCompleteness.TRUNCATED + summary.totalSteps shouldBe 5 + summary.sqlSteps shouldBe 4 + summary.nonSqlSteps shouldBe 1 + summary.groups shouldBe listOf( + SqlAggregate(10, null, false, 2, 4_294_967_294L, Int.MAX_VALUE, 1), + SqlAggregate(-7, null, false, 2, 17L, 20, 1), + ) + } + + for (steps in listOf(emptyList(), listOf(ProfileStep("MESSAGE", 0), ProfileStep("SOCKET", 1)))) { + test("a profile with ${steps.size} non-SQL steps has no invented SQL groups") { + val summary = summarizeSql(ProfileRecord("42", steps), true) + summary.txid shouldBe "42" + summary.totalSteps shouldBe steps.size + summary.sqlSteps shouldBe 0 + summary.nonSqlSteps shouldBe steps.size + summary.groups shouldBe emptyList() + } + } + + test("statement resolution distinguishes consistent, conflicting, missing and mismatched dictionaries") { + val sql = listOf( + SqlProfile(7, 1, 0, statement = DictionaryText(7, "select a")), + SqlProfile(7, 1, 0, statement = DictionaryText(7, "select b")), + SqlProfile(8, 1, 0, statement = DictionaryText(999, "wrong statement")), + SqlProfile(9, 1, 0, statement = DictionaryText(9, "select c")), + SqlProfile(9, 1, 0, statement = DictionaryText(9, "select c")), + SqlProfile(10, 1, 0), + SqlProfile(11, 1, 0, statement = DictionaryText(11)), + ) + val summary = summarizeSql(ProfileRecord("42", sql.mapIndexed { index, value -> ProfileStep("SQL3", index, value) }), true) + val groups = summary.groups.associateBy { it.hash } + groups.getValue(7).statementConflict shouldBe true + groups.getValue(7).statement shouldBe DictionaryText(7) + groups.getValue(8).statement shouldBe DictionaryText(8) + groups.getValue(9).statementConflict shouldBe false + groups.getValue(9).statement shouldBe DictionaryText(9, "select c") + groups.getValue(10).statement shouldBe DictionaryText(10) + groups.getValue(11).statement shouldBe DictionaryText(11) + } + + test("redacted grouping cannot reveal supplied SQL text or text conflicts") { + val profile = ProfileRecord("42", listOf( + ProfileStep("SQL3", 0, SqlProfile(7, 1, -9, statement = DictionaryText(7, "private statement a"), error = DictionaryText(-9, "private error"))), + ProfileStep("SQL3", 1, SqlProfile(7, 1, 0, statement = DictionaryText(7, "private statement b"))), + )) + summarizeSql(profile, false).groups.single() shouldBe SqlAggregate(7, null, false, 2, 2, 1, 1) + } + + test("equal duration groups use signed hash order for stable output") { + val profile = ProfileRecord("42", listOf(ProfileStep("SQL3", 0, SqlProfile(9, 3, 0)), + ProfileStep("SQL3", 1, SqlProfile(-7, 3, 0)))) + summarizeSql(profile, false).groups.map { it.hash } shouldBe listOf(-7, 9) + } +}) diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index 82f0743..65f46b9 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -32,7 +32,7 @@ enum class Fault { NONE, STATUS_MALFORMED, CONFIG_MALFORMED, CONFIG_DUPLICATE, CONFIG_INVALID_VISIBLE, CONFIG_INVALID_AGENT_TEXT, VERSION, AUTH, SESSION, MISSING_META, INVALID_META, STALLED_CURSOR, DISCONNECT, EMPTY_OBJECTS, STRUCTURAL_ONLY, MISSING_TEXT, EMPTY_TEXT, MANY_SQL, TEXT_WRONG_TYPE, TEXT_WRONG_HASH, TEXT_DUPLICATE, TEXT_PACK, TEXT_DISCONNECT, TEXT_SESSION, XLOG_MISSING, XLOG_WRONG_ID, - PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, + PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_REPEATED_SQL, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, } class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPassword: String? = null, @@ -289,7 +289,13 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas }, ApiCallStep2().apply { index = 4; parent = 1; start_time = 650; elapsed = 50 }, StepControl().apply { code = 7; message = "private-control-message" }, - )) else if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( + )) else if (fault == Fault.PROFILE_REPEATED_SQL) Step.toBytes(Array(3004) { number -> + if (number < 4) HashedMessageStep().apply { index = number; hash = 77 } + else SqlStep3().apply { + index = number; hash = 123; elapsed = 2; error = if (number % 100 == 0) -789 else 0 + param = "password=secret-token" + } + }) else if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( SqlStep3().apply { index = 1; hash = 123; elapsed = 7 }, StepControl().apply { code = if (fault == Fault.PROFILE_TRUNCATED) 0 else 7; message = " ** Profile Truncated ** " } )) else if (fault == Fault.PROFILE_MANY_STEPS) Step.toBytes(Array(1001) { number -> From 609294faa3b9627b100dc8dce73a5c4f31512c0a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 12 Nov 2026 15:18:59 +0900 Subject: [PATCH 15/17] =?UTF-8?q?ci:=20=F0=9F=91=B7=20Update=20Java=20setu?= =?UTF-8?q?p=20and=20GitHub=20Script=20actions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why Update the Java setup and GitHub Script actions while preserving the artifact API required by local act runs. ## Changes - Update `actions/setup-java` from v6.1.0 to v6.2.0 and `actions/github-script` from v8 to v9.0.0, pinned to full commit SHAs. - Keep the three act-only artifact uploads on v4.6.2. Hosted uploads already use v7 and remain unchanged. - Synchronize the branch with current develop, including SQL grouping. ## Validation - act v0.2.97 workflow validation passed; both workflows and reusable job graph resolve. - Ran the actual comment script with v9's pinned execution function on Node24 across seven scenarios, including creation/update, stale/closed PRs, missing/oversized reports and report content treated as text. External GitHub API calls were replaced locally. - Confirmed upstream tags/SHAs, Node24 runtime and retained input compatibility. Full local act execution was unavailable because Docker Desktop's Linux engine/WSL integration is unavailable. No environment settings were changed. Latest-commit hosted CI must pass before merge. The develop workflow skips Windows packaging and scale/release jobs; they are not claimed as validated by this PR run. ## Compatibility and risks Existing workflow permissions, conditions, script content and artifact inputs are unchanged. Local act upload compatibility keeps its existing v4 pin; the v9 script uses the supplied GitHub client and Node built-ins, avoiding the removed CommonJS package import and injected-name conflicts. --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: oxspan <36991763+oxspan@users.noreply.github.com> --- .github/workflows/capture-scale.yml | 4 ++-- .github/workflows/ci-release.yml | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/capture-scale.yml b/.github/workflows/capture-scale.yml index bb017f1..a215f04 100644 --- a/.github/workflows/capture-scale.yml +++ b/.github/workflows/capture-scale.yml @@ -16,7 +16,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + - uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0 with: distribution: temurin java-version: '21' @@ -51,7 +51,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + - uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0 with: distribution: temurin java-version: '21' diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.yml index 7d044bb..f77c481 100644 --- a/.github/workflows/ci-release.yml +++ b/.github/workflows/ci-release.yml @@ -55,7 +55,7 @@ jobs: master:develop|master:hotfix/*|master:release-please--branches--master) ;; *) echo "Unsupported PR route: $HEAD_BRANCH -> $BASE_BRANCH" >&2; exit 1 ;; esac - - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + - uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0 with: distribution: temurin java-version: '21' @@ -164,7 +164,7 @@ jobs: name: pr-test-summary path: ci-summary - name: Create or update the PR comment - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: BUILD_RESULT: ${{ needs.build.result }} with: @@ -212,7 +212,7 @@ jobs: with: ref: ${{ inputs.release_tag || github.sha }} persist-credentials: false - - uses: actions/setup-java@b843ebf9b913146b38da2eaa7aeddf76074ae1ad # v6.1.0 + - uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0 with: distribution: temurin java-version: '21' From 10c2019d70fe25a8aa7d6e721f85754435b5b6e0 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Fri, 13 Nov 2026 00:11:25 +0900 Subject: [PATCH 16/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Validate=20reported?= =?UTF-8?q?=20Collector=20time=20zones?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When a Collector runs in KST, declaring the same instants as a Los Angeles storage day can return an empty successful stream. `doctor` and `collector status` now expose the reported JVM time-zone ID and the CLI-derived UTC offset at the Collector's clock time, with explicit provenance. Online XLog search, capture, counter history and counter export reject differing offsets with `INVALID_ARGUMENT` before streams or capture writes; equivalent offsets remain accepted, including historical/DST interval checks. The ID comes from the existing authenticated `SERVER_ENV` response, and the offset uses client JDK rules. Missing/unparseable metadata remains unknown. This does not certify the Collector's runtime default, TZDB or fixed-day storage calendar. Storage-date verification remains unverified, and capture schemas remain unchanged. English and Korean guides explain these limits. Validation: the complete Gradle regression suite passes, line coverage 95.35% and branch coverage 81.11%; release-input checks pass. The 22 new packaged CLI/zone-rule cases also pass on native Windows Java 21. Regression tests fail against the previous CLI and with deliberately broken fallback/DST handling. Actual Java 8 Collector 2.21.3 KST/LA/KST queries change from 3/0/3 records to 3/error2/3; v3 profile capture bytes match the previous CLI and remain readable by its strict reader. Refs #57 --- docs/README.md | 1 + docs/cli-contract.md | 2 + docs/collector-timezone.md | 40 +++++++ docs/inspection.md | 2 + docs/ko/README.md | 1 + docs/ko/collector-timezone.md | 19 ++++ .../adapters/collector/CollectorSource.kt | 5 + .../collector/CollectorTimezoneDecoder.kt | 21 ++++ .../serialization/CollectorTimezoneJson.kt | 21 ++++ .../application/CollectorTimezone.kt | 34 ++++++ .../dev/scouterctl/application/Ports.kt | 1 + .../dev/scouterctl/cli/OnlineCommands.kt | 11 +- .../dev/scouterctl/core/CollectorTimezone.kt | 3 + .../dev/scouterctl/CollectorTimezoneTest.kt | 103 ++++++++++++++++++ src/test/kotlin/dev/scouterctl/MemoryPorts.kt | 1 + .../kotlin/dev/scouterctl/RecoveryProxy.kt | 6 +- .../dev/scouterctl/TimezoneRulesTest.kt | 53 +++++++++ src/test/kotlin/dev/scouterctl/WireFixture.kt | 14 +++ 18 files changed, 334 insertions(+), 4 deletions(-) create mode 100644 docs/collector-timezone.md create mode 100644 docs/ko/collector-timezone.md create mode 100644 src/main/kotlin/dev/scouterctl/adapters/collector/CollectorTimezoneDecoder.kt create mode 100644 src/main/kotlin/dev/scouterctl/adapters/serialization/CollectorTimezoneJson.kt create mode 100644 src/main/kotlin/dev/scouterctl/application/CollectorTimezone.kt create mode 100644 src/main/kotlin/dev/scouterctl/core/CollectorTimezone.kt create mode 100644 src/test/kotlin/dev/scouterctl/CollectorTimezoneTest.kt create mode 100644 src/test/kotlin/dev/scouterctl/TimezoneRulesTest.kt diff --git a/docs/README.md b/docs/README.md index 56b9583..1b7730a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,6 +23,7 @@ See [command and output examples](examples.md) for actual response shapes. - [Capture formats and older files](capture-compatibility.md) - [Interrupted capture recovery](capture-recovery.md) - [Profile completeness](profile-completeness.md) +- [Collector time zone and query checks](collector-timezone.md) - [Grouped profile SQL output](profile-sql-summary.md) - [Security](../SECURITY.md) diff --git a/docs/cli-contract.md b/docs/cli-contract.md index 41eda1c..f0c910e 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -71,6 +71,8 @@ A consumer should accept the stream schema, process records, then require both t `storageDateVerification: "unverified"` means the CLI used the supplied XLog storage date without confirmation from the Collector. An incorrect date can produce an empty completed query. XLog `capture` reports the same field in its outer result envelope. +Online window queries reject offset mismatches against the reported Collector JVM zone before streaming or writing captures. `doctor` and `collector status` expose the property's ID and a client-rule offset with explicit provenance; unavailable metadata remains `null`. These checks do not verify the storage calendar. See [Collector time zones](collector-timezone.md). + `counter history` uses its own `scouterctl.counter-stream/v1` schema with the same start/record/complete lifecycle. Its start includes the dated target and counter metadata, and records contain stored timestamps and numeric strings. See [counter history](counter-history.md) for missing values, units and query limits. `counter export` returns a single manifest result and writes JSONL chunk files. Its `scouterctl.counter-capture/v1` format is separate from XLog captures. See [resumable counter export](counter-export.md) for budgets, checkpoints and completion semantics. diff --git a/docs/collector-timezone.md b/docs/collector-timezone.md new file mode 100644 index 0000000..b1aec26 --- /dev/null +++ b/docs/collector-timezone.md @@ -0,0 +1,40 @@ +# Collector time zone + +English | [한국어](ko/collector-timezone.md) + +`doctor` and `collector status` report a `timezone` object alongside the existing `timezoneDisplayName`. For `collector status`, it is inside `data.collector`: + +```json +{ + "id": "Asia/Seoul", + "utcOffsetSeconds": 32400, + "referenceTimeEpochMs": 1793491200000, + "idSource": "jvm-property", + "offsetSource": "client-zone-rules" +} +``` + +The ID comes from the Collector's reported JVM `user.timezone` property through the existing authenticated `SERVER_ENV` request. Java short IDs, when recognized, are expanded using Java's short-ID mapping. The offset is calculated by the **CLI's time-zone rules** at the Collector's `SERVER_TIME` timestamp; it is not a remotely measured offset. `timezoneDisplayName` remains the localized login label and is not used to infer an ID. Other JVM environment values are not included in output. + +If the property is missing or cannot be parsed, `timezone` is `null`. No CLI-host zone or display-name guess is substituted. Invalid protocol types fail as protocol errors without printing their values. A configured property can differ from a runtime default changed after startup, and Collector/CLI time-zone database versions can differ; read the provenance before treating the values as proof of runtime state. + +## Query mismatch checks + +Online `xlog search`, `capture`, `counter history` and `counter export` compare `--timezone` offsets against the reported Collector zone using client rules over the requested half-open interval. Different IDs with equal offsets throughout the interval are accepted. The check uses the query's instants, including internal DST transitions, rather than comparing a historical query with today's offset. + +A mismatch returns `INVALID_ARGUMENT` (exit 2), naming `--timezone` and the reported Collector zone, before emitting stream events or creating capture files. There is no override option. With unavailable zone metadata, the previous unverified query behavior remains. Transport and malformed-response errors still fail normally; they are not silently treated as missing metadata. + +For a Collector reporting `Asia/Seoul`, this declared date/zone combination matches: + +```sh +scouterctl xlog search --object-hash 42 --date 20261109 --timezone Asia/Seoul \ + --from 2026-11-09T13:00:00+09:00 --to 2026-11-09T13:01:00+09:00 --no-input +``` + +The same instants declared as `America/Los_Angeles` with date `20261108` are rejected. A UTC timestamp suffix is still valid input with `--timezone Asia/Seoul`; the timestamp syntax itself is not required to use the Collector's offset. + +## Storage dates remain unverified + +This is a comparison with a **reported JVM zone**, not confirmation of the storage partition. XLog `storageDateVerification: "unverified"` and capture `timezoneVerification: "operator-declared"` remain unchanged. Capture formats and strict-reader compatibility are unchanged; the new diagnostic fields are live output only. + +Scouter's `DateTimeHelper` in 2.8.1 and 2.21.3 builds a calendar from local `20000101` with fixed 86,400,000-ms days. That calendar can differ from civil time around DST. For example, in `America/Chicago`, `2026-07-01T05:30:00Z` is July 1 by current zone rules but June 30 in that storage calendar. A matching zone check cannot certify the correct partition or prove that an empty result means no traffic. Confirm storage dates with the Collector operator when results are unexpected. This change does not repair that upstream calendar behavior or verify the server's cached calendar through the protocol. diff --git a/docs/inspection.md b/docs/inspection.md index 86e4332..98fbb5e 100644 --- a/docs/inspection.md +++ b/docs/inspection.md @@ -12,6 +12,8 @@ scouterctl collector config --key net_tcp_listen_port Status reports `usedHeapBytes`, `committedHeapBytes` and `serverTimeEpochMs`. Committed heap is not the JVM maximum heap. Collector metadata accompanies both results. +`doctor` and status also report the [Collector's JVM time-zone ID and client-rule offset](collector-timezone.md), including their provenance and reference time. This is separate from storage-date verification. + Configuration lists runtime settings exposed by the Collector alongside their defaults. Values are strings. The CLI shows values only for selected port, HTTP-enable and retention settings; other entries have `value: null`, `defaultValue: null` and `redacted: true`. Filtering by `--key` does not reveal hidden values. A missing setting exits 7. This is not a raw configuration-file export, and it does not include arbitrary custom properties. These commands do not write settings or require a local Agent. They authenticate with the selected Collector account. Successful access does not prove that the Collector enforces distinct permissions for every operation; `effectivePermission` remains `unknown`. diff --git a/docs/ko/README.md b/docs/ko/README.md index 3479bf2..772988e 100644 --- a/docs/ko/README.md +++ b/docs/ko/README.md @@ -12,6 +12,7 @@ - [설치·업데이트·이전 버전으로 전환·제거](installation.md) - [빠른 시작](../../QUICKSTART.ko.md) - [연결과 비밀번호](connections.md) +- [Collector 시간대와 조회 검사](collector-timezone.md) - [프로파일 SQL 집계](profile-sql-summary.md) - [기여 안내](../../CONTRIBUTING.ko.md) · [보안 안내](../../SECURITY.ko.md) diff --git a/docs/ko/collector-timezone.md b/docs/ko/collector-timezone.md new file mode 100644 index 0000000..7512e5b --- /dev/null +++ b/docs/ko/collector-timezone.md @@ -0,0 +1,19 @@ +# Collector 시간대 + +[English](../collector-timezone.md) | 한국어 + +`doctor`와 `collector status`에 기존 `timezoneDisplayName` 외에 `timezone` 객체를 추가합니다. 상태 조회에서는 `data.collector.timezone`에 있습니다. + +```json +{"id":"Asia/Seoul","utcOffsetSeconds":32400,"referenceTimeEpochMs":1793491200000,"idSource":"jvm-property","offsetSource":"client-zone-rules"} +``` + +ID는 인증된 기존 `SERVER_ENV` 요청으로 받은 JVM `user.timezone` 속성입니다. 인식 가능한 Java 단축 ID는 Java 매핑으로 확장합니다. 오프셋은 Collector의 `SERVER_TIME` 시각을 기준으로 **CLI의 시간대 규칙으로 계산한 값**이며, 서버에서 직접 측정한 오프셋은 아닙니다. 지역화된 표시명으로 ID를 추정하지 않으며 다른 JVM 환경 값은 출력하지 않습니다. 속성이 없거나 해석할 수 없으면 `timezone: null`입니다. 잘못된 프로토콜 타입은 값을 공개하지 않고 오류로 처리합니다. + +`xlog search`, `capture`, `counter history`, `counter export`는 요청 구간의 오프셋을 보고된 Collector 시간대와 비교합니다. ID가 달라도 구간 전체의 오프셋이 같으면 허용합니다. 과거 조회를 현재 오프셋과 비교하지 않으며 구간 내부의 DST 전환과 종료 시각 미포함 조건도 검사합니다. 불일치하면 `--timezone`과 Collector 시간대를 명시한 `INVALID_ARGUMENT`(종료 코드 2)을 스트림 시작·캡처 파일 생성 전에 반환합니다. 우회 옵션은 없습니다. 메타데이터가 없으면 기존의 미검증 조회 동작을 유지하며, 통신·응답 오류를 미확인 상태로 숨기지 않습니다. + +예를 들어 `Asia/Seoul`을 보고한 Collector에서는 같은 시각을 `America/Los_Angeles`와 그 지역 날짜로 선언한 요청을 거부합니다. `--from`·`--to`를 UTC 문자열로 주면서 `--timezone Asia/Seoul`을 지정하는 방식은 계속 사용할 수 있습니다. + +이 검사는 보고된 JVM 시간대와의 비교입니다. 실행 중 변경된 기본 시간대, 서버와 CLI의 시간대 데이터 차이, 실제 저장 파티션까지 보장하지 않습니다. XLog의 `storageDateVerification: "unverified"`와 캡처의 `timezoneVerification: "operator-declared"`는 유지합니다. 새 진단 필드는 실시간 출력에만 추가하며 캡처 형식은 바꾸지 않습니다. + +Scouter 2.8.1·2.21.3의 `DateTimeHelper`는 해당 지역의 `20000101`을 기준으로 하루를 86,400,000ms씩 증가시킵니다. 이 저장 달력은 DST가 반영된 실제 지역 날짜와 다를 수 있습니다. `America/Chicago`에서 `2026-07-01T05:30:00Z`는 지역 시간상 7월 1일이지만 저장 달력에서는 6월 30일입니다. 시간대가 일치해도 빈 결과를 트래픽 부재로 단정하지 말고, 예상과 다르면 운영자와 저장 날짜를 확인하세요. 이번 변경은 이 Scouter 저장 달력 동작을 수정하지 않습니다. 자세한 예시는 [영문 가이드](../collector-timezone.md)를 참고하세요. diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt index db67c46..4eb3733 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorSource.kt @@ -39,6 +39,11 @@ class CollectorSource private constructor(private val socket: Socket, private va private var session = 0L override lateinit var info: ServerInfo private set + private val reportedTimezone by lazy { + collectorOperation { decodeCollectorTimezone(exchange(RequestCmd.SERVER_ENV), exchange(RequestCmd.SERVER_TIME)) } + } + + override fun timezone(): CollectorTimezone? = reportedTimezone companion object { fun connect(endpoint: Endpoint, password: String, clientVersion: String): CollectorSource = collectorOperation { diff --git a/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorTimezoneDecoder.kt b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorTimezoneDecoder.kt new file mode 100644 index 0000000..496bb1c --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/collector/CollectorTimezoneDecoder.kt @@ -0,0 +1,21 @@ +package dev.scouterctl.adapters.collector + +import dev.scouterctl.application.describeCollectorTimezone +import dev.scouterctl.core.* +import scouter.lang.pack.MapPack +import scouter.lang.pack.Pack +import scouter.lang.value.DecimalValue +import scouter.lang.value.TextValue + +internal fun decodeCollectorTimezone(environment: List, clock: List): CollectorTimezone? { + if (environment.isEmpty()) return null + val properties = environment.singleOrNull() as? MapPack + ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid Collector environment response") + val value = properties.get("user.timezone") ?: return null + val id = (value as? TextValue)?.value + ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid Collector time zone field") + val time = ((clock.singleOrNull() as? MapPack)?.get("time") as? DecimalValue)?.value + ?: throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid Collector clock response") + if (time < 0) throw OperationFailure(ErrorCode.PROTOCOL_ERROR, "Invalid Collector clock value") + return describeCollectorTimezone(id, time) +} diff --git a/src/main/kotlin/dev/scouterctl/adapters/serialization/CollectorTimezoneJson.kt b/src/main/kotlin/dev/scouterctl/adapters/serialization/CollectorTimezoneJson.kt new file mode 100644 index 0000000..a3d0049 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/serialization/CollectorTimezoneJson.kt @@ -0,0 +1,21 @@ +package dev.scouterctl.adapters.serialization + +import dev.scouterctl.core.* +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.* + +@Serializable +data class CollectorTimezoneDto( + val id: String, val utcOffsetSeconds: Int, val referenceTimeEpochMs: Long, + val idSource: String = "jvm-property", val offsetSource: String = "client-zone-rules", +) + +private fun CollectorTimezone?.toJson(): JsonElement = this?.let { + json.encodeToJsonElement(CollectorTimezoneDto(it.id, it.utcOffsetSeconds, it.referenceTimeEpochMs)) +} ?: JsonNull + +fun ServerInfo.toLiveJson(timezone: CollectorTimezone?): JsonObject = + JsonObject(json.encodeToJsonElement(toDto()).jsonObject + ("timezone" to timezone.toJson())) + +fun CollectorStatus.toLiveJson(info: ServerInfo, timezone: CollectorTimezone?): JsonObject = + JsonObject(json.encodeToJsonElement(toDto(info)).jsonObject + ("collector" to info.toLiveJson(timezone))) diff --git a/src/main/kotlin/dev/scouterctl/application/CollectorTimezone.kt b/src/main/kotlin/dev/scouterctl/application/CollectorTimezone.kt new file mode 100644 index 0000000..3da0414 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/application/CollectorTimezone.kt @@ -0,0 +1,34 @@ +package dev.scouterctl.application + +import dev.scouterctl.core.* +import java.time.DateTimeException +import java.time.Instant +import java.time.ZoneId + +fun describeCollectorTimezone(id: String, referenceTimeEpochMs: Long): CollectorTimezone? { + val zone = try { ZoneId.of(id, ZoneId.SHORT_IDS) } catch (_: DateTimeException) { return null } + val offset = zone.rules.getOffset(Instant.ofEpochMilli(referenceTimeEpochMs)) + return CollectorTimezone(zone.id, offset.totalSeconds, referenceTimeEpochMs) +} + +fun validateCollectorTimezone(declared: String, from: Long, to: Long, collector: CollectorTimezone?) { + if (collector == null) return + val requested = ZoneId.of(declared).rules + val reported = ZoneId.of(collector.id).rules + val start = Instant.ofEpochMilli(from) + val end = Instant.ofEpochMilli(to) + fun check(instant: Instant) { + if (requested.getOffset(instant) != reported.getOffset(instant)) { + throw OperationFailure(ErrorCode.INVALID_ARGUMENT, + "--timezone $declared differs from reported Collector time zone ${collector.id} in the requested interval") + } + } + check(start) + for (rules in listOf(requested, reported)) { + var transition = rules.nextTransition(start) + while (transition != null && transition.instant < end) { + check(transition.instant) + transition = rules.nextTransition(transition.instant) + } + } +} diff --git a/src/main/kotlin/dev/scouterctl/application/Ports.kt b/src/main/kotlin/dev/scouterctl/application/Ports.kt index 448e184..b130c7b 100644 --- a/src/main/kotlin/dev/scouterctl/application/Ports.kt +++ b/src/main/kotlin/dev/scouterctl/application/Ports.kt @@ -4,6 +4,7 @@ import dev.scouterctl.core.* interface ObservationSource : AutoCloseable { val info: ServerInfo + fun timezone(): CollectorTimezone? fun status(): CollectorStatus fun settings(): List fun agentStatus(objHash: Int): AgentStatus diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 670f292..ff4b597 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -22,8 +22,15 @@ class OnlineCommands( val secret = password.read(connection.interactive, connection.noInput, request is OnlineRequest.Doctor && request.login) return source(endpoint, secret).use { observation -> when (request) { - is OnlineRequest.Doctor -> output.result(json.encodeToJsonElement(observation.info.toDto())) - OnlineRequest.CollectorStatus -> output.result(json.encodeToJsonElement(observation.status().toDto(observation.info))) + is OnlineRequest.Search -> validateCollectorTimezone(request.query.timezone, request.query.from, request.query.to, observation.timezone()) + is OnlineRequest.Capture -> validateCollectorTimezone(request.query.timezone, request.query.from, request.query.to, observation.timezone()) + is OnlineRequest.CounterHistory -> validateCollectorTimezone(request.query.timezone, request.query.from, request.query.to, observation.timezone()) + is OnlineRequest.CounterExport -> validateCollectorTimezone(request.query.timezone, request.query.from, request.query.to, observation.timezone()) + else -> Unit + } + when (request) { + is OnlineRequest.Doctor -> output.result(observation.info.toLiveJson(observation.timezone())) + OnlineRequest.CollectorStatus -> output.result(observation.status().toLiveJson(observation.info, observation.timezone())) is OnlineRequest.CollectorConfig -> output.result(json.encodeToJsonElement(collectorSettings(observation, request.key).toSettingsDto(observation.info))) is OnlineRequest.AgentStatus -> output.result(json.encodeToJsonElement(observation.agentStatus(request.objHash).toDto(observation.info))) is OnlineRequest.AgentConfig -> output.result(json.encodeToJsonElement(agentSettings(observation, request.objHash, request.key).toDto(observation.info))) diff --git a/src/main/kotlin/dev/scouterctl/core/CollectorTimezone.kt b/src/main/kotlin/dev/scouterctl/core/CollectorTimezone.kt new file mode 100644 index 0000000..c93153b --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/core/CollectorTimezone.kt @@ -0,0 +1,3 @@ +package dev.scouterctl.core + +data class CollectorTimezone(val id: String, val utcOffsetSeconds: Int, val referenceTimeEpochMs: Long) diff --git a/src/test/kotlin/dev/scouterctl/CollectorTimezoneTest.kt b/src/test/kotlin/dev/scouterctl/CollectorTimezoneTest.kt new file mode 100644 index 0000000..ac0a306 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/CollectorTimezoneTest.kt @@ -0,0 +1,103 @@ +package dev.scouterctl + +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.* +import java.nio.file.Files + +class CollectorTimezoneTest : FunSpec({ + for (command in listOf(listOf("doctor"), listOf("collector", "status"))) { + test("${command.joinToString(" ")} reports JVM zone and client-derived offset with provenance") { + WireFixture(Fault.TIMEZONE_SEOUL).use { server -> + val result = ContractCli(tempdir().toPath()).run(server, command) + result.exit shouldBe 0 + result.stderr shouldBe "" + val info = if (command.first() == "doctor") result.data() else result.data()["collector"]!!.jsonObject + val zone = info["timezone"]!!.jsonObject + zone["id"] shouldBe JsonPrimitive("Asia/Seoul") + zone["utcOffsetSeconds"] shouldBe JsonPrimitive(32400) + zone["referenceTimeEpochMs"] shouldBe JsonPrimitive(WireFixture.baseTime) + zone["idSource"] shouldBe JsonPrimitive("jvm-property") + zone["offsetSource"] shouldBe JsonPrimitive("client-zone-rules") + result.output shouldNotContain "private-zone-env-token" + } + } + } + + test("XLog timezone mismatch fails before a stream and equal-offset aliases remain usable") { + WireFixture(Fault.TIMEZONE_SEOUL).use { server -> + val cli = ContractCli(tempdir().toPath()) + val wrong = cli.run(server, listOf("xlog", "search") + evidenceWindow) + wrong.exit shouldBe 2 + val error = wrong.events.single()["error"]!!.jsonObject + error["code"] shouldBe JsonPrimitive("INVALID_ARGUMENT") + error["message"]!!.jsonPrimitive.content.contains("--timezone") shouldBe true + error["message"]!!.jsonPrimitive.content.contains("Asia/Seoul") shouldBe true + + val alias = cli.run(server, listOf("xlog", "search") + evidenceWindow.map { if (it == "UTC") "Asia/Tokyo" else it }) + alias.exit shouldBe 0 + alias.events.count { it["type"] == JsonPrimitive("record") } shouldBe 3 + alias.events.first()["storageDateVerification"] shouldBe JsonPrimitive("unverified") + } + } + + for (profiles in listOf(false, true)) { + test("timezone mismatch cannot create a ${if (profiles) "profile" else "XLog"} capture") { + val work = tempdir().toPath() + val destination = work.resolve("capture") + WireFixture(Fault.TIMEZONE_SEOUL).use { server -> + val result = ContractCli(work).run(server, listOf("capture", "--directory", destination.toString()) + evidenceWindow + + if (profiles) listOf("--include-profiles") else emptyList()) + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["code"] shouldBe JsonPrimitive("INVALID_ARGUMENT") + Files.exists(destination) shouldBe false + } + } + } + + for (export in listOf(false, true)) { + test("counter ${if (export) "export" else "history"} rejects the reported timezone mismatch") { + val work = tempdir().toPath() + val destination = work.resolve("counter-capture") + WireFixture(Fault.TIMEZONE_SEOUL).use { server -> + val window = listOf("--date", "20260101", "--from", "2026-01-01T00:00:00Z", "--to", "2026-01-01T00:01:00Z", + "--timezone", "UTC", "--object-hash", "42") + val args = if (export) listOf("counter", "export", "--counter", "HeapUsed", "--directory", destination.toString()) + window.drop(2) + else listOf("counter", "history", "--counter", "HeapUsed") + window + val result = ContractCli(work).run(server, args) + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["code"] shouldBe JsonPrimitive("INVALID_ARGUMENT") + Files.exists(destination) shouldBe false + } + } + } + + for (fault in listOf(Fault.TIMEZONE_MISSING, Fault.TIMEZONE_INVALID)) { + test("$fault leaves unknown metadata explicit and preserves unverified searches") { + WireFixture(fault).use { server -> + val cli = ContractCli(tempdir().toPath()) + val doctor = cli.run(server, listOf("doctor")) + doctor.exit shouldBe 0 + doctor.data()["timezone"] shouldBe JsonNull + doctor.output shouldNotContain "private-" + val search = cli.run(server, listOf("xlog", "search") + evidenceWindow) + search.exit shouldBe 0 + search.events.count { it["type"] == JsonPrimitive("record") } shouldBe 3 + search.events.first()["storageDateVerification"] shouldBe JsonPrimitive("unverified") + } + } + } + + for (fault in listOf(Fault.TIMEZONE_TYPE, Fault.TIMEZONE_TIME_TYPE)) { + test("$fault is a protocol error without exposing environment values") { + WireFixture(fault).use { server -> + val result = ContractCli(tempdir().toPath()).run(server, listOf("doctor")) + result.exit shouldBe 6 + result.events.single()["error"]!!.jsonObject["code"] shouldBe JsonPrimitive("PROTOCOL_ERROR") + result.output shouldNotContain "private-" + } + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt index 33e2e08..ede9ba2 100644 --- a/src/test/kotlin/dev/scouterctl/MemoryPorts.kt +++ b/src/test/kotlin/dev/scouterctl/MemoryPorts.kt @@ -39,6 +39,7 @@ internal class MemorySource( private val profile: (Long) -> ProfileRecord = { ProfileRecord(it.toString(), emptyList()) }, ) : ObservationSource { override val info = portSource + override fun timezone(): CollectorTimezone? = null var closed = false override fun status() = CollectorStatus(1, 2, 3) override fun settings(): List = emptyList() diff --git a/src/test/kotlin/dev/scouterctl/RecoveryProxy.kt b/src/test/kotlin/dev/scouterctl/RecoveryProxy.kt index adb0645..caf7ff7 100644 --- a/src/test/kotlin/dev/scouterctl/RecoveryProxy.kt +++ b/src/test/kotlin/dev/scouterctl/RecoveryProxy.kt @@ -71,9 +71,11 @@ internal class RecoveryProxy( val command = try { input.readText() } catch (_: EOFException) { return } check(command in setOf(RequestCmd.LOGIN, RequestCmd.TRANX_LOAD_TIME_GROUP_V2, RequestCmd.TRANX_PROFILE, RequestCmd.XLOG_READ_BY_TXID, RequestCmd.GET_TEXT_PACK, - RequestCmd.OBJECT_LIST_LOAD_DATE, RequestCmd.GET_XML_COUNTER, RequestCmd.COUNTER_PAST_TIME)) + RequestCmd.OBJECT_LIST_LOAD_DATE, RequestCmd.GET_XML_COUNTER, RequestCmd.COUNTER_PAST_TIME, + RequestCmd.SERVER_ENV, RequestCmd.SERVER_TIME)) val session = input.readLong() - val request = if (command == RequestCmd.GET_XML_COUNTER) null else input.readMapPack() + val request = if (command in setOf(RequestCmd.GET_XML_COUNTER, RequestCmd.SERVER_ENV, + RequestCmd.SERVER_TIME)) null else input.readMapPack() if (command == commandToPause && ++matches == occurrence) { entered.countDown() check(released.await(15, TimeUnit.SECONDS)) { "Recovery boundary was not released" } diff --git a/src/test/kotlin/dev/scouterctl/TimezoneRulesTest.kt b/src/test/kotlin/dev/scouterctl/TimezoneRulesTest.kt new file mode 100644 index 0000000..3a0a440 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/TimezoneRulesTest.kt @@ -0,0 +1,53 @@ +package dev.scouterctl + +import dev.scouterctl.application.describeCollectorTimezone +import dev.scouterctl.application.validateCollectorTimezone +import dev.scouterctl.core.* +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import java.time.Instant + +class TimezoneRulesTest : FunSpec({ + val summer = Instant.parse("2026-07-01T12:00:00Z").toEpochMilli() + for ((id, canonical, offset) in listOf(Triple("America/New_York", "America/New_York", -14400), + Triple("GMT+05:45", "GMT+05:45", 20700), Triple("PST", "America/Los_Angeles", -25200))) { + test("reported $id uses client zone rules at the Collector reference instant") { + describeCollectorTimezone(id, summer) shouldBe CollectorTimezone(canonical, offset, summer) + } + } + for (id in listOf("", "private-invalid-zone", "Central Standard Time")) { + test("unresolvable reported zone '$id' has no invented metadata") { + describeCollectorTimezone(id, summer) shouldBe null + } + } + test("matching request offsets do not depend on the Collector's current seasonal offset") { + val winter = Instant.parse("2026-01-01T12:00:00Z").toEpochMilli() + val collector = describeCollectorTimezone("America/New_York", winter) + collector?.utcOffsetSeconds shouldBe -18000 + validateCollectorTimezone("America/New_York", summer, summer + 1000, collector) + validateCollectorTimezone("Asia/Tokyo", summer, summer + 1000, describeCollectorTimezone("Asia/Seoul", winter)) + } + test("offset differences inside an interval are rejected even when both endpoints agree") { + val start = Instant.parse("2026-03-01T00:00:00Z").toEpochMilli() + val end = Instant.parse("2026-12-01T00:00:00Z").toEpochMilli() + val error = shouldThrow { + validateCollectorTimezone("America/Panama", start, end, describeCollectorTimezone("America/New_York", start)) + } + error.code shouldBe ErrorCode.INVALID_ARGUMENT + } + for (includeTransition in listOf(false, true)) { + test("DST offset check respects the exclusive end; transition included=$includeTransition") { + val start = Instant.parse("2026-03-08T06:00:00Z").toEpochMilli() + val end = Instant.parse("2026-03-08T07:00:00Z").toEpochMilli() + if (includeTransition) 1 else 0 + val collector = describeCollectorTimezone("America/New_York", start) + if (includeTransition) shouldThrow { + validateCollectorTimezone("America/Panama", start, end, collector) + }.code shouldBe ErrorCode.INVALID_ARGUMENT + else validateCollectorTimezone("America/Panama", start, end, collector) + } + } + test("missing Collector evidence does not invent an offset mismatch") { + validateCollectorTimezone("UTC", summer, summer + 1000, null) + } +}) diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index 65f46b9..7d2e08f 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -25,6 +25,7 @@ import java.util.concurrent.Executors import java.util.concurrent.TimeUnit enum class Fault { + TIMEZONE_SEOUL, TIMEZONE_MISSING, TIMEZONE_INVALID, TIMEZONE_TYPE, TIMEZONE_TIME_TYPE, COUNTER_EXPORT, HISTORY_EMPTY, HISTORY_MALFORMED, HISTORY_TIME_TYPE, HISTORY_OUTSIDE, HISTORY_NAN, HISTORY_TARGET_DUPLICATE, COUNTER_MALFORMED, COUNTER_DUPLICATE, COUNTER_NAN, COUNTER_XML_DTD, COUNTER_EMPTY, COUNTER_CUSTOM, @@ -129,6 +130,19 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas put("used", 1024L); put("total", 4096L); put("time", baseTime) if (fault == Fault.STATUS_MALFORMED) put("used", "private-status-value") }) + RequestCmd.SERVER_ENV -> send(output, MapPack().apply { + if (fault != Fault.TIMEZONE_MISSING) put("user.timezone", when (fault) { + Fault.TIMEZONE_SEOUL -> "Asia/Seoul" + Fault.TIMEZONE_INVALID -> "private-invalid-zone" + else -> "UTC" + }) + if (fault == Fault.TIMEZONE_TYPE) put("user.timezone", 42L) + put("private.password", "private-zone-env-token") + }) + RequestCmd.SERVER_TIME -> send(output, MapPack().apply { + put("time", baseTime) + if (fault == Fault.TIMEZONE_TIME_TYPE) put("time", "private-time-value") + }) RequestCmd.GET_XML_COUNTER -> send(output, MapPack().apply { val xml = if (fault == Fault.COUNTER_XML_DTD) """]>&secret;""" From df26cd353b786d4e1123d13d9bd6e3e53cb01dd3 Mon Sep 17 00:00:00 2001 From: Woojin Son <36991763+oxspan@users.noreply.github.com> Date: Fri, 13 Nov 2026 01:09:26 +0900 Subject: [PATCH 17/17] =?UTF-8?q?fix:=20=F0=9F=90=9B=20Diagnose=20Windows?= =?UTF-8?q?=20file=20ancestors=20before=20missing=20paths?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Windows reports a child of a regular file as `NoSuchFileException`. Capture and inspect therefore advised creating a missing directory even though a file blocked the path. Validate the lexical parent before reporting a missing path, so nested file ancestors receive the existing not-a-directory guidance while genuinely missing directories retain their existing message. The correction adds one line and preserves error codes, integrity checks, existing contents and genuine filesystem failures. No tests were changed, weakened or skipped. The existing three CapturePathTest cases fail before the change and pass afterward on native Windows, covering both capture formats and resume paths. Validation: - Complete hosted Linux and Windows suites and native command package verification pass for the exact head in [run 43251124659](https://github.com/oxspan/scouterctl/actions/runs/43251124659). Required PR CI also passes. - Line coverage 95.35% and branch coverage 81.08%, with unchanged thresholds. - Both ZIP checksums verify; Linux and Windows share byte-identical JVM payloads. The downloaded Windows executable passes version, redirected help, argument-error, invalid/missing-path and read-only grouped inspection checks; capture bytes remain unchanged. Release-readiness [run 43247008118](https://github.com/oxspan/scouterctl/actions/runs/43247008118) exposed the original Windows failure. No master promotion, release publication or version change is included. Refs #75 --- .../kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt | 1 + 1 file changed, 1 insertion(+) diff --git a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt index 2c31595..7e3aee9 100644 --- a/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt +++ b/src/main/kotlin/dev/scouterctl/adapters/files/CaptureDirectory.kt @@ -39,6 +39,7 @@ private fun requireDirectory(path: Path, missing: String, invalid: String) { val attributes = try { Files.readAttributes(path, BasicFileAttributes::class.java) } catch (_: NoSuchFileException) { + path.toAbsolutePath().parent?.let { requireDirectory(it, missing, invalid) } throw OperationFailure(ErrorCode.INVALID_ARGUMENT, missing) } catch (_: NotDirectoryException) { throw OperationFailure(ErrorCode.INVALID_ARGUMENT, invalid)