diff --git a/docs/README.md b/docs/README.md index f2e4997..56b9583 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,6 +23,7 @@ See [command and output examples](examples.md) for actual response shapes. - [Capture formats and older files](capture-compatibility.md) - [Interrupted capture recovery](capture-recovery.md) - [Profile completeness](profile-completeness.md) +- [Grouped profile SQL output](profile-sql-summary.md) - [Security](../SECURITY.md) ## Development and maintenance diff --git a/docs/cli-contract.md b/docs/cli-contract.md index 6f685e2..41eda1c 100644 --- a/docs/cli-contract.md +++ b/docs/cli-contract.md @@ -84,4 +84,4 @@ A consumer should accept the stream schema, process records, then require both t - Removing/renaming existing fields, changing their type, units or meaning, or changing established event semantics requires a new affected output schema version and documented migration. Existing error codes and exit meanings must not be silently repurposed. Enum/status additions that require new interpretation also need an explicit compatibility decision rather than being assumed safe. - New commands and additive fields can be introduced without changing existing envelopes. Changing existing command syntax or defaults in a way that breaks scripts requires release notes and a migration path. This document does not promise indefinite support for every old release. -Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes. +Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, [grouped SQL output](profile-sql-summary.md) for the optional profile summary schema, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes. diff --git a/docs/ko/README.md b/docs/ko/README.md index c7e28eb..3479bf2 100644 --- a/docs/ko/README.md +++ b/docs/ko/README.md @@ -12,6 +12,7 @@ - [설치·업데이트·이전 버전으로 전환·제거](installation.md) - [빠른 시작](../../QUICKSTART.ko.md) - [연결과 비밀번호](connections.md) +- [프로파일 SQL 집계](profile-sql-summary.md) - [기여 안내](../../CONTRIBUTING.ko.md) · [보안 안내](../../SECURITY.ko.md) ## 상세 사용법 diff --git a/docs/ko/profile-sql-summary.md b/docs/ko/profile-sql-summary.md new file mode 100644 index 0000000..abbdf94 --- /dev/null +++ b/docs/ko/profile-sql-summary.md @@ -0,0 +1,32 @@ +# 프로파일 SQL 집계 + +[English](../profile-sql-summary.md) | 한국어 + +SQL이 반복되어 프로파일 출력이 커지면 `--group-sql`로 SQL 해시별 집계를 조회할 수 있습니다. + +```sh +scouterctl profile get --date YYYYMMDD --txid TXID --group-sql +scouterctl inspect --directory CAPTURE --txid TXID --group-sql +``` + +이 옵션은 `profile get`과 `inspect`에서만 사용할 수 있습니다. SQL 문장을 보려면 `--include-text`도 지정하세요. 오프라인 조회에서는 텍스트를 포함해 저장한 캡처가 필요합니다. 옵션을 생략하면 개별 스텝을 반환합니다. 집계 조회는 캡처 파일이나 저장 형식을 바꾸지 않습니다. + +온라인 결과의 `data`, 오프라인 결과의 `data.record.profile`이 `scouterctl.profile-sql-summary/v1` 집계 객체가 됩니다. 바깥쪽 `scouterctl.result/v1` 응답과 오프라인 캡처·XLog·누락 정보는 유지합니다. 사용할 수 없는 오프라인 프로파일은 `null`입니다. + +| 필드 | 의미 | +| --- | --- | +| `txid` | 부호를 포함한 10진 문자열 트랜잭션 ID | +| `totalSteps`, `sqlSteps`, `nonSqlSteps` | 디코딩된 전체·SQL·나머지 스텝 수 | +| `groups[].hash`, `count` | SQL 해시와 해당 SQL 스텝 수 | +| `elapsedTotalMs`, `maxElapsedMs` | 그룹의 경과 시간 합계와 최댓값, 밀리초 | +| `errors` | 오류 해시가 0이 아닌 SQL 스텝 수 | +| `textIncluded`, `completeness` | 요청에 따른 텍스트 포함 여부와 원래 프로파일의 잘림 상태 | +| `basis` | `decoded-profile-steps`: 디코딩된 스텝을 기준으로 집계 | + +서로 다른 SQL 스텝 종류도 해시가 같으면 한 그룹에 포함합니다. 경과 시간 합계 내림차순, 같은 합계에서는 부호 있는 해시 오름차순으로 정렬합니다. 합계는 64비트 정수이며 원래 음수 값도 유지합니다. 실행 수는 영향받은 행 수가 아니며, 스텝 시간이 겹칠 수 있으므로 합계가 트랜잭션 전체 시간과 같다고 해석하면 안 됩니다. + +`--include-text`를 지정하면 `statement`에 사전의 `hash`, `text`, `status`를 반환합니다. 텍스트가 없거나 다른 해시의 사전 항목만 있으면 미해결 상태입니다. 같은 해시에 서로 다른 문장이 확인되면 `statementConflict: true`와 미해결 텍스트를 반환합니다. 텍스트 옵션을 생략하면 `statement: null`이며 충돌 정보도 공개하지 않습니다. 집계는 오류 텍스트·바인드 값·스텝 순서·SQL 외 스텝 상세를 포함하지 않습니다. 상세 분석에는 기본 스텝 출력을 사용하세요. + +출력 크기는 서로 다른 SQL 해시 수에 비례합니다. 반복 SQL에 효과가 크지만 Collector 전송량이나 디코딩 작업을 줄이지 않으며, 스텝·그룹 수에 새 제한을 두지 않습니다. 누락된 스텝을 복원하지도 않습니다. + +[기존 프로파일 완전성·종료 코드](../profile-completeness.md)를 유지합니다. 명시적인 잘림은 가능한 집계와 함께 `completeness: "truncated"`, `executionComplete: false`, 종료 코드 8을 반환합니다. `unknown`은 완전함을 보장하지 않습니다. 불완전한 캡처와 사용할 수 없는 프로파일도 기존의 종료 코드 8을 유지합니다. 오프라인 집계 역시 선택한 트랜잭션 뒤의 페이지까지 캡처 전체 무결성을 검사하며, 손상되면 결과 대신 오류를 반환합니다. [영문 가이드](../profile-sql-summary.md)에서 전체 JSON 예시를 볼 수 있습니다. diff --git a/docs/profile-completeness.md b/docs/profile-completeness.md index 7d1aea6..d4f2ba4 100644 --- a/docs/profile-completeness.md +++ b/docs/profile-completeness.md @@ -16,6 +16,8 @@ scouterctl profile get --date YYYYMMDD --txid TXID scouterctl inspect --directory CAPTURE --txid TXID ``` +Add [`--group-sql`](profile-sql-summary.md) to aggregate decoded SQL steps by hash. The summary preserves these completeness and exit semantics. + ## Capture compatibility and limits The completeness value is stored with profiles in newly written v3 captures and survives text redaction during inspection. Existing captures without the field read as `unknown`; their former control-step text/code was not stored, so it cannot reliably be reconstructed. Reading an old capture does not rewrite it. diff --git a/docs/profile-sql-summary.md b/docs/profile-sql-summary.md new file mode 100644 index 0000000..10bced9 --- /dev/null +++ b/docs/profile-sql-summary.md @@ -0,0 +1,44 @@ +# Group profile SQL steps + +English | [한국어](ko/profile-sql-summary.md) + +Use `--group-sql` when repeated SQL makes a profile too large to read: + +```sh +scouterctl profile get --date YYYYMMDD --txid TXID --group-sql +scouterctl inspect --directory CAPTURE --txid TXID --group-sql +``` + +The option is accepted only by `profile get` and `inspect`. Add `--include-text` to include resolved SQL statements; offline inspection also requires a capture that stored text. Without `--group-sql`, both commands return their usual individual steps. Capture files and their schemas are unchanged, and inspection never rewrites them. + +## Output + +Online `data`, or offline `data.record.profile`, contains a `scouterctl.profile-sql-summary/v1` object instead of individual profile steps. The outer result envelope remains `scouterctl.result/v1`. Offline capture, XLog and availability metadata keep their existing fields. An unavailable offline profile remains `null`. + +Example summary with text omitted: + +```json +{ + "txid": "9223372036854775804", + "completeness": "unknown", + "textIncluded": false, + "totalSteps": 3004, + "sqlSteps": 3000, + "nonSqlSteps": 4, + "groups": [ + {"hash":123,"statement":null,"statementConflict":false,"count":3000,"elapsedTotalMs":6000,"maxElapsedMs":2,"errors":30} + ], + "schemaVersion": "scouterctl.profile-sql-summary/v1", + "basis": "decoded-profile-steps" +} +``` + +Each group represents one signed SQL hash across all decoded SQL step variants. Groups are sorted by descending `elapsedTotalMs`, then ascending signed hash. `count` counts SQL steps, `elapsedTotalMs` sums their elapsed milliseconds using a 64-bit integer, `maxElapsedMs` is their largest elapsed value, and `errors` counts steps with a nonzero error hash. Original signed duration values are retained. These are decoded-step statistics, not affected-row counts or transaction wall time; overlapping steps can make their sum exceed transaction elapsed time. `txid` remains a decimal string. + +With `--include-text`, `statement` has the usual dictionary `hash`, `text` and `status`. Missing text is unresolved. If multiple distinct resolved statements share a hash, the group has `statementConflict: true` and unresolved text rather than choosing one. Dictionary entries for a different hash are ignored. Without `--include-text`, statements are `null` and text conflicts are not reported. Error text, bind values, individual step order and non-SQL step details are not part of the summary; request the default profile output when you need step details. + +## Limits and completion + +Output grows with the number of distinct SQL hashes. Repeated statements benefit most; this mode imposes no step or group cap and does not reduce Collector transfer or profile decoding. It cannot restore missing steps or establish that a trace is complete. + +The original [profile completeness](profile-completeness.md) and exit behavior apply: known truncation returns available groups with `completeness: "truncated"`, `executionComplete: false` and exit 8. `unknown` does not certify a complete trace. Offline inspection still validates the entire capture before returning a summary, including pages after the selected transaction. Incomplete captures and unavailable profiles retain their existing exit 8 behavior; corrupt capture data remains an error. diff --git a/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt b/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt new file mode 100644 index 0000000..1bc9288 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/adapters/serialization/ProfileSqlSummaryJson.kt @@ -0,0 +1,31 @@ +package dev.scouterctl.adapters.serialization + +import dev.scouterctl.core.* +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.* + +@Serializable +data class SqlAggregateDto( + val hash: Int, val statement: DictionaryTextDto?, val statementConflict: Boolean, + val count: Int, val elapsedTotalMs: Long, val maxElapsedMs: Int, val errors: Int, +) + +@Serializable +data class ProfileSqlSummaryDto( + val txid: String, val completeness: ProfileCompletenessDto, val textIncluded: Boolean, + val totalSteps: Int, val sqlSteps: Int, val nonSqlSteps: Int, val groups: List, + val schemaVersion: String = "scouterctl.profile-sql-summary/v1", val basis: String = "decoded-profile-steps", +) + +fun ProfileSqlSummary.toDto(): ProfileSqlSummaryDto = ProfileSqlSummaryDto(txid, + when (completeness) { ProfileCompleteness.UNKNOWN -> ProfileCompletenessDto.UNKNOWN; ProfileCompleteness.TRUNCATED -> ProfileCompletenessDto.TRUNCATED }, + textIncluded, totalSteps, sqlSteps, nonSqlSteps, groups.map { + SqlAggregateDto(it.hash, it.statement?.toDto(), it.statementConflict, it.count, it.elapsedTotalMs, it.maxElapsedMs, it.errors) + }) + +fun Inspection.toSqlSummaryJson(summary: ProfileSqlSummary?): JsonObject { + val base = json.encodeToJsonElement(copy(record = record.copy(profile = null)).toDto()).jsonObject + val record = base.getValue("record").jsonObject + val grouped = summary?.let { json.encodeToJsonElement(it.toDto()) } ?: JsonNull + return JsonObject(base + ("record" to JsonObject(record + ("profile" to grouped)))) +} diff --git a/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt b/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt new file mode 100644 index 0000000..47cd784 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt @@ -0,0 +1,15 @@ +package dev.scouterctl.application + +import dev.scouterctl.core.* + +fun summarizeSql(profile: ProfileRecord, includeText: Boolean): ProfileSqlSummary { + val sql = profile.steps.mapNotNull { it.sql } + val groups = sql.groupBy { it.hash }.map { (hash, steps) -> + val texts = if (includeText) steps.mapNotNull { it.statement?.takeIf { text -> text.hash == hash }?.text }.distinct() + else emptyList() + SqlAggregate(hash, if (includeText) DictionaryText(hash, texts.singleOrNull()) else null, texts.size > 1, + steps.size, steps.sumOf { it.elapsedMs.toLong() }, steps.maxOf { it.elapsedMs }, steps.count { it.errorHash != 0 }) + }.sortedWith(compareByDescending { it.elapsedTotalMs }.thenBy { it.hash }) + return ProfileSqlSummary(profile.txid, profile.completeness, includeText, profile.steps.size, sql.size, + profile.steps.size - sql.size, groups) +} diff --git a/src/main/kotlin/dev/scouterctl/cli/Cli.kt b/src/main/kotlin/dev/scouterctl/cli/Cli.kt index 4c8d1e7..4fde5a4 100644 --- a/src/main/kotlin/dev/scouterctl/cli/Cli.kt +++ b/src/main/kotlin/dev/scouterctl/cli/Cli.kt @@ -42,6 +42,8 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut ]) var includeProfiles: Boolean = false @Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in xlog search, profile get, profile capture or inspect"]) var includeText: Boolean = false + @Option(names = ["--group-sql"], description = ["profile get / inspect: aggregate decoded SQL steps by hash instead of returning steps"]) + var groupSql: Boolean = false @Option(names = ["--limit"], description = ["inspect search/summary/compare: 1..1000 requests/services; default 20"]) var limit: Int? = null @Option(names = ["--sort"], description = ["inspect summary: errors, elapsed (maximum), requests; default errors"]) var summarySort: String? = null @Option(names = ["--min-elapsed-ms"], description = ["inspect search: inclusive minimum request duration"]) var minElapsedMs: Int? = null diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt index 044b1f2..a756791 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt @@ -51,15 +51,15 @@ internal val commandGuides = listOf( connectionNotes + "Use the same query and directory with --resume; chunk boundaries split at Collector midnight."), CommandGuide("xlog search", "Stream XLogs for a storage date and time window.", xlogOptions, connectionOptions + pagingOptions + "--include-text", "xlog search $windowExample --user reader --no-input", connectionNotes), - CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + "--include-text", - "profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes), + CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + setOf("--include-text", "--group-sql"), + "profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes + "Use --group-sql for SQL hash aggregates; --include-text is still required for statements."), CommandGuide("capture", "Save a resumable XLog capture, optionally including profiles.", xlogOptions + "--directory", connectionOptions + pagingOptions + setOf("--resume", "--include-profiles", "--include-text"), "capture $windowExample --directory capture --user reader --no-input", connectionNotes + listOf("--include-text requires --include-profiles. Text can contain sensitive values.", "Use the same query and directory with --resume. Profile capture defaults to page size 25, maximum 100.")), - CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions, - "inspect --directory capture --txid 123"), + CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions + "--group-sql", + "inspect --directory capture --txid 123", listOf("Use --group-sql for SQL hash aggregates; --include-text requires stored text.")), CommandGuide("inspect search", "Search captured requests offline.", setOf("--directory"), offlineOptions + setOf("--limit", "--min-elapsed-ms", "--errors-only", "--service-hash", "--service-name"), "inspect search --directory capture --errors-only", listOf("--service-name requires --include-text and stored dictionary text.")), diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt index 654e2c5..aa484fa 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt @@ -8,6 +8,9 @@ import java.time.format.DateTimeFormatter internal fun Cli.interpret(): CommandRequest { val command = words.joinToString(" ") + if (spec.commandLine().parseResult.hasMatchedOption("--group-sql") && command !in setOf("profile get", "inspect")) { + invalid("--group-sql requires profile get or inspect") + } if (command in setOf("context add", "context list", "context use")) return contextRequest(command) if (name != null) invalid("--name requires context add/use") if (command == "inspect compare") { @@ -65,7 +68,7 @@ internal fun Cli.interpret(): CommandRequest { } "inspect" -> { if (resume) invalid("--resume requires capture") - OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText) + OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText, groupSql) } else -> onlineRequest(command) } @@ -110,7 +113,7 @@ private fun Cli.onlineRequest(command: String): OnlineRequest { return when (command) { "doctor", "auth login" -> OnlineRequest.Doctor(command == "auth login") "object list" -> OnlineRequest.Objects(date?.also(::validateStorageDate)) - "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText) + "profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText, groupSql) "xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages, includeText) "capture" -> OnlineRequest.Capture( required(query, "query"), required(directory, "--directory"), resume, maxPages, includeProfiles, includeText, diff --git a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt index 29519c9..bc82b40 100644 --- a/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt +++ b/src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt @@ -11,7 +11,7 @@ sealed interface ContextRequest : CommandRequest { data class Use(val name: String?) : ContextRequest } sealed interface OfflineRequest : CommandRequest { - data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean) : OfflineRequest + data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OfflineRequest data class Search(val directory: Path, val filter: EvidenceFilter, val includeText: Boolean) : OfflineRequest data class Summary(val directory: Path, val options: ServiceSummaryOptions) : OfflineRequest data class Compare(val before: Path, val after: Path, val options: ComparisonOptions) : OfflineRequest @@ -26,7 +26,7 @@ sealed interface OnlineRequest : CommandRequest { data class CounterHistory(val query: CounterHistoryQuery) : OnlineRequest data class Counters(val objHash: Int, val name: String?) : OnlineRequest data class Objects(val date: String?) : OnlineRequest - data class Profile(val date: String, val txid: Long, val includeText: Boolean) : OnlineRequest + data class Profile(val date: String, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OnlineRequest data class Search(val query: Query, val maxPages: Int, val includeText: Boolean = false) : OnlineRequest data class Capture(val query: Query, val directory: Path, val resume: Boolean, val maxPages: Int, val includeProfiles: Boolean, val includeText: Boolean) : OnlineRequest diff --git a/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt index 963d318..72c50cf 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt @@ -16,7 +16,9 @@ class OfflineCommands(private val repository: (Path) -> EvidenceRepository, priv } private fun inspect(request: OfflineRequest.Inspect): Int { val result = inspectCapture(repository(request.directory), request.txid, request.includeText) - return output.result(json.encodeToJsonElement(result.toDto()), result.captureExecutionComplete && result.unavailableProfiles == 0 && + val data = if (request.groupSql) result.toSqlSummaryJson(result.record.profile?.let { summarizeSql(it, request.includeText) }) + else json.encodeToJsonElement(result.toDto()) + return output.result(data, result.captureExecutionComplete && result.unavailableProfiles == 0 && result.record.profile?.completeness != ProfileCompleteness.TRUNCATED) } private fun search(request: OfflineRequest.Search): Int { diff --git a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt index 053ae70..670f292 100644 --- a/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt +++ b/src/main/kotlin/dev/scouterctl/cli/OnlineCommands.kt @@ -49,7 +49,8 @@ class OnlineCommands( } private fun profile(source: ObservationSource, request: OnlineRequest.Profile): Int { val profile = source.profile(request.date, request.txid, request.includeText, includeStepDetails = true) - return output.result(profile.toLiveJson(), profile.completeness != ProfileCompleteness.TRUNCATED) + val data = if (request.groupSql) json.encodeToJsonElement(summarizeSql(profile, request.includeText).toDto()) else profile.toLiveJson() + return output.result(data, profile.completeness != ProfileCompleteness.TRUNCATED) } private fun capture(source: ObservationSource, endpoint: Endpoint, request: OnlineRequest.Capture): Int { diff --git a/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt b/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt new file mode 100644 index 0000000..93e4719 --- /dev/null +++ b/src/main/kotlin/dev/scouterctl/core/ProfileSqlSummary.kt @@ -0,0 +1,11 @@ +package dev.scouterctl.core + +data class SqlAggregate( + val hash: Int, val statement: DictionaryText?, val statementConflict: Boolean, + val count: Int, val elapsedTotalMs: Long, val maxElapsedMs: Int, val errors: Int, +) + +data class ProfileSqlSummary( + val txid: String, val completeness: ProfileCompleteness, val textIncluded: Boolean, + val totalSteps: Int, val sqlSteps: Int, val nonSqlSteps: Int, val groups: List, +) diff --git a/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt b/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt new file mode 100644 index 0000000..181285d --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/ProfileSqlGroupingTest.kt @@ -0,0 +1,109 @@ +package dev.scouterctl + +import dev.scouterctl.adapters.files.EvidenceStore +import io.kotest.core.spec.style.FunSpec +import io.kotest.engine.spec.tempdir +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldNotContain +import kotlinx.serialization.json.* +import java.nio.file.Files + +class ProfileSqlGroupingTest : FunSpec({ + for (text in listOf(false, true)) { + test("profile get groups 3000 repeated SQL steps with text policy $text") { + WireFixture(Fault.PROFILE_REPEATED_SQL).use { server -> + val args = listOf("profile", "get", "--date", "20261101", "--txid", WireFixture.ids.first().toString(), "--group-sql") + + if (text) listOf("--include-text") else emptyList() + val result = ContractCli(tempdir().toPath()).run(server, args) + result.exit shouldBe 0 + result.stderr shouldBe "" + val summary = result.data() + summary["schemaVersion"] shouldBe JsonPrimitive("scouterctl.profile-sql-summary/v1") + summary["txid"] shouldBe JsonPrimitive(WireFixture.ids.first().toString()) + summary["totalSteps"] shouldBe JsonPrimitive(3004) + summary["sqlSteps"] shouldBe JsonPrimitive(3000) + summary["nonSqlSteps"] shouldBe JsonPrimitive(4) + summary["completeness"] shouldBe JsonPrimitive("unknown") + summary.containsKey("steps") shouldBe false + val group = summary["groups"]!!.jsonArray.single().jsonObject + group["hash"] shouldBe JsonPrimitive(123) + group["count"] shouldBe JsonPrimitive(3000) + group["elapsedTotalMs"] shouldBe JsonPrimitive(6000) + group["maxElapsedMs"] shouldBe JsonPrimitive(2) + group["errors"] shouldBe JsonPrimitive(30) + if (text) group["statement"]!!.jsonObject["text"] shouldBe JsonPrimitive("select marker from fixture where id = ?") + else group["statement"] shouldBe JsonNull + result.output shouldNotContain "secret-token" + result.output shouldNotContain "fixture-password" + } + } + } + + test("grouped offline inspection redacts by request and never rewrites stored steps") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + WireFixture().use { server -> + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--include-text", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 0 + } + val before = Files.list(directory).use { files -> files.toList().associate { it.fileName.toString() to Files.readString(it) } } + val args = listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids.first().toString(), "--group-sql") + for (text in listOf(false, true)) { + val result = runOfflineCli(work, args + if (text) listOf("--include-text") else emptyList()) + result.exit shouldBe 0 + val inspection = result.data() + inspection["textIncluded"] shouldBe JsonPrimitive(text) + val summary = inspection["record"]!!.jsonObject["profile"]!!.jsonObject + summary["schemaVersion"] shouldBe JsonPrimitive("scouterctl.profile-sql-summary/v1") + summary["groups"]!!.jsonArray.map { it.jsonObject["hash"]!!.jsonPrimitive.int } shouldBe listOf(456, 123) + if (!text) result.output shouldNotContain "select marker" + else summary["groups"]!!.jsonArray.last().jsonObject["statement"]!!.jsonObject["text"] shouldBe + JsonPrimitive("select marker from fixture where id = ?") + } + Files.list(directory).use { files -> files.toList().associate { it.fileName.toString() to Files.readString(it) } } shouldBe before + val raw = runOfflineCli(work, args.dropLast(1)) + raw.data()["record"]!!.jsonObject["profile"]!!.jsonObject["steps"]!!.jsonArray.size shouldBe 3 + + val manifest = EvidenceStore(directory).load() + Files.writeString(directory.resolve(manifest.pages.last().name), "damaged later page") + runOfflineCli(work, args).exit shouldBe 10 + } + + for (offline in listOf(false, true)) { + test("grouped ${if (offline) "inspect" else "profile"} preserves explicit profile truncation") { + val work = tempdir().toPath() + WireFixture(Fault.PROFILE_TRUNCATED).use { server -> + val result = if (offline) { + val directory = work.resolve("capture") + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 0 + runOfflineCli(work, listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids.first().toString(), "--group-sql")) + } else ContractCli(work).run(server, listOf("profile", "get", "--date", "20261101", "--txid", WireFixture.ids.first().toString(), "--group-sql")) + result.exit shouldBe 8 + result.events.single()["executionComplete"] shouldBe JsonPrimitive(false) + val profile = if (offline) result.data()["record"]!!.jsonObject["profile"]!!.jsonObject else result.data() + profile["completeness"] shouldBe JsonPrimitive("truncated") + profile["groups"]!!.jsonArray.single().jsonObject["count"] shouldBe JsonPrimitive(1) + } + } + } + + test("grouped inspect retains unavailable profiles and refuses text that was not stored") { + val work = tempdir().toPath() + val directory = work.resolve("capture") + WireFixture(Fault.PROFILE_MISSING).use { server -> + ContractCli(work).run(server, listOf("capture", "--include-profiles", "--directory", directory.toString()) + evidenceWindow).exit shouldBe 8 + } + val args = listOf("inspect", "--directory", directory.toString(), "--txid", WireFixture.ids[1].toString(), "--group-sql") + val result = runOfflineCli(work, args) + result.exit shouldBe 8 + result.data()["record"]!!.jsonObject["profile"] shouldBe JsonNull + runOfflineCli(work, args + "--include-text").exit shouldBe 2 + } + + for (command in listOf(listOf("doctor"), listOf("capture"), listOf("inspect", "search"), listOf("context", "list"))) { + test("${command.joinToString(" ")} rejects the SQL grouping option before execution") { + val result = runOfflineCli(tempdir().toPath(), command + "--group-sql") + result.exit shouldBe 2 + result.events.single()["error"]!!.jsonObject["message"]!!.jsonPrimitive.content.contains("--group-sql") shouldBe true + } + } +}) diff --git a/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt b/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt new file mode 100644 index 0000000..f8bd3a7 --- /dev/null +++ b/src/test/kotlin/dev/scouterctl/SqlSummaryTest.kt @@ -0,0 +1,76 @@ +package dev.scouterctl + +import dev.scouterctl.application.summarizeSql +import dev.scouterctl.core.* +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe + +class SqlSummaryTest : FunSpec({ + test("SQL aggregates preserve signed hashes, wide duration totals, errors and profile scope") { + val profile = ProfileRecord("9223372036854775806", listOf( + ProfileStep("SQL3", 0, SqlProfile(10, Int.MAX_VALUE, 0)), + ProfileStep("SQL", 1, SqlProfile(-7, 20, -99)), + ProfileStep("SQL3", 2, SqlProfile(10, Int.MAX_VALUE, 7)), + ProfileStep("SQL", 3, SqlProfile(-7, -3, 0)), + ProfileStep("SOCKET", 4), + ), completeness = ProfileCompleteness.TRUNCATED) + + val summary = summarizeSql(profile, false) + + summary.txid shouldBe profile.txid + summary.completeness shouldBe ProfileCompleteness.TRUNCATED + summary.totalSteps shouldBe 5 + summary.sqlSteps shouldBe 4 + summary.nonSqlSteps shouldBe 1 + summary.groups shouldBe listOf( + SqlAggregate(10, null, false, 2, 4_294_967_294L, Int.MAX_VALUE, 1), + SqlAggregate(-7, null, false, 2, 17L, 20, 1), + ) + } + + for (steps in listOf(emptyList(), listOf(ProfileStep("MESSAGE", 0), ProfileStep("SOCKET", 1)))) { + test("a profile with ${steps.size} non-SQL steps has no invented SQL groups") { + val summary = summarizeSql(ProfileRecord("42", steps), true) + summary.txid shouldBe "42" + summary.totalSteps shouldBe steps.size + summary.sqlSteps shouldBe 0 + summary.nonSqlSteps shouldBe steps.size + summary.groups shouldBe emptyList() + } + } + + test("statement resolution distinguishes consistent, conflicting, missing and mismatched dictionaries") { + val sql = listOf( + SqlProfile(7, 1, 0, statement = DictionaryText(7, "select a")), + SqlProfile(7, 1, 0, statement = DictionaryText(7, "select b")), + SqlProfile(8, 1, 0, statement = DictionaryText(999, "wrong statement")), + SqlProfile(9, 1, 0, statement = DictionaryText(9, "select c")), + SqlProfile(9, 1, 0, statement = DictionaryText(9, "select c")), + SqlProfile(10, 1, 0), + SqlProfile(11, 1, 0, statement = DictionaryText(11)), + ) + val summary = summarizeSql(ProfileRecord("42", sql.mapIndexed { index, value -> ProfileStep("SQL3", index, value) }), true) + val groups = summary.groups.associateBy { it.hash } + groups.getValue(7).statementConflict shouldBe true + groups.getValue(7).statement shouldBe DictionaryText(7) + groups.getValue(8).statement shouldBe DictionaryText(8) + groups.getValue(9).statementConflict shouldBe false + groups.getValue(9).statement shouldBe DictionaryText(9, "select c") + groups.getValue(10).statement shouldBe DictionaryText(10) + groups.getValue(11).statement shouldBe DictionaryText(11) + } + + test("redacted grouping cannot reveal supplied SQL text or text conflicts") { + val profile = ProfileRecord("42", listOf( + ProfileStep("SQL3", 0, SqlProfile(7, 1, -9, statement = DictionaryText(7, "private statement a"), error = DictionaryText(-9, "private error"))), + ProfileStep("SQL3", 1, SqlProfile(7, 1, 0, statement = DictionaryText(7, "private statement b"))), + )) + summarizeSql(profile, false).groups.single() shouldBe SqlAggregate(7, null, false, 2, 2, 1, 1) + } + + test("equal duration groups use signed hash order for stable output") { + val profile = ProfileRecord("42", listOf(ProfileStep("SQL3", 0, SqlProfile(9, 3, 0)), + ProfileStep("SQL3", 1, SqlProfile(-7, 3, 0)))) + summarizeSql(profile, false).groups.map { it.hash } shouldBe listOf(-7, 9) + } +}) diff --git a/src/test/kotlin/dev/scouterctl/WireFixture.kt b/src/test/kotlin/dev/scouterctl/WireFixture.kt index 82f0743..65f46b9 100644 --- a/src/test/kotlin/dev/scouterctl/WireFixture.kt +++ b/src/test/kotlin/dev/scouterctl/WireFixture.kt @@ -32,7 +32,7 @@ enum class Fault { NONE, STATUS_MALFORMED, CONFIG_MALFORMED, CONFIG_DUPLICATE, CONFIG_INVALID_VISIBLE, CONFIG_INVALID_AGENT_TEXT, VERSION, AUTH, SESSION, MISSING_META, INVALID_META, STALLED_CURSOR, DISCONNECT, EMPTY_OBJECTS, STRUCTURAL_ONLY, MISSING_TEXT, EMPTY_TEXT, MANY_SQL, TEXT_WRONG_TYPE, TEXT_WRONG_HASH, TEXT_DUPLICATE, TEXT_PACK, TEXT_DISCONNECT, TEXT_SESSION, XLOG_MISSING, XLOG_WRONG_ID, - PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, + PROFILE_NETWORK, PROFILE_TRUNCATED, PROFILE_OTHER_CONTROL, PROFILE_MANY_STEPS, PROFILE_REPEATED_SQL, PROFILE_MISSING, PROFILE_MISSING_ONCE, PROFILE_DISCONNECT, PROFILE_MALFORMED, TEXT_TIMEOUT, } class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPassword: String? = null, @@ -289,7 +289,13 @@ class WireFixture(private val fault: Fault = Fault.NONE, private val expectedPas }, ApiCallStep2().apply { index = 4; parent = 1; start_time = 650; elapsed = 50 }, StepControl().apply { code = 7; message = "private-control-message" }, - )) else if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( + )) else if (fault == Fault.PROFILE_REPEATED_SQL) Step.toBytes(Array(3004) { number -> + if (number < 4) HashedMessageStep().apply { index = number; hash = 77 } + else SqlStep3().apply { + index = number; hash = 123; elapsed = 2; error = if (number % 100 == 0) -789 else 0 + param = "password=secret-token" + } + }) else if (fault == Fault.PROFILE_TRUNCATED || fault == Fault.PROFILE_OTHER_CONTROL) Step.toBytes(arrayOf( SqlStep3().apply { index = 1; hash = 123; elapsed = 7 }, StepControl().apply { code = if (fault == Fault.PROFILE_TRUNCATED) 0 else 7; message = " ** Profile Truncated ** " } )) else if (fault == Fault.PROFILE_MANY_STEPS) Step.toBytes(Array(1001) { number ->