Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ See [command and output examples](examples.md) for actual response shapes.
- [Capture formats and older files](capture-compatibility.md)
- [Interrupted capture recovery](capture-recovery.md)
- [Profile completeness](profile-completeness.md)
- [Grouped profile SQL output](profile-sql-summary.md)
- [Security](../SECURITY.md)

## Development and maintenance
Expand Down
2 changes: 1 addition & 1 deletion docs/cli-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,4 +84,4 @@ A consumer should accept the stream schema, process records, then require both t
- Removing/renaming existing fields, changing their type, units or meaning, or changing established event semantics requires a new affected output schema version and documented migration. Existing error codes and exit meanings must not be silently repurposed. Enum/status additions that require new interpretation also need an explicit compatibility decision rather than being assumed safe.
- New commands and additive fields can be introduced without changing existing envelopes. Changing existing command syntax or defaults in a way that breaks scripts requires release notes and a migration path. This document does not promise indefinite support for every old release.

Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes.
Use the [command guides](README.md#using-scouterctl) for command-specific fields and behavior, [profile completeness](profile-completeness.md) for truncation signals, [grouped SQL output](profile-sql-summary.md) for the optional profile summary schema, and [architecture](architecture.md) for presentation/serialization ownership. Core and application models do not own JSON envelopes or numeric process exit codes.
1 change: 1 addition & 0 deletions docs/ko/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- [설치·업데이트·이전 버전으로 전환·제거](installation.md)
- [빠른 시작](../../QUICKSTART.ko.md)
- [연결과 비밀번호](connections.md)
- [프로파일 SQL 집계](profile-sql-summary.md)
- [기여 안내](../../CONTRIBUTING.ko.md) · [보안 안내](../../SECURITY.ko.md)

## 상세 사용법
Expand Down
32 changes: 32 additions & 0 deletions docs/ko/profile-sql-summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# 프로파일 SQL 집계

[English](../profile-sql-summary.md) | 한국어

SQL이 반복되어 프로파일 출력이 커지면 `--group-sql`로 SQL 해시별 집계를 조회할 수 있습니다.

```sh
scouterctl profile get --date YYYYMMDD --txid TXID --group-sql
scouterctl inspect --directory CAPTURE --txid TXID --group-sql
```

이 옵션은 `profile get`과 `inspect`에서만 사용할 수 있습니다. SQL 문장을 보려면 `--include-text`도 지정하세요. 오프라인 조회에서는 텍스트를 포함해 저장한 캡처가 필요합니다. 옵션을 생략하면 개별 스텝을 반환합니다. 집계 조회는 캡처 파일이나 저장 형식을 바꾸지 않습니다.

온라인 결과의 `data`, 오프라인 결과의 `data.record.profile`이 `scouterctl.profile-sql-summary/v1` 집계 객체가 됩니다. 바깥쪽 `scouterctl.result/v1` 응답과 오프라인 캡처·XLog·누락 정보는 유지합니다. 사용할 수 없는 오프라인 프로파일은 `null`입니다.

| 필드 | 의미 |
| --- | --- |
| `txid` | 부호를 포함한 10진 문자열 트랜잭션 ID |
| `totalSteps`, `sqlSteps`, `nonSqlSteps` | 디코딩된 전체·SQL·나머지 스텝 수 |
| `groups[].hash`, `count` | SQL 해시와 해당 SQL 스텝 수 |
| `elapsedTotalMs`, `maxElapsedMs` | 그룹의 경과 시간 합계와 최댓값, 밀리초 |
| `errors` | 오류 해시가 0이 아닌 SQL 스텝 수 |
| `textIncluded`, `completeness` | 요청에 따른 텍스트 포함 여부와 원래 프로파일의 잘림 상태 |
| `basis` | `decoded-profile-steps`: 디코딩된 스텝을 기준으로 집계 |

서로 다른 SQL 스텝 종류도 해시가 같으면 한 그룹에 포함합니다. 경과 시간 합계 내림차순, 같은 합계에서는 부호 있는 해시 오름차순으로 정렬합니다. 합계는 64비트 정수이며 원래 음수 값도 유지합니다. 실행 수는 영향받은 행 수가 아니며, 스텝 시간이 겹칠 수 있으므로 합계가 트랜잭션 전체 시간과 같다고 해석하면 안 됩니다.

`--include-text`를 지정하면 `statement`에 사전의 `hash`, `text`, `status`를 반환합니다. 텍스트가 없거나 다른 해시의 사전 항목만 있으면 미해결 상태입니다. 같은 해시에 서로 다른 문장이 확인되면 `statementConflict: true`와 미해결 텍스트를 반환합니다. 텍스트 옵션을 생략하면 `statement: null`이며 충돌 정보도 공개하지 않습니다. 집계는 오류 텍스트·바인드 값·스텝 순서·SQL 외 스텝 상세를 포함하지 않습니다. 상세 분석에는 기본 스텝 출력을 사용하세요.

출력 크기는 서로 다른 SQL 해시 수에 비례합니다. 반복 SQL에 효과가 크지만 Collector 전송량이나 디코딩 작업을 줄이지 않으며, 스텝·그룹 수에 새 제한을 두지 않습니다. 누락된 스텝을 복원하지도 않습니다.

[기존 프로파일 완전성·종료 코드](../profile-completeness.md)를 유지합니다. 명시적인 잘림은 가능한 집계와 함께 `completeness: "truncated"`, `executionComplete: false`, 종료 코드 8을 반환합니다. `unknown`은 완전함을 보장하지 않습니다. 불완전한 캡처와 사용할 수 없는 프로파일도 기존의 종료 코드 8을 유지합니다. 오프라인 집계 역시 선택한 트랜잭션 뒤의 페이지까지 캡처 전체 무결성을 검사하며, 손상되면 결과 대신 오류를 반환합니다. [영문 가이드](../profile-sql-summary.md)에서 전체 JSON 예시를 볼 수 있습니다.
2 changes: 2 additions & 0 deletions docs/profile-completeness.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ scouterctl profile get --date YYYYMMDD --txid TXID
scouterctl inspect --directory CAPTURE --txid TXID
```

Add [`--group-sql`](profile-sql-summary.md) to aggregate decoded SQL steps by hash. The summary preserves these completeness and exit semantics.

## Capture compatibility and limits

The completeness value is stored with profiles in newly written v3 captures and survives text redaction during inspection. Existing captures without the field read as `unknown`; their former control-step text/code was not stored, so it cannot reliably be reconstructed. Reading an old capture does not rewrite it.
Expand Down
44 changes: 44 additions & 0 deletions docs/profile-sql-summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Group profile SQL steps

English | [한국어](ko/profile-sql-summary.md)

Use `--group-sql` when repeated SQL makes a profile too large to read:

```sh
scouterctl profile get --date YYYYMMDD --txid TXID --group-sql
scouterctl inspect --directory CAPTURE --txid TXID --group-sql
```

The option is accepted only by `profile get` and `inspect`. Add `--include-text` to include resolved SQL statements; offline inspection also requires a capture that stored text. Without `--group-sql`, both commands return their usual individual steps. Capture files and their schemas are unchanged, and inspection never rewrites them.

## Output

Online `data`, or offline `data.record.profile`, contains a `scouterctl.profile-sql-summary/v1` object instead of individual profile steps. The outer result envelope remains `scouterctl.result/v1`. Offline capture, XLog and availability metadata keep their existing fields. An unavailable offline profile remains `null`.

Example summary with text omitted:

```json
{
"txid": "9223372036854775804",
"completeness": "unknown",
"textIncluded": false,
"totalSteps": 3004,
"sqlSteps": 3000,
"nonSqlSteps": 4,
"groups": [
{"hash":123,"statement":null,"statementConflict":false,"count":3000,"elapsedTotalMs":6000,"maxElapsedMs":2,"errors":30}
],
"schemaVersion": "scouterctl.profile-sql-summary/v1",
"basis": "decoded-profile-steps"
}
```

Each group represents one signed SQL hash across all decoded SQL step variants. Groups are sorted by descending `elapsedTotalMs`, then ascending signed hash. `count` counts SQL steps, `elapsedTotalMs` sums their elapsed milliseconds using a 64-bit integer, `maxElapsedMs` is their largest elapsed value, and `errors` counts steps with a nonzero error hash. Original signed duration values are retained. These are decoded-step statistics, not affected-row counts or transaction wall time; overlapping steps can make their sum exceed transaction elapsed time. `txid` remains a decimal string.

With `--include-text`, `statement` has the usual dictionary `hash`, `text` and `status`. Missing text is unresolved. If multiple distinct resolved statements share a hash, the group has `statementConflict: true` and unresolved text rather than choosing one. Dictionary entries for a different hash are ignored. Without `--include-text`, statements are `null` and text conflicts are not reported. Error text, bind values, individual step order and non-SQL step details are not part of the summary; request the default profile output when you need step details.

## Limits and completion

Output grows with the number of distinct SQL hashes. Repeated statements benefit most; this mode imposes no step or group cap and does not reduce Collector transfer or profile decoding. It cannot restore missing steps or establish that a trace is complete.

The original [profile completeness](profile-completeness.md) and exit behavior apply: known truncation returns available groups with `completeness: "truncated"`, `executionComplete: false` and exit 8. `unknown` does not certify a complete trace. Offline inspection still validates the entire capture before returning a summary, including pages after the selected transaction. Incomplete captures and unavailable profiles retain their existing exit 8 behavior; corrupt capture data remains an error.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package dev.scouterctl.adapters.serialization

import dev.scouterctl.core.*
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.*

@Serializable
data class SqlAggregateDto(
val hash: Int, val statement: DictionaryTextDto?, val statementConflict: Boolean,
val count: Int, val elapsedTotalMs: Long, val maxElapsedMs: Int, val errors: Int,
)

@Serializable
data class ProfileSqlSummaryDto(
val txid: String, val completeness: ProfileCompletenessDto, val textIncluded: Boolean,
val totalSteps: Int, val sqlSteps: Int, val nonSqlSteps: Int, val groups: List<SqlAggregateDto>,
val schemaVersion: String = "scouterctl.profile-sql-summary/v1", val basis: String = "decoded-profile-steps",
)

fun ProfileSqlSummary.toDto(): ProfileSqlSummaryDto = ProfileSqlSummaryDto(txid,
when (completeness) { ProfileCompleteness.UNKNOWN -> ProfileCompletenessDto.UNKNOWN; ProfileCompleteness.TRUNCATED -> ProfileCompletenessDto.TRUNCATED },
textIncluded, totalSteps, sqlSteps, nonSqlSteps, groups.map {
SqlAggregateDto(it.hash, it.statement?.toDto(), it.statementConflict, it.count, it.elapsedTotalMs, it.maxElapsedMs, it.errors)
})

fun Inspection.toSqlSummaryJson(summary: ProfileSqlSummary?): JsonObject {
val base = json.encodeToJsonElement(copy(record = record.copy(profile = null)).toDto()).jsonObject
val record = base.getValue("record").jsonObject
val grouped = summary?.let { json.encodeToJsonElement(it.toDto()) } ?: JsonNull
return JsonObject(base + ("record" to JsonObject(record + ("profile" to grouped))))
}
15 changes: 15 additions & 0 deletions src/main/kotlin/dev/scouterctl/application/SummarizeSql.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package dev.scouterctl.application

import dev.scouterctl.core.*

fun summarizeSql(profile: ProfileRecord, includeText: Boolean): ProfileSqlSummary {
val sql = profile.steps.mapNotNull { it.sql }
val groups = sql.groupBy { it.hash }.map { (hash, steps) ->
val texts = if (includeText) steps.mapNotNull { it.statement?.takeIf { text -> text.hash == hash }?.text }.distinct()
else emptyList()
SqlAggregate(hash, if (includeText) DictionaryText(hash, texts.singleOrNull()) else null, texts.size > 1,
steps.size, steps.sumOf { it.elapsedMs.toLong() }, steps.maxOf { it.elapsedMs }, steps.count { it.errorHash != 0 })
}.sortedWith(compareByDescending<SqlAggregate> { it.elapsedTotalMs }.thenBy { it.hash })
return ProfileSqlSummary(profile.txid, profile.completeness, includeText, profile.steps.size, sql.size,
profile.steps.size - sql.size, groups)
}
2 changes: 2 additions & 0 deletions src/main/kotlin/dev/scouterctl/cli/Cli.kt
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ class Cli(private val dispatcher: CommandDispatcher, private val output: JsonOut
]) var includeProfiles: Boolean = false
@Option(names = ["--include-text"], description = ["Include potentially sensitive dictionary text in xlog search, profile get, profile capture or inspect"])
var includeText: Boolean = false
@Option(names = ["--group-sql"], description = ["profile get / inspect: aggregate decoded SQL steps by hash instead of returning steps"])
var groupSql: Boolean = false
@Option(names = ["--limit"], description = ["inspect search/summary/compare: 1..1000 requests/services; default 20"]) var limit: Int? = null
@Option(names = ["--sort"], description = ["inspect summary: errors, elapsed (maximum), requests; default errors"]) var summarySort: String? = null
@Option(names = ["--min-elapsed-ms"], description = ["inspect search: inclusive minimum request duration"]) var minElapsedMs: Int? = null
Expand Down
8 changes: 4 additions & 4 deletions src/main/kotlin/dev/scouterctl/cli/CommandGuide.kt
Original file line number Diff line number Diff line change
Expand Up @@ -51,15 +51,15 @@ internal val commandGuides = listOf(
connectionNotes + "Use the same query and directory with --resume; chunk boundaries split at Collector midnight."),
CommandGuide("xlog search", "Stream XLogs for a storage date and time window.", xlogOptions, connectionOptions + pagingOptions + "--include-text",
"xlog search $windowExample --user reader --no-input", connectionNotes),
CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + "--include-text",
"profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes),
CommandGuide("profile get", "Read one stored transaction profile.", setOf("--date", "--txid"), connectionOptions + setOf("--include-text", "--group-sql"),
"profile get --date 20261101 --txid 123 --user reader --no-input", connectionNotes + "Use --group-sql for SQL hash aggregates; --include-text is still required for statements."),
CommandGuide("capture", "Save a resumable XLog capture, optionally including profiles.", xlogOptions + "--directory",
connectionOptions + pagingOptions + setOf("--resume", "--include-profiles", "--include-text"),
"capture $windowExample --directory capture --user reader --no-input",
connectionNotes + listOf("--include-text requires --include-profiles. Text can contain sensitive values.",
"Use the same query and directory with --resume. Profile capture defaults to page size 25, maximum 100.")),
CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions,
"inspect --directory capture --txid 123"),
CommandGuide("inspect", "Read a captured profile offline.", setOf("--directory", "--txid"), offlineOptions + "--group-sql",
"inspect --directory capture --txid 123", listOf("Use --group-sql for SQL hash aggregates; --include-text requires stored text.")),
CommandGuide("inspect search", "Search captured requests offline.", setOf("--directory"), offlineOptions +
setOf("--limit", "--min-elapsed-ms", "--errors-only", "--service-hash", "--service-name"),
"inspect search --directory capture --errors-only", listOf("--service-name requires --include-text and stored dictionary text.")),
Expand Down
7 changes: 5 additions & 2 deletions src/main/kotlin/dev/scouterctl/cli/CommandInterpreter.kt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ import java.time.format.DateTimeFormatter

internal fun Cli.interpret(): CommandRequest {
val command = words.joinToString(" ")
if (spec.commandLine().parseResult.hasMatchedOption("--group-sql") && command !in setOf("profile get", "inspect")) {
invalid("--group-sql requires profile get or inspect")
}
if (command in setOf("context add", "context list", "context use")) return contextRequest(command)
if (name != null) invalid("--name requires context add/use")
if (command == "inspect compare") {
Expand Down Expand Up @@ -65,7 +68,7 @@ internal fun Cli.interpret(): CommandRequest {
}
"inspect" -> {
if (resume) invalid("--resume requires capture")
OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText)
OfflineRequest.Inspect(required(directory, "--directory"), required(txid, "--txid"), includeText, groupSql)
}
else -> onlineRequest(command)
}
Expand Down Expand Up @@ -110,7 +113,7 @@ private fun Cli.onlineRequest(command: String): OnlineRequest {
return when (command) {
"doctor", "auth login" -> OnlineRequest.Doctor(command == "auth login")
"object list" -> OnlineRequest.Objects(date?.also(::validateStorageDate))
"profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText)
"profile get" -> OnlineRequest.Profile(required(date, "--date"), required(txid, "--txid"), includeText, groupSql)
"xlog search" -> OnlineRequest.Search(required(query, "query"), maxPages, includeText)
"capture" -> OnlineRequest.Capture(
required(query, "query"), required(directory, "--directory"), resume, maxPages, includeProfiles, includeText,
Expand Down
4 changes: 2 additions & 2 deletions src/main/kotlin/dev/scouterctl/cli/CommandRequest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sealed interface ContextRequest : CommandRequest {
data class Use(val name: String?) : ContextRequest
}
sealed interface OfflineRequest : CommandRequest {
data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean) : OfflineRequest
data class Inspect(val directory: Path, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OfflineRequest
data class Search(val directory: Path, val filter: EvidenceFilter, val includeText: Boolean) : OfflineRequest
data class Summary(val directory: Path, val options: ServiceSummaryOptions) : OfflineRequest
data class Compare(val before: Path, val after: Path, val options: ComparisonOptions) : OfflineRequest
Expand All @@ -26,7 +26,7 @@ sealed interface OnlineRequest : CommandRequest {
data class CounterHistory(val query: CounterHistoryQuery) : OnlineRequest
data class Counters(val objHash: Int, val name: String?) : OnlineRequest
data class Objects(val date: String?) : OnlineRequest
data class Profile(val date: String, val txid: Long, val includeText: Boolean) : OnlineRequest
data class Profile(val date: String, val txid: Long, val includeText: Boolean, val groupSql: Boolean = false) : OnlineRequest
data class Search(val query: Query, val maxPages: Int, val includeText: Boolean = false) : OnlineRequest
data class Capture(val query: Query, val directory: Path, val resume: Boolean,
val maxPages: Int, val includeProfiles: Boolean, val includeText: Boolean) : OnlineRequest
Expand Down
4 changes: 3 additions & 1 deletion src/main/kotlin/dev/scouterctl/cli/OfflineCommands.kt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ class OfflineCommands(private val repository: (Path) -> EvidenceRepository, priv
}
private fun inspect(request: OfflineRequest.Inspect): Int {
val result = inspectCapture(repository(request.directory), request.txid, request.includeText)
return output.result(json.encodeToJsonElement(result.toDto()), result.captureExecutionComplete && result.unavailableProfiles == 0 &&
val data = if (request.groupSql) result.toSqlSummaryJson(result.record.profile?.let { summarizeSql(it, request.includeText) })
else json.encodeToJsonElement(result.toDto())
return output.result(data, result.captureExecutionComplete && result.unavailableProfiles == 0 &&
result.record.profile?.completeness != ProfileCompleteness.TRUNCATED)
}
private fun search(request: OfflineRequest.Search): Int {
Expand Down
Loading
Loading