diff --git a/.dev_ops/bench_db/DESIGN.md b/.dev_ops/bench_db/DESIGN.md deleted file mode 100644 index d2289879bdc..00000000000 --- a/.dev_ops/bench_db/DESIGN.md +++ /dev/null @@ -1,262 +0,0 @@ -# DB Metrics 快速性能校验设计 - -本文归纳 `feat/db_metric` 优化工作的三个前置部分：问题描述、设计方案，以及 -设计过程中评估过的替代方案和权衡。具体代码修改与验收记录见 -[`IMPLEMENTATION.md`](IMPLEMENTATION.md)，固定区块实验操作方案见 -[`README.md`](README.md)。 - -## 1. 问题描述 - -### 1.1 背景 - -目标是通过同步或回放一段固定的真实区块，快速判断数据库实现、数据库参数或 -RocksDB 表配置变化是否带来可重复的性能收益。 - -最终判断不能只看数据库内部数据，也不能只看节点 blocks/s，需要建立三层证据： - -```text -L0 配置与正确性 - └─ 参数是否生效、区块范围是否一致、执行是否成功 - │ - ▼ -L1 数据库内部行为 - └─ get/put/batch、cache、Bloom、flush、compaction、stall - │ - ▼ -L2 节点业务表现 - └─ blocks/s、block latency、CPU/block、disk bytes/block、GC -``` - -L0 是实验准入条件，L1 用来解释原因，L2 才是数据库方案是否值得采用的最终判断。 - -### 1.2 原 `feat/db_metric` 的主要问题 - -#### 指标和普通 Prometheus 强绑定 - -原实现只要启用全局 Prometheus，就会在每次 DB `get/put/delete/batch` 上启动 -Histogram timer。这样存在两个问题： - -- 普通节点为了使用系统、P2P 或业务指标，也被迫承担 DB 热点路径观测成本； -- 无法做“相同代码、相同 DB、DB 指标关闭/开启”的观测开销对照。 - -#### 快速测试采样周期不可用 - -`DbStatService` 原来每 6 小时采集一次 SST 和容量。一次快速测试通常只有 -20～60 分钟，除启动值外基本得不到中间状态，也无法观察 compaction backlog、 -write stall 或内存随区块推进的变化。 - -#### 数据库内部证据不足 - -原分支已经提供操作耗时、payload 大小和 RocksDB memory gauge，但仍缺少： - -- block cache 的 data/index/filter 命中与未命中； -- Bloom filter 是否真正过滤了无效读取； -- 读取落在 memtable、L0、L1 或 L2+ 的分布； -- flush/compaction 的实际读写量； -- pending compaction、running flush/compaction 和 write stall； -- RocksDB 逻辑 bytes 与操作系统实际磁盘 IO 的交叉证据。 - -#### 指标本身可能成为性能变量 - -原实现每次操作都通过 metric key 和字符串 labels 查找 Histogram child。DB 命中 -可能处于微秒级，此类重复数组分配、map 查找和 label 解析可能成为可见开销。 - -#### 直方图无法区分长尾 stall - -DB latency Histogram 的最大显式 bucket 是 10ms，所有超过 10ms 的操作都进入 -`+Inf`。这会丢失 20ms、100ms、500ms 等长尾差异，而这些长尾正是 compaction -和 IO stall 调查最关心的部分。 - -### 1.3 目标 - -- DB 指标必须独立 opt-in，默认不改变普通 Prometheus 节点的热点路径； -- 支持 10～30 分钟快速实验中的周期性状态采集； -- A、B 使用完全相同的指标实现和采集周期； -- 指标开启后尽量减少与业务无关的额外 allocation 和 label lookup； -- 同时提供操作层和 RocksDB 内部层证据； -- amd64 RocksDB JNI 5.15.10 与 aarch64 9.7.4 的属性差异不能导致采集线程退出； -- 指标能够解释结果，但不能被描述成真实业务收益本身。 - -### 1.4 非目标 - -- 不在本次修改中实现完整的固定区块导出和离线回放工具； -- 不把单元测试耗时当作生产数据库性能数据； -- 不以 value payload bytes 替代 WAL/SST/compaction 或操作系统磁盘 bytes； -- 不在尚未完成 D0/D1 固定区块实验前声称数据库性能已经提升； -- 不用指标自动决定数据库配置或执行数据库修复。 - -## 2. 设计方案 - -### 2.1 独立开关 - -在全局 Prometheus 下增加数据库子配置： - -```hocon -node.metrics.prometheus { - enable = true - database { - enable = true - statIntervalSeconds = 30 - } -} -``` - -语义如下： - -| 配置 | 含义 | -|---|---| -| `prometheus.enable=false` | 所有 Prometheus 指标关闭 | -| `prometheus.enable=true, database.enable=false` | 保留原有业务/系统指标，不注入 DB 热点路径指标 | -| `prometheus.enable=true, database.enable=true` | 开启 DB 操作 Histogram、周期 property 和 RocksDB Statistics | -| `statIntervalSeconds` | DB 状态采集周期，允许 5～3600 秒，默认 30 秒 | - -独立开关在节点初始化、数据库打开前确定，不支持运行期动态切换。性能实验中的 -metrics-off/metrics-on 对照应通过重启新 JVM 和 D0 新副本完成。 - -### 2.2 热点操作指标 - -每个 DB 实例在打开时一次性绑定固定 label 组合： - -```text -engine × db × operation -``` - -operation 的有界集合为 `get/put/delete/batch`。预绑定对象持有对应的 -`Histogram.Child`： - -- 开关关闭时，child 为 `null`，热点路径只执行一次空值判断； -- 开关开启时，操作开始只调用已绑定 child 的 `startTimer()`； -- 不在每次操作中重新构造 labels 或执行 `histogram.labels(...)`。 - -操作指标： - -| Metric | 说明 | -|---|---| -| `tron:db_operate_latency_seconds` | DB 操作耗时分布 | -| `tron:db_operate_bytes` | get 返回 value、put value、batch 非空 values 的 payload 分布 | - -latency buckets 保留 1～100µs 的密集区间，并从原最大 10ms 扩展到 1s，以区分 -长尾 IO/compaction stall。 - -### 2.3 周期状态指标 - -`DbStatService` 只在 database metrics 开启时注册周期任务，按 -`statIntervalSeconds` 运行。 - -继续保留： - -- `tron:db_sst_level`； -- `tron:db_size_bytes`； -- `tron:db_memory_bytes`。 - -新增即时 RocksDB property gauge： - -```text -tron:db_rocksdb_property{type,db,property} -``` - -覆盖： - -- pending compaction bytes； -- running compactions / flushes； -- actual delayed write rate； -- write stopped； -- immutable memtable； -- pending flush / compaction； -- background errors。 - -不同 RocksDB JNI 版本不支持某个 property 时，只记录 DEBUG 并跳过该 property， -不能让定时任务因异常永久停止。 - -### 2.4 RocksDB Statistics - -database metrics 开启时，为每个 RocksDB Options 启用 Statistics，StatsLevel 固定为： - -```text -EXCEPT_DETAILED_TIMERS -``` - -选择它是为了获得 ticker，同时避免 `ALL` 中详细 timer 带来的额外观测成本。 -Prometheus 主动轮询 ticker，因此关闭 RocksDB 自带的周期 stats log dump，避免形成 -第二套重复采集和日志噪声。 - -导出： - -```text -tron:db_rocksdb_ticker_total{type,db,ticker} -``` - -ticker 包括： - -- block cache 总体及 data/index/filter hit/miss； -- Bloom useful； -- memtable、L0、L1、L2+ hit； -- keys read/written； -- logical bytes read/written； -- flush write bytes； -- compaction read/write bytes； -- stall microseconds。 - -RocksDB ticker 是数据库打开以来的累计值。实现保存上次采样快照，只向 Prometheus -Counter 增加 delta；DB reset/reopen 时清空快照。 - -### 2.5 Native 资源生命周期 - -`Options.statistics()` 会返回新的 Java/native wrapper，不能在每次周期采集时反复 -调用而不关闭。设计为： - -1. DB 打开时获取一次 Statistics wrapper； -2. 周期任务始终复用该 wrapper； -3. DB 关闭时按 `RocksDB → Statistics wrapper → Options` 的顺序释放； -4. 设置 Options 时使用的临时 Statistics wrapper 通过 try-with-resources 立即释放， - Options 保留 native shared reference。 - -### 2.6 测试和结果解释 - -指标开启后的正确比较方式： - -```text -A(metrics on) vs B(metrics on) # 比较数据库候选 -A(metrics off) vs A(metrics on) # 测量观测成本 -``` - -DB 指标只用于 L1 归因： - -- `db_operate_bytes` 是 value payload，不是物理写盘量； -- shared block cache 下，各 DB 的 `block-cache-usage` 不应未经验证直接求和； -- RocksDB Statistics 与 OS `iostat`/exporter 必须同时观察； -- 最终仍以固定区块窗口的 blocks/s、latency、CPU/block、disk bytes/block 为准。 - -## 3. 备选方案与权衡 - -| 议题 | 备选方案 | 结论与原因 | -|---|---|---| -| DB 指标开关 | 跟随全局 Prometheus | 放弃；无法隔离热点路径观测成本 | -| DB 指标开关 | 独立 opt-in | 采用；普通指标和 DB 压测指标解耦 | -| 状态采集周期 | 固定 6 小时 | 放弃；快速实验拿不到中间状态 | -| 状态采集周期 | 固定 10 秒 | 放弃；普通诊断过密，缺乏环境适配 | -| 状态采集周期 | 5～3600 秒可配置、默认 30 秒 | 采用；兼顾短测和运行成本 | -| 状态触发 | HTTP 按需采集接口 | 暂缓；增加 API、安全和并发语义，不是最小修改 | -| 操作 labels | 每次调用动态查找 | 放弃；热点路径有重复 allocation/map lookup | -| 操作 labels | DB 打开时预绑定 child | 采用；label 集合固定且基数有界 | -| 操作采样 | 每 N 次记录一次 | 暂缓；会使 Histogram count/ops 解释复杂，需要额外 sample ratio | -| 内部统计 | 只使用 Java 层 timer | 不足；无法解释 cache/Bloom/compaction/stall | -| 内部统计 | 只使用 RocksDB Statistics | 不足；LevelDB 无对应能力，也缺少 java-tron 调用边界 | -| 内部统计 | Java 操作指标 + RocksDB Statistics | 采用；兼顾调用层和引擎层 | -| StatsLevel | `ALL` | 放弃；详细 timer 观测成本更高 | -| StatsLevel | `EXCEPT_DETAILED_TIMERS` | 采用；目标 ticker 可用，成本更可控 | -| RocksDB stats 输出 | 保留每 60 秒 LOG dump | 放弃；与 Prometheus 重复并产生大量日志 | -| RocksDB stats 输出 | Prometheus 主动轮询 | 采用；统一采集窗口和标签 | -| property 兼容性 | 任一不支持就失败 | 放弃；amd64/aarch64 JNI 属性集合可能不同 | -| property 兼容性 | 单项跳过并 DEBUG | 采用；保留其余指标，避免周期任务终止 | -| 初始性能实验 | 一开始直接 ABBA | 放弃；复杂度和成本过高，容易二次衍生问题 | -| 初始性能实验 | preflight → A/B → 明显差异后 ABBA | 采用；先筛选，再用 ABBA 排除顺序漂移 | -| 区块来源 | 只做真实 P2P 同步 | 不足；网络和 peer 队列会干扰 DB 归因 | -| 区块来源 | 只做 JUnit 单元测试 | 放弃；不能反映真实状态、VM 和 maintenance | -| 区块来源 | 离线真实区块回放 + 固定 peer 同步 | 目标方案；分别回答 DB 自身和端到端表现 | - -## 设计边界 - -本设计完成的是“可用于受控实验的指标能力”。它不直接证明任何数据库参数有 -性能收益。性能结论必须在同一 D0、同一区块窗口、同一代码基线和相同指标配置下 -另行执行，并按 [`README.md`](README.md) 的阶段门禁记录结果。 diff --git a/.dev_ops/bench_db/DESIGN_BLOCK_APPLY.md b/.dev_ops/bench_db/DESIGN_BLOCK_APPLY.md deleted file mode 100644 index 7ec5b604813..00000000000 --- a/.dev_ops/bench_db/DESIGN_BLOCK_APPLY.md +++ /dev/null @@ -1,294 +0,0 @@ -# 固定区块导出与离线回放设计 - -本文记录 `feature/block_apply` 的问题定义、设计方案和方案权衡。实现与验收证据见 -[`IMPLEMENTATION_BLOCK_APPLY.md`](IMPLEMENTATION_BLOCK_APPLY.md)，完整数据库 -性能实验流程见 [`README.md`](README.md)。 - -## 1. 问题描述 - -### 1.1 背景 - -真实 P2P 同步能够反映完整业务表现，但数据库 A/B 初筛容易受到以下因素干扰： - -- peer 响应速度和网络抖动； -- block fetch 批次、队列和网络线程调度； -- 数据源继续增长或不同轮次取得不同区块； -- 同步连接建立、断开及其他 peer 流量； -- API 请求和后台任务与区块处理争用资源。 - -为了快速判断数据库方案是否值得进入复杂测试，需要从一个固定 D1 中提取真实 -区块，并在同一 D0 的不同副本上离线回放同一高度区间： - -```text -停止的 D1 - block-index: height -> block ID - block: block ID -> protobuf - │ - ▼ - versioned block file - │ - ▼ -D0 fresh copy -> sync processing path -> D0 + N blocks -``` - -该方式排除区块获取网络，但保留真实交易组合、状态依赖、共识校验、VM、 -maintenance、revoking DB 和持久化行为。 - -### 1.2 要解决的问题 - -#### 可重复的数据输入 - -A、B 必须处理完全相同的区块。如果每轮重新向 peer 获取数据，即使高度区间相同， -数据源状态、请求节奏和连接行为也可能变化。固定文件需要成为每轮只读输入。 - -#### 不能只复制数据库原始 KV - -直接把 `block`、`block-index` 记录写入 D0 只证明数据库可以写入原始数据，不会 -执行交易、VM、maintenance 和状态变更，无法代表数据库在复杂业务中的表现。 - -回放必须进入节点真实的同步区块处理路径。 - -#### 跨数据库导出的完整性 - -区块需要先从 `block-index` 取得 block ID，再到 `block` 读取 protobuf。两个库 -之间没有供外部工具使用的原子快照，因此导出应要求 D1 正常停止，并对高度、 -父块和文件结构做二次校验。 - -#### 防止误修改数据库 - -离线回放会真实修改 D0。如果文件接错快照或用户误把生产目录当作测试副本， -可能产生不可逆的状态变化。因此命令默认只校验文件，应用必须显式授权，并在 -写入前校验 D0 head。 - -#### 大区间内存和故障恢复 - -测试窗口可能包含数万乃至更多区块。实现不能一次性把全部区块加载到内存；导出 -过程中发生错误也不能留下一个看起来完整的目标文件。 - -### 1.3 目标 - -- 从停止节点导出闭区间 `[start, end]` 的主链真实区块； -- 使用版本化、可流式处理的文件格式； -- 检测非法范围、缺块、protobuf 损坏、CRC 错误、区块不连续和尾随数据； -- 默认拒绝覆盖已有文件，并通过临时文件完成原子发布； -- 提供不修改 D0 的只读 verify 模式； -- 只有显式 `--apply` 才打开 D0 并执行回放； -- 首块必须为 `D0 head + 1`，parent ID 必须匹配 D0 head ID； -- 逐块通过真实同步入口处理，并在每块后校验新 head； -- 支持预热区间和最大处理块数，服务快速 preflight； -- 输出固定窗口的处理块数、耗时和 blocks/s。 - -### 1.4 非目标 - -- 不替代固定 peer 的真实 P2P 同步验收； -- 不支持在线节点的一致性导出； -- 不从任意 fork 或 raw block 记录推断主链，区块来源以 `block-index` 为准； -- 不绕过共识、交易、VM 或状态执行来制造更高的基准数字； -- 不自动复制、恢复或清理 D0； -- 不允许在同一已经应用过的 D0 上重复一轮 A/B； -- 不因工具链测试通过就声称数据库性能提升。 - -## 2. 设计方案 - -### 2.1 模块边界 - -```text -common - BlockFile - ├─ versioned header - ├─ streaming writer - ├─ streaming reader - └─ structural validation - -plugins / Toolkit.jar - db block export - └─ read block-index + block - -framework / FullNode.jar - BlockReplay - ├─ verify-only - └─ apply via TronNetDelegate.processBlock(block, true) -``` - -文件格式放在 `common`，使 Toolkit 写入端和 FullNode 读取端共享同一份协议实现， -避免两个模块分别维护编码规则。 - -### 2.2 区块文件格式 - -文件头： - -| 字段 | 类型 | 说明 | -|---|---|---| -| magic | 8 bytes | `TRONBLK1` | -| version | int32 | 当前为 1 | -| start | int64 | 首块高度 | -| end | int64 | 末块高度 | -| count | int64 | 必须等于 `end - start + 1` | - -每条记录： - -| 字段 | 类型 | 说明 | -|---|---|---| -| height | int64 | 当前高度 | -| block ID | 32 bytes | 来源 `block-index` 的主链 ID | -| protobuf length | int32 | 最大允许 64 MiB | -| protobuf | bytes | 原始 `Protocol.Block` | -| CRC32 | int32 | protobuf 内容校验 | - -选择固定头加 length-prefixed record，可以逐块读写，不需要为整个窗口分配内存。 -block ID 独立保存，用于在 FullNode 使用目标链配置重新计算 ID 后进行核对。 - -### 2.3 导出流程 - -Toolkit 命令： - -```text -db block export - --database-directory - --start - --end - --output - [--overwrite] -``` - -流程： - -1. 检查 `block` 和 `block-index` 目录已经存在，避免因路径错误创建空库； -2. 按高度从 `block-index` 读取 block ID； -3. 使用 block ID 从 `block` 读取 protobuf； -4. 校验 protobuf 高度和前后父块连续性； -5. 写入与目标文件同目录的临时文件； -6. 全部成功后原子移动为目标文件； -7. 任一步失败时删除临时文件并保留原目标文件。 - -默认禁止覆盖，只有显式 `--overwrite` 才允许替换。 - -### 2.4 只读校验 - -`BlockReplay` 不带 `--apply` 时： - -- 加载指定 config，以使用正确的链和加密引擎配置； -- 校验 magic、version、范围和 count； -- 校验每条记录的长度和 CRC； -- 解析 protobuf 并校验记录高度； -- 校验文件内部父块连续性； -- 重新计算 block ID，并与文件保存的源 block ID 比较； -- 读完整个文件并拒绝尾随数据； -- 不创建 Spring context，不打开或修改 D0。 - -### 2.5 离线应用 - -`--apply` 模式要求指定一个已存在的 D0 output directory，并强制： - -```text ---p2p-disable true -``` - -Spring context `refresh()` 完成真实 Manager、VM 和 Store Bean 初始化，但**不会启动 -Consensus**。正常 FullNode 是在 `ApplicationImpl.startup()` 中显式调用 -`ConsensusService.start()`。Replay 不启动普通 API/P2P 服务，因此必须在 `refresh()` 后、 -处理第一块前单独启动 `ConsensusService`。区块处理入口为： - -```java -tronNetDelegate.processBlock(block, true); -``` - -它保留同步区块的锁、fresh-block cache 和业务指标，然后进入 -`Manager.pushBlock`。不使用 `pushVerifiedBlock`，因为该入口会设置 -`generatedByMyself=true` 并绕过部分外部区块校验。 - -离线应用生命周期必须保持以下顺序： - -```text -context.refresh() - -> ConsensusService.start() - -> processBlock(block, true) - -> ConsensusService.stop() - -> stop and await RewardViCalService - -> close Manager / RocksDB - -> destroy Spring beans -``` - -`RewardViCalService` 会在 Manager 初始化时启动后台 RocksDB 遍历。退出时必须先停止并等待 -该线程，再关闭底层数据库；仅依赖 Spring `@PreDestroy` 会晚于 -`TronApplicationContext.doClose()` 中的 `Application.shutdown()`/`Manager.close()`，可能与 -RocksDB JNI 关闭发生竞态。 - -应用门禁： - -1. 文件 start 必须等于 `D0 head + 1`； -2. 首块 parent ID 必须等于 D0 head ID； -3. 文件 block ID 必须等于当前 config 下重新计算的 block ID； -4. 每块处理后 D0 head 高度和 ID 必须等于该块； -5. 任一校验或业务执行失败，命令立即停止。 - -### 2.6 计时边界 - -计时只包围： - -```text -TronNetDelegate.processBlock(block, true) -``` - -不包含： - -- JVM 和 Spring 启动； -- config 解析和数据库打开； -- 文件读取、CRC、protobuf 解析和 block ID 校验； -- 最终 context 关闭。 - -`--warmup-blocks N` 表示前 N 块仍然完整应用，但不计入 elapsed 和吞吐。 -`--max-blocks N` 用于只处理文件开头的 N 块，适合 100～1,000 块 preflight。 - -输出使用固定 Locale，便于脚本解析： - -```text -mode=apply range=[start,end] processed=N warmup=W measured=M -elapsed_ms=... blocks_per_second=... -``` - -### 2.7 实验使用边界 - -固定文件只控制区块输入；以下变量仍需实验脚本控制： - -- A、B 每轮必须从同一 D0 的全新副本开始； -- 使用同一 config、JDK、JVM、CPU 和磁盘； -- 指标代码和开关保持一致； -- 每轮使用新 JVM； -- 记录是否覆盖 maintenance、flush 和 compaction； -- 离线回放用于筛选和归因，最终仍需固定 peer 同步确认端到端结果。 - -## 3. 备选方案与权衡 - -| 议题 | 备选方案 | 结论与原因 | -|---|---|---| -| 区块来源 | 每轮从 P2P 获取 | 不作为初筛；网络和 peer 行为成为干扰变量 | -| 区块来源 | 固定真实区块文件 | 采用；每轮输入完全一致且可复用 | -| 数据格式 | JSON | 放弃；体积大、转换慢且可能改变 protobuf 未知字段 | -| 数据格式 | Java serialization | 放弃；版本耦合且不适合作为稳定数据协议 | -| 数据格式 | 原始 protobuf 顺序拼接 | 不足；缺少版本、范围、ID 和损坏检测 | -| 数据格式 | 版本头 + length-prefixed protobuf + ID + CRC | 采用；流式、可校验、可演进 | -| 导出读取 | 只遍历 `block` | 放弃；raw block 可能包含非当前主链记录 | -| 导出读取 | 以 `block-index` 按高度定位 `block` | 采用；明确选择当前主链区块 | -| 在线一致性 | 允许运行节点直接导出 | 放弃；两个数据库没有外部原子快照 | -| 在线一致性 | 要求 D1 正常停止 | 采用；简单、低风险且适合固定数据源 | -| 文件写入 | 直接写目标文件 | 放弃；失败会留下看似可用的半文件 | -| 文件写入 | 同目录临时文件后原子移动 | 采用；失败时不发布不完整结果 | -| 默认行为 | 直接应用 D0 | 放弃；误操作风险高 | -| 默认行为 | verify-only，显式 `--apply` | 采用；先验证再修改 | -| 回放入口 | 直接写 block/block-index KV | 放弃；不执行复杂业务状态 | -| 回放入口 | `Manager.pushBlock` | 可执行核心业务，但少同步入口锁、cache 和 process metric | -| 回放入口 | `TronNetDelegate.processBlock(block, true)` | 采用；最贴近真实同步且不需要网络获取 | -| 回放入口 | `pushVerifiedBlock` | 放弃；会标记本地产生并绕过部分校验 | -| 工具承载 | 普通短生命周期 JUnit | 不作为用户入口；不适合真实 D0 和长窗口 | -| 工具承载 | 独立 CLI + 聚焦单测 | 采用；CLI 执行真实实验，单测验证格式和门禁 | -| 重复测试 | 在同一 D0 连续 replay | 放弃；后轮状态已变化，不能形成 A/B | -| 重复测试 | 每轮恢复 D0 新副本 | 采用；保持起始状态一致 | - -## 设计边界 - -该设计建立的是“固定真实输入、排除网络获取”的低噪声集成测试能力。它比构造 -少量假交易的单元测试更接近真实业务，但仍不包含 P2P、peer 队列和网络反压。 - -因此离线回放出现明显差异后，仍需在固定单 peer 的真实同步中确认方向；只有 -正确性门禁、重复性和端到端结果同时成立，才能形成数据库性能结论。 diff --git a/.dev_ops/bench_db/IMPLEMENTATION.md b/.dev_ops/bench_db/IMPLEMENTATION.md deleted file mode 100644 index c805e5a28b7..00000000000 --- a/.dev_ops/bench_db/IMPLEMENTATION.md +++ /dev/null @@ -1,305 +0,0 @@ -# DB Metrics 实现与验收记录 - -本文归纳 `feat/db_metric` 优化工作的后两个部分：实现过程与验收过程。问题定义、 -设计和方案权衡见 [`DESIGN.md`](DESIGN.md)。 - -## 1. 实现过程 - -### 1.1 分支和提交 - -| 项目 | 值 | -|---|---| -| 工作目录 | `/Users/blade/java/src/awork/java-tron` | -| 分支 | `feat/db_metric` | -| 本次同步基线 | `upstream/develop` at `4a21592f95` | -| develop merge commit | `fdcf7f9673` | -| 优化 commit | `4e514b15fd61b942be863452b16f10fdc8d7ce3d` | -| commit subject | `feat(metrics): make db metrics benchmark-ready` | -| 提交规模 | 20 files, 454 insertions, 48 deletions | -| 推送状态 | 未推送；分支没有 tracking branch | - -`.dev_ops/bench_db` 由 `.git/info/exclude` 排除，本文、`DESIGN.md` 和 `README.md` -都是本地操作文档，不在上述 commit 中。 - -### 1.2 同步最新 develop - -优化前，本地 `feat/db_metric` 相对当时最新 `upstream/develop` 落后 45 个提交、 -领先 7 个指标提交。先刷新 upstream，再使用仓库要求的 `--no-ff` 合并： - -```bash -git fetch upstream -git merge upstream/develop --no-ff -``` - -合并无源码冲突，生成 `fdcf7f9673`。最终优化 commit 后，相对该 -`upstream/develop` 为 `0 behind / 9 ahead`，9 个 ahead 包括原 7 个指标提交、 -develop merge commit 和本次优化 commit。 - -### 1.3 配置实现 - -新增配置 Bean： - -```text -MetricsConfig -└─ PrometheusConfig - └─ DatabaseConfig - ├─ enable=false - └─ statIntervalSeconds=30 -``` - -涉及文件： - -- `common/.../MetricsConfig.java`：配置绑定和 5～3600 秒边界校验； -- `CommonParameter.java`：运行期配置字段； -- `framework/.../Args.java`：配置桥接，并按开关启用 RocksDB Statistics； -- `reference.conf`、`framework/config.conf`：默认配置和示例。 - -DB 指标最终生效条件： - -```java -metricsPrometheusEnable && metricsPrometheusDatabaseEnable -``` - -### 1.4 热点路径实现 - -新增 `DbOperationMetrics`，在 LevelDB/RocksDB datasource 构造时预绑定： - -- get latency / bytes； -- put latency / bytes； -- delete latency； -- batch latency / bytes。 - -原调用方式： - -```text -每次操作 → metric key lookup → labels(engine, db, op) → child → observe -``` - -优化后： - -```text -DB open → 一次性绑定 child -每次操作 → cached child → timer/observe -``` - -database metrics 关闭时所有 child 为 `null`，不计算 batch payload 总量，也不访问 -Prometheus child。 - -### 1.5 周期状态实现 - -`DbStatService` 从固定 6 小时调整为配置驱动的秒级周期，只在 database metrics -开启时注册任务。 - -RocksDB datasource 的 `stat()` 依次采集： - -1. SST level 和 size； -2. RocksDB memory； -3. compaction/flush/write pressure properties； -4. Statistics ticker delta。 - -LevelDB logger event counter 也受 database metrics 子开关控制，避免只开启普通 -Prometheus 时继续解析事件并更新 Counter。 - -### 1.6 RocksDB Statistics 实现 - -`RocksDbSettings` 在 database metrics 开启时： - -- 创建 Statistics； -- 设置 `EXCEPT_DETAILED_TIMERS`； -- 注入 Options； -- 关闭 stats dump； -- 释放设置阶段的临时 Java wrapper。 - -`RocksDbDataSourceImpl` 在打开 DB 后获取并缓存一个 Statistics wrapper。每次 -`stat()` 读取选定 ticker 的当前累计值，与上次快照求 delta，再增加 Prometheus -Counter。 - -实现过程中发现 `options.statistics()` 每次调用都会创建新的 Java/native wrapper。 -如果直接在 30 秒周期任务中调用而不关闭，会形成长期 native wrapper 泄漏。因此 -最终实现只在 DB open 时调用一次，并在 close 时显式释放。 - -### 1.7 Histogram 和指标文档 - -DB latency buckets 从最高 10ms 扩展到 1s，新增 20ms、50ms、100ms、500ms 和 -1s 等区间。 - -`METRICS_CHANGELOG.md` 补充： - -- 独立配置示例； -- metrics-off/metrics-on 控制要求； -- RocksDB property 和 ticker 定义； -- shared cache 不应直接跨 DB 求和； -- logical payload/bytes 不能替代 OS 物理磁盘指标。 - -### 1.8 实现中遇到的问题 - -#### Worktree 与 JGit - -最初为了保护其他分支工作区，在 `/private/tmp/java-tron-db-metric` worktree 中修改。 -源码编译可以通过，但 `framework:generateGitProperties` 使用的 JGit 无法识别： - -```text -.git/worktrees/java-tron-db-metric -``` - -导致 Checkstyle 前置任务失败。曾使用普通 `/private/tmp` clone 做验证；随后按用户 -要求将主目录直接切换到 `feat/db_metric`，把全部修改迁回主目录继续工作。 - -#### 沙箱内 Gradle daemon - -一次 Checkstyle 执行因沙箱禁止 Gradle daemon 绑定本地 socket 而失败。该失败 -发生在 Gradle 启动阶段，不是源码或测试失败；在允许的执行环境中重跑成功。 - -#### Native Statistics wrapper - -第一版 ticker 采集每次调用 `options.statistics()`。代码复核和 JNI bytecode 检查 -确认它会返回新 wrapper，因此改为 open 时获取一次、close 时释放。随后增加真实 -RocksDB ticker 导出测试，覆盖该路径。 - -#### LevelDB 文件换行 - -`LevelDbDataSourceImpl.java` 原文件使用 CRLF。修改时保留原有文件风格并确保 -`git diff --check` 无新增 trailing whitespace，避免把整个文件变成无关换行 diff。 - -## 2. 验收过程 - -### 2.1 验收层级 - -本次只验收指标实现和最小采集链路： - -```text -配置绑定 - → Java 编译 - → Checkstyle - → 指标开关/Histogram 单测 - → LevelDB/RocksDB datasource 回归 - → 真实 RocksDB read/write/stat/ticker preflight -``` - -本次不包含固定 D0/D1 区块同步，因此验收结果不能解释为数据库配置性能提升。 - -### 2.2 编译 - -执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home \ - :common:compileJava \ - :chainbase:compileJava \ - :framework:compileJava -``` - -结果：`BUILD SUCCESSFUL`。 - -覆盖配置 Bean、Prometheus 公共类、LevelDB/RocksDB datasource 和 framework Args -桥接的编译依赖。 - -### 2.3 配置测试 - -执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home :common:test \ - --tests org.tron.core.config.args.MetricsConfigTest \ - --tests org.tron.core.config.args.ConfigParityGateTest -``` - -结果：`BUILD SUCCESSFUL`。 - -验证内容： - -- database metrics 默认关闭； -- 默认周期为 30 秒； -- benchmark 可配置为 10 秒； -- 小于 5 秒的配置被拒绝； -- `reference.conf` 和配置 Bean 字段完整对齐。 - -### 2.4 指标和 datasource 测试 - -执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home :framework:test \ - --tests org.tron.common.storage.metric.DbOperationMetricsTest \ - --tests org.tron.common.storage.rocksdb.RocksDbDataSourceImplTest \ - --tests org.tron.common.storage.leveldb.LevelDbDataSourceImplTest -``` - -结果：全部通过。 - -验证内容： - -- 全局 Prometheus 开启但 database metrics 关闭时，不创建 DB timer； -- database metrics 开启时，预绑定 latency/bytes child 正常累加； -- LevelDB datasource 打开、读写、engine 检查和 watchdog 回归； -- RocksDB datasource 打开、读写、backup 和 engine 检查回归； -- 真实 RocksDB 临时库执行 put/get 后调用 `stat()`； -- Prometheus 能读到 `number_keys_written >= 1`； -- datasource 关闭后 Statistics/Options 资源释放路径无异常。 - -### 2.5 Checkstyle 和 diff - -执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home \ - :framework:checkstyleMain \ - :framework:checkstyleTest - -git diff --check -``` - -结果：全部通过。 - -曾出现一次测试 import 顺序告警，调整 `CollectorRegistry` import 后重跑通过。 - -### 2.6 已通过的验收项 - -| 验收项 | 状态 | 证据边界 | -|---|---|---| -| 最新 develop 合并 | 通过 | 本地 `upstream/develop` at `4a21592f95` | -| DB 指标独立开关 | 通过 | 配置及关闭路径单测 | -| 周期配置 | 通过 | 默认、覆盖和非法边界测试 | -| 热点 label 预绑定 | 通过 | Histogram count 单测 | -| LevelDB 回归 | 通过 | 聚焦 datasource 测试 | -| RocksDB 回归 | 通过 | 聚焦 datasource 测试 | -| RocksDB ticker 导出 | 通过 | 真实临时库 put/get/stat preflight | -| Native wrapper 生命周期 | 通过 | 单次 wrapper 设计、关闭路径和 preflight | -| 配置 parity | 通过 | `ConfigParityGateTest` | -| Checkstyle | 通过 | main/test tasks | -| 提交规范 | 通过 | `4e514b15fd feat(metrics): make db metrics benchmark-ready` | - -### 2.7 尚未完成的性能验收 - -以下工作明确未执行： - -- metrics-off 与 metrics-on 的实际 CPU/block、latency、blocks/s 开销对照； -- 同一 D0 上固定区块离线回放 A/B； -- 固定单 peer 的真实同步 A/B； -- flush/compaction 确实发生时的 Bloom/block size 评价； -- amd64 JNI 5.15.10 与 aarch64 JNI 9.7.4 的同窗口对照； -- 出现明显差异后的 ABBA 反向复测； -- Arthas/async-profiler 差异归因。 - -因此当前可接受的结论仅为： - -> `feat/db_metric` 已具备用于受控快速实验的配置、操作指标、RocksDB 内部指标和 -> 最小采集链路，且聚焦编译、风格、配置和 datasource 回归通过。 - -当前不能接受的结论是： - -> 某个数据库或 RocksDB 配置已经获得确定的性能提升。 - -### 2.8 下一阶段验收顺序 - -按低成本到高成本执行： - -1. A、B 各回放 100～1,000 块 preflight，验证构建、D0/D1、指标和正常关闭； -2. 相同方案做 metrics-off/metrics-on，得到观测成本； -3. 离线固定区块 `A1 → B1`，筛选 DB 自身差异； -4. 固定 peer 同步 `A1 → B1`，确认端到端方向； -5. 只有出现明显差异且 Candidate 基本确定后，补 `B2 → A2` 形成 ABBA； -6. 只有差异来源不清时再采集 Arthas/async-profiler。 - -若 Candidate 在预热阶段失败、没有进入固定测量窗口，结果必须记录为“测量前 -失败”，不能计算性能退化百分比。 diff --git a/.dev_ops/bench_db/IMPLEMENTATION_BLOCK_APPLY.md b/.dev_ops/bench_db/IMPLEMENTATION_BLOCK_APPLY.md deleted file mode 100644 index 104d0ab0f51..00000000000 --- a/.dev_ops/bench_db/IMPLEMENTATION_BLOCK_APPLY.md +++ /dev/null @@ -1,436 +0,0 @@ -# 固定区块导出与离线回放实现及验收 - -本文记录 `feature/block_apply` 的实现过程和验收过程。问题定义、设计与方案权衡 -见 [`DESIGN_BLOCK_APPLY.md`](DESIGN_BLOCK_APPLY.md)。 - -## 1. 实现过程 - -### 1.1 分支和基线 - -| 项目 | 值 | -|---|---| -| 工作目录 | `/Users/blade/java/src/awork/java-tron` | -| 分支 | `feature/block_apply` | -| 父分支 | `feat/db_metric` | -| 父分支 HEAD | `1c4f9ae35e docs(metrics): document db benchmark design` | -| 网络处理 | replay 强制禁用 P2P | -| 当前状态 | 首次真实 apply 暴露生命周期缺陷；代码修复完成，真实 D0 复验待执行 | - -用户最初指定 `feat/block_apply`。仓库贡献约定要求功能分支使用 `feature/*`，因此 -实际创建 `feature/block_apply`，并从已完成的 DB metrics 分支继续开发，使离线 -回放可以直接采集上一阶段新增的数据库指标。 - -### 1.2 现有能力调查 - -实现前确认了以下现有入口： - -- `block-index`：高度到当前主链 block ID； -- `block`：block ID 到原始 `Protocol.Block` protobuf； -- `BlockCapsule(Block)`：解析区块并计算 block ID； -- `Manager.pushBlock`：执行共识、交易、VM、maintenance 和持久化； -- `TronNetDelegate.processBlock(block, true)`：真实同步区块处理入口； -- Toolkit picocli 命令框架：适合增加导出命令； -- FullNode fat jar：可以通过 classpath 直接运行独立 replay main class。 - -最初原型直接调用 `Manager.pushBlock`。与已有性能设计复核后，改为 -`TronNetDelegate.processBlock(block, true)`，以保留同步入口锁、fresh-block cache -以及 `block_process_latency{sync=true}` 指标。 - -### 1.3 公共文件格式 - -新增： - -```text -common/src/main/java/org/tron/common/utils/BlockFile.java -``` - -实现内容： - -- `TRONBLK1` magic 和 version 1； -- start、end、count 文件头； -- height、32-byte block ID、length、protobuf、CRC32 记录； -- 64 MiB 单块大小上限； -- 流式 `RecordSource` writer； -- 流式 `Reader`； -- 高度、protobuf 高度、父块、CRC、截断和尾随数据校验； -- 同目录临时文件和原子移动； -- 默认禁止覆盖，失败清理临时文件。 - -Writer 和 Reader 位于 common，使 Toolkit 和 FullNode 使用同一协议实现。 - -### 1.4 Toolkit 导出命令 - -新增： - -```text -plugins/src/main/java/common/org/tron/plugins/DbBlock.java -plugins/src/main/java/common/org/tron/plugins/DbBlockExport.java -``` - -并在 `Db` 根命令注册： - -```text -Toolkit.jar db block export -``` - -实现流程： - -1. 规范化 database directory； -2. 在打开数据库前确认 `block`、`block-index` 目录存在； -3. 按 height 从 `block-index` 读取 ID； -4. 按 ID 从 `block` 读取 protobuf； -5. 交给 `BlockFile.write` 做格式和连续性校验； -6. 成功后输出导出范围、数量和目标路径； -7. finally 中关闭两个数据库。 - -该实现不会遍历 raw block 库猜测主链，也不会在路径错误时静默创建空数据库。 - -### 1.5 FullNode 离线回放命令 - -新增： - -```text -framework/src/main/java/org/tron/program/BlockReplay.java -``` - -运行方式： - -```bash -java -cp framework/build/libs/FullNode.jar org.tron.program.BlockReplay [options] -``` - -命令分两种模式： - -#### verify-only - -不带 `--apply`： - -- 只加载 config 和区块文件； -- 流式解析全部记录； -- 重新计算 block ID； -- 不创建 Spring context； -- 不打开或修改 D0。 - -#### apply - -带 `--apply`： - -- 要求 `--output-directory` 已存在； -- 向 Args 注入 `--p2p-disable true`； -- 初始化 metrics 和 Spring context； -- 获取 `TronNetDelegate`、`ChainBaseManager`； -- 校验文件 start 和 D0 head； -- 逐块调用 `processBlock(block, true)`； -- 每块后校验 D0 head； -- 最终关闭 context 和数据库。 - -`context.refresh()` 不会执行正常 FullNode 的 `ApplicationImpl.startup()`，因此 replay 还需 -显式启动 `ConsensusService`，并在关闭数据库前停止和等待 `RewardViCalService` 后台任务。 - -### 1.6 预热和计时 - -支持： - -| 参数 | 作用 | -|---|---| -| `--warmup-blocks N` | 前 N 块正常应用，但不计入时间 | -| `--max-blocks N` | 最多处理文件前 N 块，用于 preflight | - -计时使用 `System.nanoTime()`，只覆盖同步处理入口。输出用 `Locale.ROOT` 格式化， -避免不同系统小数点格式影响自动脚本。 - -### 1.7 文档 - -[`README.md`](README.md) 已补充： - -- fat jar 构建命令； -- D1 导出命令； -- verify-only 命令； -- D0 apply 命令； -- `database directory` 与 `output directory` 的区别； -- overwrite、D0 head、warmup、max-blocks 和计时边界； -- 每轮必须恢复 D0 新副本的要求。 - -### 1.8 实现中遇到的问题 - -#### 分支命名 - -用户指定 `feat/block_apply`，但 java-tron 本地约定要求 `feature/`。 -最终使用 `feature/block_apply`，功能基线仍来自 `feat/db_metric`。 - -#### Toolkit 模块不依赖 framework - -Toolkit 适合直接读取数据库，但不能调用 FullNode 的 Manager。没有为了复用命令 -而把 framework 整体引入 plugins；文件协议下沉 common，导出和回放分别位于其 -自然模块中。 - -#### Manager 与同步入口 - -第一版 replay 直接调用 `Manager.pushBlock`。它能执行核心业务，但没有覆盖 -`TronNetDelegate` 的同步锁、fresh block cache 和 block process latency 指标。 -最终改为 `processBlock(block, true)`。 - -#### Checkstyle 任务差异 - -首次组合验收尝试执行 `:common:checkstyleMain`，但 common 模块没有注册该任务， -Gradle 在测试前停止。这不是源码失败。最终按模块实际能力执行：common 编译和 -单测，plugins/framework 执行 Checkstyle。 - -#### Mockito 对 BlockCapsule 的比较 - -测试最初按对象实例 verify `Manager.pushBlock`，而文件读取会创建等价但不同实例 -的 `BlockCapsule`，导致 Mockito 参数比较失败。测试改用 captor 比较 block ID， -随后又随同步入口调整为验证 `TronNetDelegate.processBlock(block, true)`。 - -#### 首次真实 apply 的 Consensus 初始化失败 - -首次在真实 D0 上应用第一块时失败。代码级原因是： - -1. `BlockReplay.apply()` 只执行 `context.refresh()`； -2. 随后第一块直接进入 `TronNetDelegate.processBlock()`； -3. `Consensus` 使用的 `consensusInterface` 只会在 `ConsensusService.start()` 调用 - `Consensus.start()` 后完成赋值； -4. replay 没有调用该启动链路，第一块因此在共识处理阶段失败。 - -原设计中“Spring context 已完成 Consensus 初始化”的表述不成立，现已修正。实现改为在 -`refresh()` 后、取得区块处理 Bean 和处理第一块前显式执行 -`ConsensusService.start()`。未调用完整 `Application.startup()`，避免为离线 benchmark -启动普通 API 服务；P2P 仍由 `--p2p-disable true` 强制禁用。 - -#### 异常退出时 RocksDB JNI SIGSEGV - -主流程异常后关闭 context，又暴露了独立的关闭顺序问题： - -```text -旧顺序：Manager.close() -> RocksDB close -> Spring @PreDestroy -> reward thread stop -``` - -`RewardViCalService` 可能仍在遍历 delegation/witness/reward RocksDB，底层数据库先关闭会与 -native iterator 竞态。实际崩溃报告为： - -```text -/data/blade/node_mainnet/hs_err_pid161196.log -``` - -修复内容： - -- 为 `RewardViCalService` 增加可重复调用的 `stop()`； -- `stop()` 先设置停止标志并中断 executor，再等待线程退出； -- witness/reward iterator 改为 try-with-resources，确保 native iterator 及时关闭； -- 长 cycle 和 iterator 循环增加协作式停止检查； -- `Manager.close()` 在 `chainBaseManager.shutdown()`/RocksDB 关闭前调用该 `stop()`； -- Spring 后续再次执行 `@PreDestroy` 时保持幂等。 - -新的关键顺序为： - -```text -Consensus stop -> reward thread stop and await -> RocksDB close -> bean destroy -``` - -## 2. 验收过程 - -### 2.1 验收层级 - -```text -BlockFile 格式单测 - → Toolkit 真实 RocksDB 导出 - → replay 文件校验 - → sync path 调用和 D0 门禁测试 - → Checkstyle - → Toolkit/FullNode fat jar 构建 - → fat jar 类和 CLI 实际检查 -``` - -### 2.2 编译验收 - -执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home \ - :common:compileJava \ - :plugins:compileJava \ - :framework:compileJava \ - :common:compileTestJava \ - :plugins:compileTestJava \ - :framework:compileTestJava -``` - -结果：`BUILD SUCCESSFUL`。 - -### 2.3 BlockFile 测试 - -测试类： - -```text -org.tron.common.utils.BlockFileTest -``` - -覆盖： - -- 连续区块写入和读取； -- header 的 start/end/count； -- block ID 和 protobuf 高度； -- CRC 损坏检测； -- 默认禁止覆盖已有文件。 - -### 2.4 Toolkit 导出测试 - -测试类： - -```text -org.tron.plugins.DbBlockExportTest -``` - -测试使用临时真实 RocksDB： - -1. 创建 `block-index` 和 `block`； -2. 写入两个连续区块； -3. 关闭数据库； -4. 通过 `new CommandLine(new Toolkit())` 执行完整导出命令； -5. 用 `BlockFile.Reader` 验证文件区间和记录。 - -结果：通过。 - -### 2.5 Replay 测试 - -测试类： - -```text -org.tron.program.BlockReplayTest -``` - -覆盖： - -- verify 模式完整读取连续文件； -- 从真实命令行参数进入 verify 模式； -- apply 模式逐块调用 `TronNetDelegate.processBlock(block, true)`； -- 应用后 head 高度和 ID 校验； -- D0 head ID 不匹配时在处理首块前拒绝； -- warmup 块不计入 measured 统计。 -- replay apply 初始化阶段显式启动 `ConsensusService`。 - -结果：`BlockReplayTest` 新增 Consensus 生命周期用例后共 5 个测试，全部通过。 - -关闭竞态另新增： - -```text -org.tron.core.service.RewardViCalServiceLifecycleTest -``` - -测试向 reward executor 提交一个阻塞任务，再调用两次 `stop()`，验证工作线程收到 interrupt、 -`stop()` 等待 executor 完全终止且重复调用安全。该用例通过。 - -### 2.6 聚焦测试和 Checkstyle - -最终执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home \ - :common:test --tests org.tron.common.utils.BlockFileTest \ - :plugins:test --tests org.tron.plugins.DbBlockExportTest \ - :framework:test --tests org.tron.program.BlockReplayTest \ - :plugins:checkstyleMain \ - :plugins:checkstyleTest \ - :framework:checkstyleMain \ - :framework:checkstyleTest \ - :plugins:buildToolkitJar \ - :framework:buildFullNodeJar -``` - -结果：`BUILD SUCCESSFUL in 1m 21s`。 - -生命周期缺陷修复后执行： - -```bash -./gradlew -g /private/tmp/java-tron-gradle-home \ - :framework:test \ - --tests org.tron.program.BlockReplayTest \ - --tests org.tron.core.service.RewardViCalServiceLifecycleTest \ - :framework:checkstyleMain \ - :framework:checkstyleTest \ - :framework:buildFullNodeJar -``` - -结果：6 个聚焦测试全部通过，Checkstyle 和 FullNode fat jar 构建通过， -`BUILD SUCCESSFUL in 56s`。 - -### 2.7 Fat jar 验收 - -确认下列类实际进入构建产物： - -```text -plugins/build/libs/Toolkit.jar - org/tron/common/utils/BlockFile*.class - org/tron/plugins/DbBlock.class - org/tron/plugins/DbBlockExport.class - -framework/build/libs/FullNode.jar - org/tron/common/utils/BlockFile*.class - org/tron/program/BlockReplay*.class -``` - -实际运行以下帮助命令均成功： - -```bash -java -jar plugins/build/libs/Toolkit.jar db block export --help - -java -cp framework/build/libs/FullNode.jar \ - org.tron.program.BlockReplay --help -``` - -### 2.8 已通过的验收项 - -| 验收项 | 状态 | 证据边界 | -|---|---|---| -| 版本化文件格式 | 通过 | common round-trip 测试 | -| 流式处理 | 通过 | writer/reader 不保存完整区间 | -| CRC 损坏检测 | 通过 | 字节篡改测试 | -| 默认不覆盖 | 通过 | existing-file 测试 | -| 主链来源 | 通过 | Toolkit 按 block-index 取 ID | -| 真实 RocksDB 导出 | 通过 | plugins 临时库 CLI 测试 | -| verify-only | 通过 | direct + command-line 测试 | -| computed block ID | 通过 | replay 校验路径 | -| D0 head 防误用 | 通过 | mismatched-head 测试 | -| 同步业务入口 | 通过 | processBlock(block, true) captor | -| Consensus 启动门禁 | 通过 | replay 初始化显式调用 ConsensusService.start() | -| reward 关闭等待 | 通过 | 阻塞任务被中断，executor terminated，stop 可重复调用 | -| warmup/max-blocks | 通过 | 参数和统计逻辑测试 | -| Checkstyle | 通过 | plugins/framework main/test | -| fat jar | 通过 | 构建、jar 内容和 help 命令 | -| diff 格式 | 通过 | `git diff --check` | - -### 2.9 尚未完成的真实数据验收 - -本轮没有可用的真实停止 D1 和一次性 D0 副本，因此以下工作尚未执行： - -- 从真实主链 D1 导出数百或数千个历史区块； -- 用 FullNode fat jar 对该文件执行完整 verify； -- 在真实 D0 副本上完成 100～1,000 块 apply preflight； -- 验证包含真实交易、合约和 maintenance block 的状态执行； -- 对比最终 head、block ID、错误数和 Prometheus 指标； -- 执行 A/B、固定 peer 同步或 ABBA； -- 测量工具文件读取与实际区块处理之外的环境开销。 - -首次真实 apply 证明原实现不能接受“真实 D0 回放闭环已经完成”的结论。完成生命周期修复后， -当前可以接受的结论收缩为： - -> 固定区块文件、Toolkit 导出、同步入口和 D0 门禁已实现；Consensus 启动遗漏和 reward -> 后台线程关闭竞态已完成代码修复及聚焦测试，但必须再次在真实 D0 副本上 apply，才能恢复 -> “真实回放闭环通过”的结论。 - -当前不能接受的结论是： - -> 工具已经在真实历史窗口证明某个数据库方案更快，或离线结果可以替代真实 -> P2P 同步结果。 - -### 2.10 下一步真实验收 - -1. 正常停止一个覆盖目标高度的 D1； -2. 导出包含普通区块、合约密集区块和 maintenance block 的固定窗口； -3. 执行 verify-only 并保存文件 hash、范围和大小； -4. 从同一 D0 生成 A、B 一次性副本； -5. 两边先执行 100～1,000 块 `--max-blocks` preflight； -6. preflight 通过后执行固定 warmup 和 measurement 窗口； -7. 对齐 DB metrics、业务 metrics、CPU、GC 和磁盘指标； -8. 只有出现明显差异并确定 Candidate 后，再补反向轮次形成 ABBA； -9. 最后用固定单 peer 的真实同步确认端到端方向。 diff --git a/.dev_ops/bench_db/README.md b/.dev_ops/bench_db/README.md deleted file mode 100644 index b50c80eedff..00000000000 --- a/.dev_ops/bench_db/README.md +++ /dev/null @@ -1,401 +0,0 @@ -# java-tron 数据库性能快速校验方案 - -## 目标 - -通过处理一段固定的真实区块，快速判断数据库实现或数据库配置变化是否带来 -可重复的性能收益，同时区分： - -- 数据库内部行为变化； -- java-tron 区块及交易处理性能变化； -- 网络、缓存、JIT、GC、SSD 和后台任务造成的实验噪声。 - -本方案采用两层验证：先做低噪声的离线固定区块回放，再做固定数据源的真实 -P2P 同步。离线回放用于筛选和归因，真实同步用于确认端到端收益；两者不能 -互相替代。 - -```text -离线固定区块回放 - └─ 排除网络，快速定位 DB 差异 - │ - ▼ -固定数据源真实同步 - └─ 验证网络、队列、线程调度和业务执行叠加后的收益 -``` - -## 实验对象和控制变量 - -定义： - -- `A`：Baseline，例如当前数据库配置； -- `B`：Candidate，例如修改后的数据库配置； -- `D0`：正常关闭、最新高度为 `H` 的同一份基础数据库； -- `D1`：固定且停止增长的数据源，至少包含测试结束高度的全部区块； -- `N_warmup`：预热区块数； -- `N_measure`：正式测量区块数。 - -A、B 必须满足： - -- 从同一个代码基线构建，只保留待验证的数据库变量； -- 包含完全相同的指标代码，并使用相同的指标开关和抓取周期； -- 每轮都从 D0 的全新副本开始，不能复用上一轮已经同步过的数据库； -- 使用相同 JDK、JVM 参数、节点配置、磁盘、CPU 配额和数据源； -- 处理相同的固定高度区间； -- 测量期间关闭非必要 API 流量、事件插件和其他后台任务，或确保各轮负载一致。 - -先用 A 同步约 1,000 个区块标定速度，然后按固定时间预算换算区块数： - -```text -N_warmup = ceil(A 的标定 blocks/s × 300s) -N_measure = ceil(A 的标定 blocks/s × 1200s) -N_total = N_warmup + N_measure -``` - -正式比较区间固定为： - -```text -[H + N_warmup + 1, H + N_total] -``` - -不能让 A、B 各自运行固定时长后比较处理块数，因为两组可能处理了不同的区块 -内容，交易数量、合约类型和 maintenance block 都会成为混杂变量。 - -测试区间应尽量同时包含： - -- 普通转账和高交易数区块； -- 智能合约读写密集区块； -- 至少一个 maintenance block； -- 足以触发热点数据库 flush/compaction 的写入量。 - -如果测试期间没有产生新的 SST 或 compaction，只能评价立即生效的 cache/index -策略，不能据此判断 Bloom、block size、target file size 等新 SST 属性的收益。 - -## A/B、ABBA 和运行顺序 - -### 复杂测试前的低成本自检 - -不要一开始就进入 ABBA、随机交叉或火焰图分析。先对测试链路做一次短窗口 -preflight，目的只是尽早发现实验环境和操作错误，防止在复杂测试中继续衍生 -问题或把测试系统问题误诊为数据库问题。 - -建议 A、B 各从 D0 新副本回放或同步相同的 100～1,000 个区块，并检查： - -- 构建 commit、配置 diff 和数据库变量符合预期； -- D0 的高度和 block ID 一致，D1 覆盖目标区间且保持停止增长； -- 每轮能够从 D0 干净启动、到达预定高度并正常关闭； -- 最终高度、block ID 和错误数符合预期； -- Prometheus/Grafana 能采到所需标签，采样时间和区块窗口可以对齐； -- DB 副本、JVM、日志和指标目录在轮次之间确实完成重置； -- 没有端口冲突、额外 peer、后台 API 流量或磁盘空间不足。 - -preflight 通过只表示“测试链路可以继续”，不证明数据库正确性完备，也不产生 -正式性能结论。发现问题时应先修正测试链路并重新自检，不要带着已知异常扩大 -到 ABBA 或 profile 阶段。 - -### 单次 A/B 是什么 - -最小实验只运行： - -```text -A1 → B1 -``` - -它可以快速筛选明显差异，但版本变量和运行顺序完全绑定：A 永远先跑，B 永远 -后跑。因此以下现象都可能被误判为 B 的效果： - -- JIT、操作系统 Page Cache 或磁盘缓存逐渐变热，导致后跑的 B 更快； -- SSD 垃圾回收、温度、compaction backlog 累积，导致后跑的 B 更慢； -- 同机其他任务或硬件频率随时间变化； -- 第一次启动特有的依赖加载、类加载和文件系统元数据开销。 - -所以单次 A/B 适合快速淘汰明显无效方案，不适合直接形成最终结论。 - -### ABBA 是什么 - -ABBA 仍然是 A/B 测试，只是把同一个 A/B 重复四轮，并采用对称顺序： - -```text -A1 → B1 → B2 → A2 -``` - -每一轮都必须重置为 D0 的新副本并启动新 JVM。最终分别聚合 A1/A2 和 B1/B2， -不能把 B2 接着 B1 的数据库继续运行。 - -它主要抵消随时间近似线性的漂移。假设四轮位于时间点 1、2、3、4： - -```text -A 的平均时间位置 = (1 + 4) / 2 = 2.5 -B 的平均时间位置 = (2 + 3) / 2 = 2.5 -``` - -因此 A、B 不再分别绑定“早跑”和“晚跑”。ABBA 不能消除所有噪声，但比一次 -`A→B` 更容易识别顺序效应，成本约为后者的两倍。 - -ABBA 不作为默认起步动作。应先完成 preflight 和一次简单的 `A1→B1`；只有 -出现值得解释的明显差异、Candidate 方案基本确定并准备形成可信结论时，才补 -`B2→A2` 形成 ABBA，用来排除顺序和环境漂移干扰。初筛没有差异或 Candidate -在测量前失败时，没有必要直接增加 ABBA 的复杂度。 - -也可以使用方向相反但同样对称的： - -```text -B1 → A1 → A2 → B2 # BAAB -``` - -在多天或多台机器重复实验时，可以让一半使用 ABBA，另一半使用 BAAB。 - -### 其他可选设计 - -| 设计 | 顺序示例 | 用途 | 局限 | -|---|---|---|---| -| A/A | `A1 → A2` | 测量实验自身噪声，校验脚本和重置是否可靠 | 不比较 Candidate | -| 单次 A/B | `A1 → B1` | 最快初筛 | 无法分离版本和顺序效应 | -| 反向复测 | `A1 → B1 → B2 → A2` | 即 ABBA，适合单机快速确认 | 仍可能受非线性漂移影响 | -| 成对交叉 | 第一天 `A→B`，第二天 `B→A` | 跨天抵消顺序影响 | 环境跨天变化可能较大 | -| 随机交叉 | 随机排列多个 A/B 新副本 | 轮次较多时更稳健，可做置信区间 | 成本更高，必须提前冻结随机顺序 | -| 并行 A/B | 两台相同机器同时运行 A、B | 抵消共同时间变化 | 机器差异会替代顺序成为混杂变量；同机并行会争抢资源 | -| Latin square | A/B/C 使用 `ABC`、`BCA`、`CAB` | 比较三个以上候选 | 设计和分析更复杂 | - -建议执行顺序： - -1. 先做 100～1,000 块的 preflight，确认测试链路没有制造二次问题； -2. 运行一次简单的 A1→B1，快速筛选 Candidate； -3. 出现明显差异并基本确定方案后，再补 B2→A2 形成 ABBA； -4. 需要量化自然误差时补 A/A；收益接近噪声时再增加随机交叉轮次； -5. 只有差异来源仍不清楚时才进入 Arthas/async-profiler 分析。 - -候选改善幅度至少应大于： - -```text -max(5%, 2 × A/A 或重复轮次的变异系数) -``` - -同时要求吞吐、CPU/block、磁盘写入/block 和关键 p95/p99 中没有不可接受的 -反向退化。5% 是快速实验的初始门槛，不是固定的发布标准；最终应根据 A/A -测得的噪声调整。 - -## 第一层：离线固定区块回放 - -### 定位 - -不要实现成普通的短生命周期 JUnit 单元测试。更合适的是可重复启动的集成性能 -基准：使用真实 D0、真实历史区块和生产处理链路，只移除 P2P 获取过程。 - -```text -领先数据库 D1 - block-index → block bytes - │ - ▼ 导出 length-prefixed protobuf 文件 - blocks-H+1-to-H+N.dat - │ - ▼ -D0 新副本 → 启动真实 Manager/Consensus/VM/DB - │ - ▼ -TronNetDelegate.processBlock(block, true) - │ - ▼ -校验高度、block ID、错误数并输出性能指标 -``` - -java-tron 已有的可复用入口包括： - -- `BlockStore#getLimitNumber`：读取连续区块； -- `BlockCapsule(byte[])`：反序列化真实区块； -- `TronNetDelegate#processBlock(block, true)`：走同步区块处理路径； -- `Manager#pushBlock`：进入共识、签名、交易、VM、maintenance、revoking DB - 和持久化处理。 - -不要使用 `pushVerifiedBlock` 作为回放入口，因为它会设置 -`generatedByMyself=true`，从而绕过部分针对外部区块的 Merkle、共识和交易签名 -校验。 - -离线回放保留真实交易组合、状态依赖、VM、共识、maintenance、数据库读写及 -compaction，能够回答： - -> 相同状态、相同区块和相同 JVM 条件下，数据库方案本身是否更快？ - -它不能完整反映 P2P 请求、网络抖动、peer 队列、网络反压以及同步线程和网络 -线程之间的竞争，因此不能单独证明真实节点同步会获得相同比例的收益。 - -### 固定区块文件命令 - -构建导出和回放工具： - -```bash -./gradlew :plugins:buildToolkitJar :framework:buildFullNodeJar -``` - -从已经停止的 D1 节点导出闭区间 `[H+1, H+N]`。`-d` 指向实际数据库目录， -即其中直接包含 `block` 和 `block-index` 的目录： - -```bash -java -jar plugins/build/libs/Toolkit.jar db block export \ - -d /data/D1/database \ - --start 68000001 \ - --end 68010000 \ - -o /data/block-files/68000001-68010000.dat -``` - -导出文件包含版本头、起止高度、记录数、每块的源数据库 block ID、原始 -protobuf 和 CRC32。导出时检查高度及父块连续性，默认拒绝覆盖已有文件；确实 -需要替换时显式使用 `--overwrite`。不要对正在运行的节点执行导出，跨库读取 -无法为在线 `block-index` 和 `block` 提供一致快照。 - -修改 D0 前先做只读文件校验： - -```bash -java -cp framework/build/libs/FullNode.jar org.tron.program.BlockReplay \ - --input /data/block-files/68000001-68010000.dat \ - --config /data/config.conf -``` - -从 D0 的一次性副本执行离线回放。这里 `-d` 指向节点 output directory，而不是 -其内部的 `database` 子目录： - -```bash -java -cp framework/build/libs/FullNode.jar org.tron.program.BlockReplay \ - --input /data/block-files/68000001-68010000.dat \ - --config /data/config.conf \ - --output-directory /data/A1 \ - --apply \ - --warmup-blocks 2000 -``` - -安全和结果边界： - -- 不带 `--apply` 时只验证文件，不打开和修改 D0； -- `--apply` 要求 output directory 已存在，并强制关闭 P2P； -- 文件首块必须是 `D0 head + 1`，其 parent ID 必须等于 D0 head ID； -- 每块通过 `TronNetDelegate.processBlock(block, true)` 进入真实同步处理路径； -- 每次应用后校验 D0 head 高度和 block ID，失败立即停止； -- `--max-blocks` 可用于 100～1,000 块 preflight；`--warmup-blocks` 只从时间统计 - 中排除前段区块，不跳过实际应用； -- 输出的 `elapsed_ms` 和 `blocks_per_second` 只计逐块处理窗口，不包含 Spring、 - 数据库打开及文件预检查时间；正式 A/B 的每轮必须使用 D0 的全新副本。 - -## 第二层：固定数据源真实同步 - -使用停止增长的固定 D1 作为单一 peer，让 A、B 从各自的 D0 副本同步相同区间。 -数据源和被测节点之间应使用稳定的本机或局域网连接，并避免连接其他 peer。 - -真实同步保留网络、消息处理、队列、锁竞争和反压，是最终业务判断依据。必须 -同时观察 block fetch/receive 指标：如果网络已经成为瓶颈，blocks/s 相同不能 -证明两个数据库性能相同。 - -## 指标与证据层级 - -### L0：正确性和配置准入 - -- A、B 起始高度和 block ID 相同； -- 最终高度和 block ID 相同； -- 无 block validation、fork、DB error 或异常退出； -- `OPTIONS-*`/LOG 证明 RocksDB 参数实际生效； -- 记录测试窗口内是否发生 flush、compaction 和 write stall。 - -`OPTIONS-*` 只能证明配置生效，不能证明配置带来性能收益。 - -### L1：数据库行为 - -`feat/db_metric` 当前提供： - -- `tron:db_operate_latency_seconds{type,db,op}`； -- `tron:db_operate_bytes{type,db,op}`； -- `tron:db_event{type,db,event}`，当前主要是 LevelDB 事件； -- `tron:db_memory_bytes{type,db,property}`； -- `tron:db_rocksdb_property{type,db,property}`：compaction/flush/write stall - 即时状态； -- `tron:db_rocksdb_ticker_total{type,db,ticker}`：cache、Bloom、level hit、 - bytes、flush/compaction 和 stall 累计量； -- 原有 `tron:db_size_bytes`、`tron:db_sst_level`。 - -重点计算： - -- 每个 DB、每种操作的 ops/s、平均值、p95 和 p99； -- DB payload bytes/s 和每块 payload bytes； -- 每块 DB 操作次数及 DB latency sum； -- SST/L0 文件数、memtable、block cache、index/filter 内存变化； -- RocksDB cache hit、Bloom useful、compaction bytes/time 和 write stall； -- 系统磁盘 read/write bytes、IOPS、await 和 util。 - -注意： - -- `db_operate_bytes` 是 value payload，不等于实际 WAL/SST/compaction 写盘量； -- 共享 block cache 下，不应在未确认语义前直接累加不同 DB 标签的 cache usage； -- `DbStatService` 按 `statIntervalSeconds` 采集，快速实验建议 10～30 秒； -- RocksDB Statistics 的 StatsLevel 必须在 A、B 中保持一致。 - -### L2：业务性能 - -主要指标： - -- `tron:header_height`：计算 blocks/s； -- `tron:block_process_latency_seconds{sync="true"}`； -- `tron:block_push_latency_seconds`； -- `tron:process_transaction_latency_seconds{type="block"}`； -- `tron:block_transaction_count`； -- CPU、RSS、GC pause、磁盘吞吐和 await。 - -建议统一换算为： - -- elapsed/block； -- CPU seconds/block； -- disk read/write bytes/block； -- DB operations/block； -- DB latency sum/block。 - -数据库内部指标用于解释原因，最终收益应由固定区块窗口内的业务吞吐、延迟和 -资源消耗共同判断。 - -## `feat/db_metric` 合并和观测开销 - -`feat/db_metric` 已合并本次测试采用的最新 `upstream/develop`。开始正式实验前 -仍需记录实际 commit，并确认 A、B 从同一个 commit 构建，只改变目标 DB 变量。 - -DB 指标独立于全局 Prometheus 默认关闭。基准节点使用： - -```hocon -node.metrics.prometheus { - enable = true - database { - enable = true - statIntervalSeconds = 30 - } -} -``` - -每次 DB `get/put/delete/batch` 上更新 Histogram 可能影响微秒级热点路径。因此 -在正式数据库 A/B 前增加一个指标开销对照： - -```text -相同代码 + 相同 DB 配置 + 指标关闭 -相同代码 + 相同 DB 配置 + 指标开启 -``` - -当前实现已预绑定固定的 engine/db/op label，避免每次 DB 操作重复查找标签， -但计时和 Histogram observe 的成本仍然存在。如果开启指标后的变化已接近数据库 -Candidate 的收益，只能把指标开启结果用于归因，不能外推为无监控时的绝对性能。 - -## Arthas / async-profiler - -Arthas 或 async-profiler 用于差异出现后的定位，不建议作为每轮默认采集项。 - -- 优先使用低开销采样生成 CPU 或 wall-clock 火焰图； -- 重点观察 `Manager.processBlock → transaction/VM → Store → RocksDB`； -- blocks/s 下降但 DB 指标无明显变化时，检查锁等待、GC、签名验证和 - maintenance； -- 避免对每次 DB 调用做大范围 `trace/watch`，探针开销会污染热点路径。 - -## 推荐的快速执行阶段 - -| 阶段 | 内容 | 预计时间 | 结论边界 | -|---|---|---:|---| -| 0 | A/B 各 100～1,000 块 preflight + 标定 | 10～30 分钟 | 只确认构建、数据、重置、指标和执行链路可用 | -| 1 | 离线回放 A1→B1 | 30～60 分钟 | 快速筛选 DB 自身差异 | -| 2 | 固定 peer 同步 A1→B1 | 约 1～1.5 小时 | 初步端到端判断 | -| 3 | 明显差异且方案确定后补 B2→A2 | 总计约 2～3 小时 | 形成 ABBA，排除顺序漂移并确认重复性 | -| 4 | 对异常轮次采样 profile | 按需 | 定位差异来源 | - -preflight 是进入复杂测试前的低成本自检，不是全数据库证明或性能结论。对于 -准备采用的 Candidate，只有离线回放和真实同步方向一致、ABBA 结果可重复、 -正确性门禁通过，才能认定其具有可信收益。若 Candidate 在预热阶段失败或未 -进入正式测量窗口，结论只能记录为“测量前失败”，不能计算成性能退化百分比。 diff --git a/.gitignore b/.gitignore index 3917bb44679..c5640cca858 100644 --- a/.gitignore +++ b/.gitignore @@ -57,3 +57,6 @@ Wallet /framework/propPath .cache + +# Local development and benchmark operations records +/.dev_ops/ diff --git a/chainbase/src/main/java/org/tron/common/storage/metric/DbOperationMetrics.java b/chainbase/src/main/java/org/tron/common/storage/metric/DbOperationMetrics.java index 5b3039306a2..cf8d3065600 100644 --- a/chainbase/src/main/java/org/tron/common/storage/metric/DbOperationMetrics.java +++ b/chainbase/src/main/java/org/tron/common/storage/metric/DbOperationMetrics.java @@ -1,5 +1,6 @@ package org.tron.common.storage.metric; +import io.prometheus.client.Counter; import io.prometheus.client.Histogram; import org.tron.common.prometheus.MetricKeys; import org.tron.common.prometheus.Metrics; @@ -20,6 +21,8 @@ public final class DbOperationMetrics { private final Histogram.Child getBytes; private final Histogram.Child putBytes; private final Histogram.Child batchBytes; + private final Counter.Child getHit; + private final Counter.Child getMiss; private DbOperationMetrics(String engine, String database) { getLatency = child(MetricKeys.Histogram.DB_OPERATE_LATENCY, engine, database, "get"); @@ -29,6 +32,8 @@ private DbOperationMetrics(String engine, String database) { getBytes = child(MetricKeys.Histogram.DB_OPERATE_BYTES, engine, database, "get"); putBytes = child(MetricKeys.Histogram.DB_OPERATE_BYTES, engine, database, "put"); batchBytes = child(MetricKeys.Histogram.DB_OPERATE_BYTES, engine, database, "batch"); + getHit = counterChild(MetricKeys.Counter.DB_GET, engine, database, "hit"); + getMiss = counterChild(MetricKeys.Counter.DB_GET, engine, database, "miss"); } public static DbOperationMetrics create(String engine, String database) { @@ -59,6 +64,13 @@ public void observeGetBytes(long bytes) { observe(getBytes, bytes); } + public void observeGetOutcome(boolean hit) { + Counter.Child child = hit ? getHit : getMiss; + if (child != null) { + child.inc(); + } + } + public void observePutBytes(long bytes) { observe(putBytes, bytes); } @@ -71,6 +83,11 @@ private static Histogram.Child child(String key, String engine, String database, return Metrics.databaseHistogramChild(key, engine, database, op); } + private static Counter.Child counterChild(String key, String engine, String database, + String outcome) { + return Metrics.databaseCounterChild(key, engine, database, outcome); + } + private static Histogram.Timer timer(Histogram.Child child) { return child == null ? null : child.startTimer(); } diff --git a/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbBlockCacheTrace.java b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbBlockCacheTrace.java new file mode 100644 index 00000000000..384590aef75 --- /dev/null +++ b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbBlockCacheTrace.java @@ -0,0 +1,121 @@ +package org.tron.common.storage.rocksdb; + +import java.io.IOException; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import lombok.extern.slf4j.Slf4j; +import org.rocksdb.RocksDB; +import org.tron.common.setting.RocksDbSettings; + +/** Optional per-level block-cache trace backed by a version-matched RocksDB JNI bridge. */ +@Slf4j(topic = "DB") +final class RocksDbBlockCacheTrace implements AutoCloseable { + + private static final Object LOAD_LOCK = new Object(); + private static String loadedLibrary; + + private final RocksDB database; + private final String databaseName; + private boolean started; + + private RocksDbBlockCacheTrace(RocksDB database, String databaseName) { + this.database = database; + this.databaseName = databaseName; + } + + static RocksDbBlockCacheTrace create(RocksDB database, String databaseName, Path databasePath) { + RocksDbSettings settings = RocksDbSettings.getSettings(); + if (!settings.shouldTraceBlockCache(databaseName)) { + return null; + } + requireRocksDbVersion(); + loadLibrary(settings.getBlockCacheTraceNativeLibrary()); + Path outputDirectory = Paths.get(settings.getBlockCacheTraceOutputDirectory()) + .toAbsolutePath().normalize(); + databasePath = databasePath.toAbsolutePath().normalize(); + Path databaseRoot = databasePath.getParent(); + if (databaseRoot == null || outputDirectory.startsWith(databaseRoot)) { + throw new IllegalArgumentException("Block cache trace output must be outside database root: " + + outputDirectory); + } + try { + Files.createDirectories(outputDirectory); + } catch (IOException e) { + throw new IllegalStateException("Unable to create block cache trace directory", e); + } + Path traceFile = outputDirectory.resolve(safeFileName(databaseName) + ".csv"); + RocksDbBlockCacheTrace trace = new RocksDbBlockCacheTrace(database, databaseName); + String error = startTrace(nativeHandle(database), traceFile.toString(), + settings.getBlockCacheTraceSampleOneIn(), settings.getBlockCacheTraceMaxBytesPerDb()); + if (error != null) { + throw new IllegalStateException("Unable to start block cache trace for " + databaseName + + ": " + error); + } + trace.started = true; + logger.info("Started RocksDB block cache trace db={}, sampleOneIn={}, maxBytes={}, path={}", + databaseName, settings.getBlockCacheTraceSampleOneIn(), + settings.getBlockCacheTraceMaxBytesPerDb(), traceFile); + return trace; + } + + private static void requireRocksDbVersion() { + try { + Object version = RocksDB.class.getMethod("rocksdbVersion").invoke(null); + Method major = version.getClass().getMethod("getMajor"); + Method minor = version.getClass().getMethod("getMinor"); + int majorValue = ((Number) major.invoke(version)).intValue(); + int minorValue = ((Number) minor.invoke(version)).intValue(); + if (majorValue != 9 || minorValue != 7) { + throw new IllegalStateException("Block cache trace bridge requires RocksDB 9.7.x, found " + + majorValue + "." + minorValue); + } + } catch (ReflectiveOperationException e) { + throw new IllegalStateException("Unable to verify RocksDB version for block cache trace", e); + } + } + + private static long nativeHandle(RocksDB database) { + try { + Method method = database.getClass().getMethod("getNativeHandle"); + return ((Number) method.invoke(database)).longValue(); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException("RocksDB JNI does not expose getNativeHandle", e); + } + } + + private static void loadLibrary(String library) { + String normalized = Paths.get(library).toAbsolutePath().normalize().toString(); + synchronized (LOAD_LOCK) { + if (loadedLibrary == null) { + System.load(normalized); + loadedLibrary = normalized; + } else if (!loadedLibrary.equals(normalized)) { + throw new IllegalStateException("Block cache trace bridge already loaded from " + + loadedLibrary); + } + } + } + + private static String safeFileName(String databaseName) { + return databaseName.replaceAll("[^A-Za-z0-9._-]", "_"); + } + + @Override + public void close() { + if (!started) { + return; + } + String error = endTrace(nativeHandle(database)); + started = false; + if (error != null) { + logger.error("Unable to end RocksDB block cache trace for {}: {}", databaseName, error); + } + } + + private static native String startTrace(long databaseHandle, String outputPath, + long sampleOneIn, long maxBytes); + + private static native String endTrace(long databaseHandle); +} diff --git a/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImpl.java b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImpl.java index d49a64faebd..536c03a23e3 100644 --- a/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImpl.java +++ b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImpl.java @@ -63,6 +63,8 @@ public class RocksDbDataSourceImpl extends DbStat implements DbSourceInter tickerSnapshots = new EnumMap<>(TickerType.class); public RocksDbDataSourceImpl(String parentPath, String name) { @@ -91,6 +93,14 @@ public void closeDB() { if (!isAlive()) { return; } + if (blockCacheTrace != null) { + try { + blockCacheTrace.close(); + } catch (RuntimeException e) { + logger.error("Failed to close block cache trace for {}", dataBaseName, e); + } + blockCacheTrace = null; + } database.close(); if (this.statistics != null) { this.statistics.close(); @@ -218,6 +228,14 @@ private void initDB() { this.options = RocksDbSettings.getOptionsByDbName(dataBaseName); tickerSnapshots.clear(); database = RocksDB.open(this.options, dbPath.toString()); + getPerfContext = RocksDbGetPerfContext.create(database, dataBaseName); + try { + blockCacheTrace = RocksDbBlockCacheTrace.create(database, dataBaseName, dbPath); + } catch (RuntimeException e) { + database.close(); + database = null; + throw e; + } statistics = this.options.statistics(); } catch (RocksDBException e) { if (Objects.equals(e.getStatus().getCode(), Status.Code.Corruption)) { @@ -269,7 +287,7 @@ public byte[] getData(byte[] key) { try (Histogram.Timer timer = dbOperationMetrics.startGet()) { throwIfNotAlive(); checkArgNotNull(key, "key"); - value = database.get(key); + value = getPerfContext == null ? database.get(key) : getPerfContext.get(key); } catch (RocksDBException e) { throw new RuntimeException(dataBaseName, e); } finally { @@ -278,6 +296,7 @@ public byte[] getData(byte[] key) { if (value != null) { dbOperationMetrics.observeGetBytes(value.length); } + dbOperationMetrics.observeGetOutcome(value != null); return value; } @@ -613,6 +632,14 @@ public String getName() { TickerType.NUMBER_KEYS_WRITTEN, TickerType.BYTES_READ, TickerType.BYTES_WRITTEN, + TickerType.NUMBER_DB_SEEK, + TickerType.NUMBER_DB_NEXT, + TickerType.NUMBER_DB_PREV, + TickerType.NUMBER_DB_SEEK_FOUND, + TickerType.NUMBER_DB_NEXT_FOUND, + TickerType.NUMBER_DB_PREV_FOUND, + TickerType.ITER_BYTES_READ, + TickerType.NO_FILE_OPENS, TickerType.COMPACT_READ_BYTES, TickerType.COMPACT_WRITE_BYTES, TickerType.FLUSH_WRITE_BYTES, diff --git a/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbGetPerfContext.java b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbGetPerfContext.java new file mode 100644 index 00000000000..b3ae0b88f71 --- /dev/null +++ b/chainbase/src/main/java/org/tron/common/storage/rocksdb/RocksDbGetPerfContext.java @@ -0,0 +1,178 @@ +package org.tron.common.storage.rocksdb; + +import io.prometheus.client.Counter; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.util.concurrent.ThreadLocalRandom; +import lombok.extern.slf4j.Slf4j; +import org.rocksdb.RocksDB; +import org.rocksdb.RocksDBException; +import org.tron.common.prometheus.MetricKeys; +import org.tron.common.prometheus.Metrics; +import org.tron.common.setting.RocksDbSettings; + +/** Sampled RocksDB PerfContext collection that remains compatible with the legacy JNI. */ +@Slf4j(topic = "DB") +final class RocksDbGetPerfContext { + + private static final String[] METRICS = { + "sampled_get", "hit", "miss", "block_read_count", "block_read_bytes", + "block_read_nanos", "index_block_read_count", "filter_block_read_count", + "block_cache_hit_count", "from_memtable_count", "user_key_comparison_count", + "read_bytes" + }; + private static final PerfApi PERF_API = PerfApi.load(); + + private final RocksDB database; + private final int sampleOneIn; + private final Counter.Child[] counters; + + private RocksDbGetPerfContext(RocksDB database, String databaseName, int sampleOneIn) { + this.database = database; + this.sampleOneIn = sampleOneIn; + this.counters = new Counter.Child[METRICS.length]; + for (int i = 0; i < METRICS.length; i++) { + counters[i] = Metrics.databaseCounterChild(MetricKeys.Counter.DB_GET_PERF, + "ROCKSDB", databaseName, METRICS[i]); + } + } + + static RocksDbGetPerfContext create(RocksDB database, String databaseName) { + RocksDbSettings settings = RocksDbSettings.getSettings(); + if (!Metrics.databaseEnabled() || !settings.shouldSamplePerfContext(databaseName)) { + return null; + } + if (!PERF_API.supported) { + logger.warn("RocksDB JNI does not expose PerfContext; skip sampling for {}", databaseName); + return null; + } + return new RocksDbGetPerfContext(database, databaseName, + settings.getPerfContextSampleOneIn()); + } + + byte[] get(byte[] key) throws RocksDBException { + if (ThreadLocalRandom.current().nextInt(sampleOneIn) != 0) { + return database.get(key); + } + + Object context = null; + boolean enabled = false; + try { + PERF_API.setPerfLevel.invoke(database, PERF_API.enableTime); + enabled = true; + context = PERF_API.getPerfContext.invoke(database); + PERF_API.reset.invoke(context); + } catch (InvocationTargetException e) { + logger.warn("Unable to enable RocksDB PerfContext", e.getCause()); + } catch (ReflectiveOperationException | RuntimeException e) { + logger.warn("Unable to enable RocksDB PerfContext", e); + } + if (context == null) { + if (enabled) { + disablePerfContext(); + } + return database.get(key); + } + + try { + byte[] value = database.get(key); + try { + record(context, value != null); + } catch (ReflectiveOperationException | RuntimeException e) { + logger.warn("Unable to record RocksDB PerfContext", e); + } + return value; + } finally { + disablePerfContext(); + closePerfContext(context); + } + } + + private void record(Object context, boolean hit) throws ReflectiveOperationException { + add(0, 1); + add(hit ? 1 : 2, 1); + add(3, PERF_API.blockReadCount.invoke(context)); + add(4, PERF_API.blockReadBytes.invoke(context)); + add(5, PERF_API.blockReadNanos.invoke(context)); + add(6, PERF_API.indexBlockReadCount.invoke(context)); + add(7, PERF_API.filterBlockReadCount.invoke(context)); + add(8, PERF_API.blockCacheHitCount.invoke(context)); + add(9, PERF_API.fromMemtableCount.invoke(context)); + add(10, PERF_API.userKeyComparisonCount.invoke(context)); + add(11, PERF_API.readBytes.invoke(context)); + } + + private void add(int index, Object value) { + long amount = ((Number) value).longValue(); + if (amount > 0) { + counters[index].inc(amount); + } + } + + private void disablePerfContext() { + try { + PERF_API.setPerfLevel.invoke(database, PERF_API.disable); + } catch (ReflectiveOperationException | RuntimeException e) { + logger.warn("Unable to disable RocksDB PerfContext", e); + } + } + + private void closePerfContext(Object context) { + try { + PERF_API.close.invoke(context); + } catch (ReflectiveOperationException | RuntimeException e) { + logger.warn("Unable to close RocksDB PerfContext", e); + } + } + + private static final class PerfApi { + private final boolean supported; + private Object enableTime; + private Object disable; + private Method setPerfLevel; + private Method getPerfContext; + private Method reset; + private Method close; + private Method blockReadCount; + private Method blockReadBytes; + private Method blockReadNanos; + private Method indexBlockReadCount; + private Method filterBlockReadCount; + private Method blockCacheHitCount; + private Method fromMemtableCount; + private Method userKeyComparisonCount; + private Method readBytes; + + private PerfApi(boolean supported) { + this.supported = supported; + } + + @SuppressWarnings({"unchecked", "rawtypes"}) + private static PerfApi load() { + try { + Class level = Class.forName("org.rocksdb.PerfLevel"); + Class context = Class.forName("org.rocksdb.PerfContext"); + PerfApi api = new PerfApi(true); + api.enableTime = Enum.valueOf((Class) level, + "ENABLE_TIME_EXCEPT_FOR_MUTEX"); + api.disable = Enum.valueOf((Class) level, "DISABLE"); + api.setPerfLevel = RocksDB.class.getMethod("setPerfLevel", level); + api.getPerfContext = RocksDB.class.getMethod("getPerfContext"); + api.reset = context.getMethod("reset"); + api.close = context.getMethod("close"); + api.blockReadCount = context.getMethod("getBlockReadCount"); + api.blockReadBytes = context.getMethod("getBlockReadByte"); + api.blockReadNanos = context.getMethod("getBlockReadTime"); + api.indexBlockReadCount = context.getMethod("getIndexBlockReadCount"); + api.filterBlockReadCount = context.getMethod("getFilterBlockReadCount"); + api.blockCacheHitCount = context.getMethod("getBlockCacheHitCount"); + api.fromMemtableCount = context.getMethod("getFromMemtableCount"); + api.userKeyComparisonCount = context.getMethod("getUserKeyComparisonCount"); + api.readBytes = context.getMethod("getReadBytes"); + return api; + } catch (ReflectiveOperationException | LinkageError e) { + return new PerfApi(false); + } + } + } +} diff --git a/chainbase/src/main/native/rocksdb_trace/CMakeLists.txt b/chainbase/src/main/native/rocksdb_trace/CMakeLists.txt new file mode 100644 index 00000000000..f7b3ace0a85 --- /dev/null +++ b/chainbase/src/main/native/rocksdb_trace/CMakeLists.txt @@ -0,0 +1,36 @@ +cmake_minimum_required(VERSION 3.16) +project(java_tron_rocksdb_trace LANGUAGES CXX) + +if(NOT DEFINED ROCKSDB_SOURCE_DIR) + message(FATAL_ERROR "ROCKSDB_SOURCE_DIR must point to RocksDB v9.7.4 source") +endif() + +if(NOT DEFINED ENV{JAVA_HOME}) + message(FATAL_ERROR "JAVA_HOME must point to a JDK containing include/jni.h") +endif() + +set(JAVA_INCLUDE_DIR "$ENV{JAVA_HOME}/include") +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(JAVA_PLATFORM_INCLUDE_DIR "${JAVA_INCLUDE_DIR}/linux") +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + set(JAVA_PLATFORM_INCLUDE_DIR "${JAVA_INCLUDE_DIR}/darwin") +else() + message(FATAL_ERROR "Unsupported JNI platform: ${CMAKE_SYSTEM_NAME}") +endif() + +if(NOT EXISTS "${JAVA_INCLUDE_DIR}/jni.h") + message(FATAL_ERROR "jni.h not found under JAVA_HOME=$ENV{JAVA_HOME}") +endif() + +add_library(java_tron_rocksdb_trace SHARED rocksdb_trace_bridge.cc) +target_compile_features(java_tron_rocksdb_trace PRIVATE cxx_std_17) +target_compile_definitions(java_tron_rocksdb_trace PRIVATE _GLIBCXX_USE_CXX11_ABI=0) +target_include_directories(java_tron_rocksdb_trace PRIVATE + ${JAVA_INCLUDE_DIR} + ${JAVA_PLATFORM_INCLUDE_DIR} + ${ROCKSDB_SOURCE_DIR}/include) +target_compile_options(java_tron_rocksdb_trace PRIVATE -Wall -Wextra -Werror) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + target_link_libraries(java_tron_rocksdb_trace PRIVATE dl) + target_link_options(java_tron_rocksdb_trace PRIVATE -Wl,-Bsymbolic) +endif() diff --git a/chainbase/src/main/native/rocksdb_trace/README.md b/chainbase/src/main/native/rocksdb_trace/README.md new file mode 100644 index 00000000000..b17df135ca8 --- /dev/null +++ b/chainbase/src/main/native/rocksdb_trace/README.md @@ -0,0 +1,43 @@ +# RocksDB block-cache trace bridge + +This optional JNI library exposes RocksDB's native per-level block-cache trace to java-tron. +It is loaded only when `blockCacheTraceDbAllowList` is non-empty. + +The bridge is ABI-bound to RocksDB `v9.7.4` (`3c27a3dde0993210c5cc30d99717093f7537916f`). +The Java runtime also rejects versions other than `9.7.x` before loading it. + +Build on the target architecture: + +```bash +git clone --depth 1 --branch v9.7.4 https://github.com/facebook/rocksdb.git \ + /tmp/rocksdb-v9.7.4 +JAVA_HOME=/path/to/jdk cmake -S . -B build \ + -DROCKSDB_SOURCE_DIR=/tmp/rocksdb-v9.7.4 -DCMAKE_BUILD_TYPE=Release +JAVA_HOME=/path/to/jdk cmake --build build +ldd -r build/libjava_tron_rocksdb_trace.so +``` + +`ldd -r` must not report unresolved RocksDB symbols. The bridge resolves the version-specific +`DBImpl::StartBlockCacheTrace` and `DBImpl::EndBlockCacheTrace` symbols from the already loaded +rocksdbjni library; it does not link a second RocksDB engine into the JVM. + +The Maven ARM rocksdbjni artifact uses the legacy libstdc++ string ABI. CMake therefore fixes +`_GLIBCXX_USE_CXX11_ABI=0`; changing it corrupts `BlockCacheTraceRecord` field offsets even when +the RocksDB source tag matches. + +Example configuration: + +```hocon +storage.dbSettings { + blockCacheTraceSampleOneIn = 100 + blockCacheTraceDbAllowList = [account, account-asset, storage-row] + # Use ["*"] or ["all"] to trace every RocksDB store with the same implementation. + blockCacheTraceOutputDirectory = "/data/traces/run-001" + blockCacheTraceMaxBytesPerDb = 536870912 + blockCacheTraceNativeLibrary = "/data/tools/libjava_tron_rocksdb_trace.so" +} +``` + +Trace output must be outside the database directory. Each database has an independent bounded CSV +file. Get requests are sampled by `get_id`, so all level events belonging to a selected Get are +retained together. diff --git a/chainbase/src/main/native/rocksdb_trace/rocksdb_trace_bridge.cc b/chainbase/src/main/native/rocksdb_trace/rocksdb_trace_bridge.cc new file mode 100644 index 00000000000..0d496b86856 --- /dev/null +++ b/chainbase/src/main/native/rocksdb_trace/rocksdb_trace_bridge.cc @@ -0,0 +1,239 @@ +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "rocksdb/block_cache_trace_writer.h" +#include "rocksdb/db.h" + +// rocksdbjni keeps RocksDB symbols local to its JNI shared object. The bridge invokes DB trace +// methods through the public virtual interface, but Status construction/stringification used by +// this translation unit still needs local definitions. Keep these two ABI-matched v9.7 methods +// local to this library; no DB/cache/table implementation is linked into the process. +namespace rocksdb { +Status::Status(Code code, SubCode subcode, const Slice& message, const Slice& message2, + Severity severity) + : code_(code), subcode_(subcode), sev_(severity), retryable_(false), data_loss_(false), + scope_(0) { + size_t first = message.size(); + size_t second = message2.size(); + size_t size = first + (second == 0 ? 0 : second + 2); + char* state = new char[size + 1]; + std::memcpy(state, message.data(), first); + if (second != 0) { + state[first] = ':'; + state[first + 1] = ' '; + std::memcpy(state + first + 2, message2.data(), second); + } + state[size] = '\0'; + state_.reset(state); +} + +std::string Status::ToString() const { + if (ok()) { + return "OK"; + } + return std::string("RocksDB status code=") + std::to_string(static_cast(code())) + + " subcode=" + std::to_string(static_cast(subcode())) + + (getState() == nullptr ? "" : std::string(" message=") + getState()); +} +} // namespace rocksdb + +namespace { + +constexpr uint64_t kReservedGetId = 0; + +using StartBlockCacheTrace = rocksdb::Status (*)( + void*, const rocksdb::BlockCacheTraceOptions&, + std::unique_ptr&&); +using EndBlockCacheTrace = rocksdb::Status (*)(void*); + +struct RocksDbTraceFunctions { + StartBlockCacheTrace start = nullptr; + EndBlockCacheTrace end = nullptr; + std::string error; +}; + +int FindRocksDbJni(struct dl_phdr_info* info, size_t, void* data) { + if (info->dlpi_name != nullptr + && std::strstr(info->dlpi_name, "librocksdbjni") != nullptr) { + *static_cast(data) = info->dlpi_name; + return 1; + } + return 0; +} + +RocksDbTraceFunctions ResolveTraceFunctions() { + RocksDbTraceFunctions functions; + std::string path; + dl_iterate_phdr(FindRocksDbJni, &path); + if (path.empty()) { + functions.error = "loaded rocksdbjni library was not found"; + return functions; + } + void* handle = dlopen(path.c_str(), RTLD_NOW | RTLD_NOLOAD); + if (handle == nullptr) { + functions.error = std::string("unable to inspect rocksdbjni: ") + dlerror(); + return functions; + } + // Resolve the version-specific DBImpl methods. The public DB vtable layout can differ when + // rocksdbjni and this bridge are compiled with different RocksDB feature macros. + functions.start = reinterpret_cast(dlsym( + handle, + "_ZN7rocksdb6DBImpl20StartBlockCacheTraceERKNS_22BlockCacheTraceOptionsEOSt10unique_ptrINS_21BlockCacheTraceWriterESt14default_deleteIS5_EE")); + functions.end = reinterpret_cast( + dlsym(handle, "_ZN7rocksdb6DBImpl18EndBlockCacheTraceEv")); + if (functions.start == nullptr || functions.end == nullptr) { + functions.error = "RocksDB 9.7 DBImpl block-cache trace symbols were not found"; + } + return functions; +} + +const RocksDbTraceFunctions& TraceFunctions() { + static const RocksDbTraceFunctions functions = ResolveTraceFunctions(); + return functions; +} + +uint64_t Mix(uint64_t value) { + value ^= value >> 30; + value *= UINT64_C(0xbf58476d1ce4e5b9); + value ^= value >> 27; + value *= UINT64_C(0x94d049bb133111eb); + return value ^ (value >> 31); +} + +uint64_t HashSlice(const rocksdb::Slice& value) { + uint64_t hash = UINT64_C(1469598103934665603); + for (size_t i = 0; i < value.size(); ++i) { + hash ^= static_cast(value.data()[i]); + hash *= UINT64_C(1099511628211); + } + return Mix(hash); +} + +void WriteCsvString(std::ostream& output, const rocksdb::Slice& value) { + output.put('"'); + for (size_t i = 0; i < value.size(); ++i) { + char current = value.data()[i]; + if (current == '"') { + output.put('"'); + } + output.put(current); + } + output.put('"'); +} + +class CsvBlockCacheTraceWriter final : public rocksdb::BlockCacheTraceWriter { + public: + CsvBlockCacheTraceWriter(const std::string& path, uint64_t sample_one_in, + uint64_t max_bytes) + : output_(path, std::ios::out | std::ios::trunc), + sample_one_in_(sample_one_in), + max_bytes_(max_bytes) {} + + bool IsOpen() const { return output_.is_open(); } + + rocksdb::Status WriteHeader() override { + std::lock_guard lock(mutex_); + output_ << "timestamp_us,get_id,level,sst_file,caller,block_type,cache_hit," + "no_insert,key_exists,block_size,cf_id,cf_name\n"; + output_.flush(); + return rocksdb::Status(); + } + + rocksdb::Status WriteBlockAccess(const rocksdb::BlockCacheTraceRecord& record, + const rocksdb::Slice& block_key, + const rocksdb::Slice& cf_name, + const rocksdb::Slice&) override { + uint64_t sample_key = record.get_id == kReservedGetId + ? HashSlice(block_key) : Mix(record.get_id); + if (sample_key % sample_one_in_ != 0) { + return rocksdb::Status(); + } + std::lock_guard lock(mutex_); + if (full_) { + return rocksdb::Status(); + } + if (static_cast(output_.tellp()) >= max_bytes_) { + full_ = true; + output_.flush(); + return rocksdb::Status(); + } + output_ << record.access_timestamp << ',' << record.get_id << ',' << record.level << ',' + << record.sst_fd_number << ',' << static_cast(record.caller) << ',' + << static_cast(record.block_type) << ',' << (record.is_cache_hit ? 1 : 0) + << ',' << (record.no_insert ? 1 : 0) << ',' + << (record.referenced_key_exist_in_block ? 1 : 0) << ',' << record.block_size + << ',' << record.cf_id << ','; + WriteCsvString(output_, cf_name); + output_.put('\n'); + return rocksdb::Status(); + } + + private: + std::ofstream output_; + uint64_t sample_one_in_; + uint64_t max_bytes_; + bool full_ = false; + std::mutex mutex_; +}; + +std::string JStringToString(JNIEnv* env, jstring value) { + const char* chars = env->GetStringUTFChars(value, nullptr); + if (chars == nullptr) { + return {}; + } + std::string result(chars); + env->ReleaseStringUTFChars(value, chars); + return result; +} + +jstring Error(JNIEnv* env, const std::string& message) { + return env->NewStringUTF(message.c_str()); +} + +} // namespace + +extern "C" JNIEXPORT jstring JNICALL +Java_org_tron_common_storage_rocksdb_RocksDbBlockCacheTrace_startTrace( + JNIEnv* env, jclass, jlong database_handle, jstring output_path, + jlong sample_one_in, jlong max_bytes) { + if (database_handle == 0 || sample_one_in <= 0 || max_bytes <= 0) { + return Error(env, "invalid block cache trace arguments"); + } + auto writer = std::make_unique( + JStringToString(env, output_path), static_cast(sample_one_in), + static_cast(max_bytes)); + if (!writer->IsOpen()) { + return Error(env, "unable to open trace output"); + } + const RocksDbTraceFunctions& functions = TraceFunctions(); + if (!functions.error.empty()) { + return Error(env, functions.error); + } + rocksdb::BlockCacheTraceOptions options; + options.sampling_frequency = 1; + rocksdb::Status status = functions.start( + reinterpret_cast(database_handle), options, std::move(writer)); + return status.ok() ? nullptr : Error(env, status.ToString()); +} + +extern "C" JNIEXPORT jstring JNICALL +Java_org_tron_common_storage_rocksdb_RocksDbBlockCacheTrace_endTrace( + JNIEnv* env, jclass, jlong database_handle) { + if (database_handle == 0) { + return Error(env, "invalid RocksDB handle"); + } + const RocksDbTraceFunctions& functions = TraceFunctions(); + if (!functions.error.empty()) { + return Error(env, functions.error); + } + rocksdb::Status status = functions.end(reinterpret_cast(database_handle)); + return status.ok() ? nullptr : Error(env, status.ToString()); +} diff --git a/common/src/main/java/org/tron/common/prometheus/MetricKeys.java b/common/src/main/java/org/tron/common/prometheus/MetricKeys.java index 2dbb5613e7a..35d94a0db0a 100644 --- a/common/src/main/java/org/tron/common/prometheus/MetricKeys.java +++ b/common/src/main/java/org/tron/common/prometheus/MetricKeys.java @@ -22,6 +22,8 @@ public static class Counter { public static final String P2P_DISCONNECT = "tron:p2p_disconnect"; public static final String INTERNAL_SERVICE_FAIL = "tron:internal_service_fail"; public static final String DB_EVENT = "tron:db_event"; + public static final String DB_GET = "tron:db_get"; + public static final String DB_GET_PERF = "tron:db_get_perf"; public static final String DB_ROCKSDB_TICKER = "tron:db_rocksdb_ticker"; private Counter() { diff --git a/common/src/main/java/org/tron/common/prometheus/Metrics.java b/common/src/main/java/org/tron/common/prometheus/Metrics.java index ce41efb479d..3c86b055f64 100644 --- a/common/src/main/java/org/tron/common/prometheus/Metrics.java +++ b/common/src/main/java/org/tron/common/prometheus/Metrics.java @@ -2,6 +2,7 @@ import io.prometheus.client.Collector; import io.prometheus.client.CollectorRegistry; +import io.prometheus.client.Counter; import io.prometheus.client.Histogram; import io.prometheus.client.exporter.HTTPServer; import io.prometheus.client.hotspot.DefaultExports; @@ -52,6 +53,13 @@ public static void counterInc(String key, double amt, String... labels) { MetricsCounter.inc(key, amt, labels); } + public static Counter.Child databaseCounterChild(String key, String... labels) { + if (!databaseEnabled()) { + return null; + } + return MetricsCounter.child(key, labels); + } + public static void gaugeInc(String key, double amt, String... labels) { MetricsGauge.inc(key, amt, labels); } diff --git a/common/src/main/java/org/tron/common/prometheus/MetricsCounter.java b/common/src/main/java/org/tron/common/prometheus/MetricsCounter.java index b14061120a5..884ca1ae4c7 100644 --- a/common/src/main/java/org/tron/common/prometheus/MetricsCounter.java +++ b/common/src/main/java/org/tron/common/prometheus/MetricsCounter.java @@ -20,6 +20,10 @@ class MetricsCounter { init(MetricKeys.Counter.INTERNAL_SERVICE_FAIL, "internal Service fail.", "class", "method"); init(MetricKeys.Counter.DB_EVENT, "db event .", "type", "db", "event"); + init(MetricKeys.Counter.DB_GET, "rocksdb point get outcomes.", + "type", "db", "outcome"); + init(MetricKeys.Counter.DB_GET_PERF, "sampled rocksdb point get perf context values.", + "type", "db", "metric"); init(MetricKeys.Counter.DB_ROCKSDB_TICKER, "rocksdb cumulative ticker values.", "type", "db", "ticker"); } @@ -46,4 +50,9 @@ static void inc(String key, double amt, String... labels) { counter.labels(labels).inc(amt); } } + + static Counter.Child child(String key, String... labels) { + Counter counter = container.get(key); + return counter == null ? null : counter.labels(labels); + } } diff --git a/common/src/main/java/org/tron/common/setting/RocksDbSettings.java b/common/src/main/java/org/tron/common/setting/RocksDbSettings.java index 3c50cbfe637..44bfa333928 100644 --- a/common/src/main/java/org/tron/common/setting/RocksDbSettings.java +++ b/common/src/main/java/org/tron/common/setting/RocksDbSettings.java @@ -2,11 +2,16 @@ import static org.tron.core.Constant.ROCKSDB; +import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Locale; import lombok.Getter; import lombok.extern.slf4j.Slf4j; import org.rocksdb.BlockBasedTableConfig; import org.rocksdb.BloomFilter; +import org.rocksdb.CompressionType; import org.rocksdb.ComparatorOptions; import org.rocksdb.InfoLogLevel; import org.rocksdb.LRUCache; @@ -18,12 +23,21 @@ import org.slf4j.LoggerFactory; import org.tron.common.utils.MarketOrderPriceComparatorForRocksDB; import org.tron.core.Constant; +import org.tron.core.config.args.StorageConfig.DbSettingsConfig; @Slf4j public class RocksDbSettings { private static RocksDbSettings rocksDbSettings; + @Getter + private String benchmarkProfile; + @Getter + private String benchmarkMode; + @Getter + private boolean useLegacyOptions; + @Getter + private boolean legacySharedBlockCache; @Getter private int levelNumber; @Getter @@ -33,23 +47,68 @@ public class RocksDbSettings { @Getter private long blockSize; @Getter + private long blockCacheSize; + @Getter + private boolean cacheIndexAndFilterBlocks; + @Getter + private boolean pinL0FilterAndIndexBlocksInCache; + @Getter + private int bloomFilterBitsPerKey; + @Getter + private List bloomFilterDbAllowList; + @Getter + private int perfContextSampleOneIn; + @Getter + private List perfContextDbAllowList; + @Getter + private int blockCacheTraceSampleOneIn; + @Getter + private List blockCacheTraceDbAllowList; + @Getter + private String blockCacheTraceOutputDirectory; + @Getter + private long blockCacheTraceMaxBytesPerDb; + @Getter + private String blockCacheTraceNativeLibrary; + @Getter + private boolean wholeKeyFiltering; + @Getter + private int blockRestartInterval; + @Getter + private long writeBufferSize; + @Getter + private int maxWriteBufferNumber; + @Getter + private int minWriteBufferNumberToMerge; + @Getter + private int maxBackgroundFlushes; + @Getter private long maxBytesForLevelBase; @Getter private double maxBytesForLevelMultiplier; @Getter + private boolean levelCompactionDynamicLevelBytes; + @Getter private int level0FileNumCompactionTrigger; @Getter + private int level0SlowdownWritesTrigger; + @Getter + private int level0StopWritesTrigger; + @Getter private long targetFileSizeBase; @Getter private int targetFileSizeMultiplier; @Getter private boolean enableStatistics; + @Getter + private CompressionType compressionType; static { RocksDB.loadLibrary(); } - private static final LRUCache cache = new LRUCache(1 * 1024 * 1024 * 1024L); + private static LRUCache cache; + private static long cacheSize; private static final String[] CI_ENVIRONMENT_VARIABLES = { "CI", @@ -68,47 +127,107 @@ private RocksDbSettings() { public static RocksDbSettings getDefaultSettings() { RocksDbSettings defaultSettings = new RocksDbSettings(); - return defaultSettings.withLevelNumber(7).withBlockSize(64).withCompactThreads(32) + return defaultSettings.withBenchmarkProfile("default").withBenchmarkMode("E1") + .withUseLegacyOptions(true) + .withLegacySharedBlockCache(false) + .withLevelNumber(7).withBlockSize(64).withBlockCacheSize(1024) + .withCacheIndexAndFilterBlocks(true).withPinL0FilterAndIndexBlocksInCache(true) + .withBloomFilterBitsPerKey(10).withBloomFilterDbAllowList(Collections.emptyList()) + .withPerfContextSampleOneIn(0).withPerfContextDbAllowList(Collections.emptyList()) + .withBlockCacheTraceSampleOneIn(0) + .withBlockCacheTraceDbAllowList(Collections.emptyList()) + .withBlockCacheTraceOutputDirectory("").withBlockCacheTraceMaxBytesPerDb(536870912L) + .withBlockCacheTraceNativeLibrary("") + .withWholeKeyFiltering(true) + .withBlockRestartInterval(16).withWriteBufferSize(64) + .withMaxWriteBufferNumber(2).withMinWriteBufferNumberToMerge(1) + .withMaxBackgroundFlushes(1).withCompactThreads(32) .withTargetFileSizeBase(256).withMaxBytesForLevelMultiplier(10) - .withTargetFileSizeMultiplier(1) - .withMaxBytesForLevelBase(256).withMaxOpenFiles(5000).withEnableStatistics(false); + .withTargetFileSizeMultiplier(1).withMaxBytesForLevelBase(256) + .withLevelCompactionDynamicLevelBytes(true).withLevel0FileNumCompactionTrigger(2) + .withLevel0SlowdownWritesTrigger(20).withLevel0StopWritesTrigger(36) + .withMaxOpenFiles(5000).withCompressionType("SNAPPY_COMPRESSION") + .withEnableStatistics(false); } public static RocksDbSettings getSettings() { return rocksDbSettings == null ? getDefaultSettings() : rocksDbSettings; } - public static RocksDbSettings initCustomSettings(int levelNumber, int compactThreads, - int blockSize, long maxBytesForLevelBase, - double maxBytesForLevelMultiplier, int level0FileNumCompactionTrigger, - long targetFileSizeBase, - int targetFileSizeMultiplier, int maxOpenFiles) { + public static RocksDbSettings initCustomSettings(DbSettingsConfig settings) { rocksDbSettings = new RocksDbSettings() - .withMaxOpenFiles(maxOpenFiles) + .withBenchmarkProfile(settings.getBenchmarkProfile()) + .withBenchmarkMode(settings.getBenchmarkMode()) + .withUseLegacyOptions(settings.isUseLegacyOptions()) + .withLegacySharedBlockCache(settings.isLegacySharedBlockCache()) + .withMaxOpenFiles(settings.getMaxOpenFiles()) .withEnableStatistics(false) - .withLevelNumber(levelNumber) - .withCompactThreads(compactThreads) - .withBlockSize(blockSize) - .withMaxBytesForLevelBase(maxBytesForLevelBase) - .withMaxBytesForLevelMultiplier(maxBytesForLevelMultiplier) - .withLevel0FileNumCompactionTrigger(level0FileNumCompactionTrigger) - .withTargetFileSizeBase(targetFileSizeBase) - .withTargetFileSizeMultiplier(targetFileSizeMultiplier); + .withLevelNumber(settings.getLevelNumber()) + .withCompactThreads(settings.getCompactThreads()) + .withBlockSize(settings.getBlocksize()) + .withBlockCacheSize(settings.getBlockCacheSize()) + .withCacheIndexAndFilterBlocks(settings.isCacheIndexAndFilterBlocks()) + .withPinL0FilterAndIndexBlocksInCache( + settings.isPinL0FilterAndIndexBlocksInCache()) + .withBloomFilterBitsPerKey(settings.getBloomFilterBitsPerKey()) + .withBloomFilterDbAllowList(settings.getBloomFilterDbAllowList()) + .withPerfContextSampleOneIn(settings.getPerfContextSampleOneIn()) + .withPerfContextDbAllowList(settings.getPerfContextDbAllowList()) + .withBlockCacheTraceSampleOneIn(settings.getBlockCacheTraceSampleOneIn()) + .withBlockCacheTraceDbAllowList(settings.getBlockCacheTraceDbAllowList()) + .withBlockCacheTraceOutputDirectory(settings.getBlockCacheTraceOutputDirectory()) + .withBlockCacheTraceMaxBytesPerDb(settings.getBlockCacheTraceMaxBytesPerDb()) + .withBlockCacheTraceNativeLibrary(settings.getBlockCacheTraceNativeLibrary()) + .withWholeKeyFiltering(settings.isWholeKeyFiltering()) + .withBlockRestartInterval(settings.getBlockRestartInterval()) + .withWriteBufferSize(settings.getWriteBufferSize()) + .withMaxWriteBufferNumber(settings.getMaxWriteBufferNumber()) + .withMinWriteBufferNumberToMerge(settings.getMinWriteBufferNumberToMerge()) + .withMaxBackgroundFlushes(settings.getMaxBackgroundFlushes()) + .withMaxBytesForLevelBase(settings.getMaxBytesForLevelBase()) + .withMaxBytesForLevelMultiplier(settings.getMaxBytesForLevelMultiplier()) + .withLevelCompactionDynamicLevelBytes(settings.isLevelCompactionDynamicLevelBytes()) + .withLevel0FileNumCompactionTrigger(settings.getLevel0FileNumCompactionTrigger()) + .withLevel0SlowdownWritesTrigger(settings.getLevel0SlowdownWritesTrigger()) + .withLevel0StopWritesTrigger(settings.getLevel0StopWritesTrigger()) + .withTargetFileSizeBase(settings.getTargetFileSizeBase()) + .withTargetFileSizeMultiplier(settings.getTargetFileSizeMultiplier()) + .withCompressionType(settings.getCompressionType()); return rocksDbSettings; } public static void loggingSettings() { - logger.info( - "level number: {}, CompactThreads: {}, Blocksize:{}, maxBytesForLevelBase: {}," - + " withMaxBytesForLevelMultiplier: {}, level0FileNumCompactionTrigger: {}, " - + "withTargetFileSizeBase: {}, withTargetFileSizeMultiplier: {}, maxOpenFiles: {}", - rocksDbSettings.getLevelNumber(), - rocksDbSettings.getCompactThreads(), rocksDbSettings.getBlockSize(), - rocksDbSettings.getMaxBytesForLevelBase(), - rocksDbSettings.getMaxBytesForLevelMultiplier(), - rocksDbSettings.getLevel0FileNumCompactionTrigger(), - rocksDbSettings.getTargetFileSizeBase(), rocksDbSettings.getTargetFileSizeMultiplier(), - rocksDbSettings.getMaxOpenFiles()); + logger.info("RocksDB benchmark profile: {}, mode: {}, settings: {}", + rocksDbSettings.getBenchmarkProfile(), rocksDbSettings.getBenchmarkMode(), + rocksDbSettings.describe()); + } + + public String describe() { + return String.format(Locale.ROOT, + "useLegacyOptions=%s,legacySharedBlockCache=%s,levels=%d,compactThreads=%d," + + "blockSize=%d,blockCacheSize=%d," + + "cacheIndexAndFilter=%s,pinL0=%s,bloomBits=%d,bloomDbs=%s," + + "perfSampleOneIn=%d,perfDbs=%s,traceSampleOneIn=%d,traceDbs=%s," + + "traceDir=%s,traceMaxBytes=%d,wholeKey=%s," + + "restartInterval=%d," + + "writeBufferSize=%d,maxWriteBuffers=%d,minWriteBuffersToMerge=%d,flushThreads=%d," + + "maxBytesForLevelBase=%d,maxBytesMultiplier=%s,dynamicLevels=%s,l0Trigger=%d," + + "l0Slowdown=%d,l0Stop=%d,targetFileBase=%d,targetFileMultiplier=%d," + + "maxOpenFiles=%d,compression=%s", + useLegacyOptions, legacySharedBlockCache, levelNumber, compactThreads, + blockSize, blockCacheSize, + cacheIndexAndFilterBlocks, + pinL0FilterAndIndexBlocksInCache, bloomFilterBitsPerKey, bloomFilterDbAllowList, + perfContextSampleOneIn, perfContextDbAllowList, + blockCacheTraceSampleOneIn, blockCacheTraceDbAllowList, + blockCacheTraceOutputDirectory, blockCacheTraceMaxBytesPerDb, + wholeKeyFiltering, + blockRestartInterval, writeBufferSize, maxWriteBufferNumber, + minWriteBufferNumberToMerge, maxBackgroundFlushes, maxBytesForLevelBase, + maxBytesForLevelMultiplier, levelCompactionDynamicLevelBytes, + level0FileNumCompactionTrigger, level0SlowdownWritesTrigger, + level0StopWritesTrigger, targetFileSizeBase, targetFileSizeMultiplier, + maxOpenFiles, compressionType); } public RocksDbSettings withMaxOpenFiles(int maxOpenFiles) { @@ -116,6 +235,26 @@ public RocksDbSettings withMaxOpenFiles(int maxOpenFiles) { return this; } + public RocksDbSettings withBenchmarkProfile(String benchmarkProfile) { + this.benchmarkProfile = benchmarkProfile; + return this; + } + + public RocksDbSettings withBenchmarkMode(String benchmarkMode) { + this.benchmarkMode = benchmarkMode; + return this; + } + + public RocksDbSettings withUseLegacyOptions(boolean useLegacyOptions) { + this.useLegacyOptions = useLegacyOptions; + return this; + } + + public RocksDbSettings withLegacySharedBlockCache(boolean legacySharedBlockCache) { + this.legacySharedBlockCache = legacySharedBlockCache; + return this; + } + public RocksDbSettings withCompactThreads(int compactThreads) { this.compactThreads = compactThreads; return this; @@ -126,6 +265,96 @@ public RocksDbSettings withBlockSize(long blockSize) { return this; } + public RocksDbSettings withBlockCacheSize(long blockCacheSize) { + this.blockCacheSize = blockCacheSize * 1024 * 1024; + return this; + } + + public RocksDbSettings withCacheIndexAndFilterBlocks(boolean enabled) { + this.cacheIndexAndFilterBlocks = enabled; + return this; + } + + public RocksDbSettings withPinL0FilterAndIndexBlocksInCache(boolean enabled) { + this.pinL0FilterAndIndexBlocksInCache = enabled; + return this; + } + + public RocksDbSettings withBloomFilterBitsPerKey(int bloomFilterBitsPerKey) { + this.bloomFilterBitsPerKey = bloomFilterBitsPerKey; + return this; + } + + public RocksDbSettings withBloomFilterDbAllowList(List dbNames) { + this.bloomFilterDbAllowList = Collections.unmodifiableList(new ArrayList<>(dbNames)); + return this; + } + + public RocksDbSettings withPerfContextSampleOneIn(int sampleOneIn) { + this.perfContextSampleOneIn = sampleOneIn; + return this; + } + + public RocksDbSettings withPerfContextDbAllowList(List dbNames) { + this.perfContextDbAllowList = Collections.unmodifiableList(new ArrayList<>(dbNames)); + return this; + } + + public RocksDbSettings withBlockCacheTraceSampleOneIn(int sampleOneIn) { + this.blockCacheTraceSampleOneIn = sampleOneIn; + return this; + } + + public RocksDbSettings withBlockCacheTraceDbAllowList(List dbNames) { + this.blockCacheTraceDbAllowList = Collections.unmodifiableList(new ArrayList<>(dbNames)); + return this; + } + + public RocksDbSettings withBlockCacheTraceOutputDirectory(String directory) { + this.blockCacheTraceOutputDirectory = directory; + return this; + } + + public RocksDbSettings withBlockCacheTraceMaxBytesPerDb(long maxBytes) { + this.blockCacheTraceMaxBytesPerDb = maxBytes; + return this; + } + + public RocksDbSettings withBlockCacheTraceNativeLibrary(String library) { + this.blockCacheTraceNativeLibrary = library; + return this; + } + + public RocksDbSettings withWholeKeyFiltering(boolean wholeKeyFiltering) { + this.wholeKeyFiltering = wholeKeyFiltering; + return this; + } + + public RocksDbSettings withBlockRestartInterval(int blockRestartInterval) { + this.blockRestartInterval = blockRestartInterval; + return this; + } + + public RocksDbSettings withWriteBufferSize(long writeBufferSize) { + this.writeBufferSize = writeBufferSize * 1024 * 1024; + return this; + } + + public RocksDbSettings withMaxWriteBufferNumber(int maxWriteBufferNumber) { + this.maxWriteBufferNumber = maxWriteBufferNumber; + return this; + } + + public RocksDbSettings withMinWriteBufferNumberToMerge(int value) { + this.minWriteBufferNumberToMerge = value; + return this; + } + + public RocksDbSettings withMaxBackgroundFlushes(int maxBackgroundFlushes) { + this.maxBackgroundFlushes = maxBackgroundFlushes; + return this; + } + public RocksDbSettings withMaxBytesForLevelBase(long maxBytesForLevelBase) { this.maxBytesForLevelBase = maxBytesForLevelBase * 1024 * 1024; return this; @@ -136,11 +365,26 @@ public RocksDbSettings withMaxBytesForLevelMultiplier(double maxBytesForLevelMul return this; } + public RocksDbSettings withLevelCompactionDynamicLevelBytes(boolean enabled) { + this.levelCompactionDynamicLevelBytes = enabled; + return this; + } + public RocksDbSettings withLevel0FileNumCompactionTrigger(int level0FileNumCompactionTrigger) { this.level0FileNumCompactionTrigger = level0FileNumCompactionTrigger; return this; } + public RocksDbSettings withLevel0SlowdownWritesTrigger(int value) { + this.level0SlowdownWritesTrigger = value; + return this; + } + + public RocksDbSettings withLevel0StopWritesTrigger(int value) { + this.level0StopWritesTrigger = value; + return this; + } + public RocksDbSettings withEnableStatistics(boolean enable) { this.enableStatistics = enable; return this; @@ -160,7 +404,25 @@ public RocksDbSettings withTargetFileSizeMultiplier(int targetFileSizeMultiplier this.targetFileSizeMultiplier = targetFileSizeMultiplier; return this; } - public static LRUCache getCache() { + + public RocksDbSettings withCompressionType(String compressionType) { + try { + this.compressionType = CompressionType.valueOf(compressionType); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException("Unsupported RocksDB compressionType: " + + compressionType, e); + } + return this; + } + + private static synchronized LRUCache getCache(long requestedSize) { + if (cache == null) { + cache = new LRUCache(requestedSize); + cacheSize = requestedSize; + } else if (cacheSize != requestedSize) { + throw new IllegalStateException("RocksDB blockCacheSize changed in the same JVM; " + + "restart the process between benchmark profiles"); + } return cache; } @@ -201,34 +463,17 @@ protected void log(InfoLogLevel infoLogLevel, String logMsg) { // Prometheus polls selected tickers; avoid a second periodic dump to the RocksDB log. options.setStatsDumpPeriodSec(0); } - options.setCreateIfMissing(true); - options.setIncreaseParallelism(1); - options.setLevelCompactionDynamicLevelBytes(true); - options.setMaxOpenFiles(settings.getMaxOpenFiles()); - - // general options supported user config - options.setNumLevels(settings.getLevelNumber()); - options.setMaxBytesForLevelMultiplier(settings.getMaxBytesForLevelMultiplier()); - options.setMaxBytesForLevelBase(settings.getMaxBytesForLevelBase()); - options.setMaxBackgroundCompactions(settings.getCompactThreads()); - options.setLevel0FileNumCompactionTrigger(settings.getLevel0FileNumCompactionTrigger()); - options.setTargetFileSizeMultiplier(settings.getTargetFileSizeMultiplier()); - options.setTargetFileSizeBase(settings.getTargetFileSizeBase()); - - // table options - final BlockBasedTableConfig tableCfg; - options.setTableFormatConfig(tableCfg = new BlockBasedTableConfig()); - tableCfg.setBlockSize(settings.getBlockSize()); - tableCfg.setBlockCache(RocksDbSettings.getCache()); - tableCfg.setCacheIndexAndFilterBlocks(true); - tableCfg.setPinL0FilterAndIndexBlocksInCache(true); - tableCfg.setFilter(new BloomFilter(10, false)); + if (settings.isUseLegacyOptions()) { + applyLegacyOptions(options, settings, dbName); + } else { + applyCustomOptions(options, settings); + } if (Constant.MARKET_PAIR_PRICE_TO_ORDER.equals(dbName)) { ComparatorOptions comparatorOptions = new ComparatorOptions(); options.setComparator(new MarketOrderPriceComparatorForRocksDB(comparatorOptions)); } - if (isRunningInCI()) { + if (isRunningInCI() && "default".equals(settings.getBenchmarkProfile())) { options.optimizeForSmallDb(); // Disable fallocate calls to avoid issues with disk space options.setAllowFAllocate(false); @@ -245,6 +490,84 @@ protected void log(InfoLogLevel infoLogLevel, String logMsg) { return options; } + private static void applyLegacyOptions(Options options, RocksDbSettings settings, + String dbName) { + options.setCreateIfMissing(true); + options.setIncreaseParallelism(1); + options.setLevelCompactionDynamicLevelBytes(true); + options.setMaxOpenFiles(settings.getMaxOpenFiles()); + options.setNumLevels(settings.getLevelNumber()); + options.setMaxBytesForLevelMultiplier(settings.getMaxBytesForLevelMultiplier()); + options.setMaxBytesForLevelBase(settings.getMaxBytesForLevelBase()); + options.setMaxBackgroundCompactions(settings.getCompactThreads()); + options.setLevel0FileNumCompactionTrigger(settings.getLevel0FileNumCompactionTrigger()); + options.setTargetFileSizeMultiplier(settings.getTargetFileSizeMultiplier()); + options.setTargetFileSizeBase(settings.getTargetFileSizeBase()); + BlockBasedTableConfig tableCfg = new BlockBasedTableConfig(); + if (settings.isLegacySharedBlockCache()) { + tableCfg.setBlockCache(RocksDbSettings.getCache(settings.getBlockCacheSize())); + } + if (settings.shouldEnableBloomFilter(dbName)) { + tableCfg.setWholeKeyFiltering(settings.isWholeKeyFiltering()); + tableCfg.setFilter(new BloomFilter(settings.getBloomFilterBitsPerKey(), false)); + } + options.setTableFormatConfig(tableCfg); + } + + boolean shouldEnableBloomFilter(String dbName) { + return bloomFilterBitsPerKey > 0 && bloomFilterDbAllowList.contains(dbName); + } + + public boolean shouldSamplePerfContext(String dbName) { + return perfContextSampleOneIn > 0 && perfContextDbAllowList.contains(dbName); + } + + public boolean shouldTraceBlockCache(String dbName) { + return blockCacheTraceSampleOneIn > 0 + && (blockCacheTraceDbAllowList.contains("*") + || blockCacheTraceDbAllowList.contains("all") + || blockCacheTraceDbAllowList.contains(dbName)); + } + + private static void applyCustomOptions(Options options, RocksDbSettings settings) { + options.setCreateIfMissing(true); + options.setIncreaseParallelism(1); + options.setLevelCompactionDynamicLevelBytes( + settings.isLevelCompactionDynamicLevelBytes()); + options.setMaxOpenFiles(settings.getMaxOpenFiles()); + options.setNumLevels(settings.getLevelNumber()); + options.setMaxBytesForLevelMultiplier(settings.getMaxBytesForLevelMultiplier()); + options.setMaxBytesForLevelBase(settings.getMaxBytesForLevelBase()); + options.setMaxBackgroundCompactions(settings.getCompactThreads()); + options.setMaxBackgroundFlushes(settings.getMaxBackgroundFlushes()); + options.setWriteBufferSize(settings.getWriteBufferSize()); + options.setMaxWriteBufferNumber(settings.getMaxWriteBufferNumber()); + options.setMinWriteBufferNumberToMerge(settings.getMinWriteBufferNumberToMerge()); + options.setLevel0FileNumCompactionTrigger(settings.getLevel0FileNumCompactionTrigger()); + options.setLevel0SlowdownWritesTrigger(settings.getLevel0SlowdownWritesTrigger()); + options.setLevel0StopWritesTrigger(settings.getLevel0StopWritesTrigger()); + options.setTargetFileSizeMultiplier(settings.getTargetFileSizeMultiplier()); + options.setTargetFileSizeBase(settings.getTargetFileSizeBase()); + options.setCompressionType(settings.getCompressionType()); + + BlockBasedTableConfig tableCfg = new BlockBasedTableConfig(); + tableCfg.setBlockSize(settings.getBlockSize()); + tableCfg.setBlockRestartInterval(settings.getBlockRestartInterval()); + tableCfg.setWholeKeyFiltering(settings.isWholeKeyFiltering()); + if (settings.getBlockCacheSize() == 0) { + tableCfg.setNoBlockCache(true); + } else { + tableCfg.setBlockCache(RocksDbSettings.getCache(settings.getBlockCacheSize())); + tableCfg.setCacheIndexAndFilterBlocks(settings.isCacheIndexAndFilterBlocks()); + tableCfg.setPinL0FilterAndIndexBlocksInCache( + settings.isPinL0FilterAndIndexBlocksInCache()); + } + if (settings.getBloomFilterBitsPerKey() > 0) { + tableCfg.setFilter(new BloomFilter(settings.getBloomFilterBitsPerKey(), false)); + } + options.setTableFormatConfig(tableCfg); + } + private static boolean isRunningInCI() { return Arrays.stream(CI_ENVIRONMENT_VARIABLES).anyMatch(System.getenv()::containsKey); } diff --git a/common/src/main/java/org/tron/core/config/Configuration.java b/common/src/main/java/org/tron/core/config/Configuration.java index 80735290b8c..8a4a52019b0 100644 --- a/common/src/main/java/org/tron/core/config/Configuration.java +++ b/common/src/main/java/org/tron/core/config/Configuration.java @@ -21,6 +21,7 @@ import static org.apache.commons.lang3.StringUtils.isBlank; import com.typesafe.config.ConfigFactory; +import com.typesafe.config.ConfigValueFactory; import java.io.File; import lombok.extern.slf4j.Slf4j; @@ -46,6 +47,63 @@ public static com.typesafe.config.Config getByFileName( return config; } + /** + * Load the node config and apply a restricted RocksDB benchmark profile. + * + *

The profile may only provide {@code rocksdb-profile.settings}. Those values override + * {@code storage.dbSettings}; all other node settings continue to come from the base config. + * This keeps benchmark profiles reusable without allowing an experiment file to accidentally + * change network, witness, or output-directory settings.

+ * + * @param confFileName base node config + * @param rocksDbProfileFile optional RocksDB profile file + * @return merged config + */ + public static com.typesafe.config.Config getByFileName( + final String confFileName, final String rocksDbProfileFile) { + com.typesafe.config.Config base = getByFileName(confFileName); + if (isBlank(rocksDbProfileFile)) { + return base; + } + + File profileFile = new File(rocksDbProfileFile); + if (!profileFile.isFile()) { + throw new IllegalArgumentException( + "RocksDB profile path is required! No Such file " + rocksDbProfileFile); + } + com.typesafe.config.Config profile = ConfigFactory.parseFile(profileFile).resolve(); + if (!profile.hasPath("rocksdb-profile.name") + || !profile.hasPath("rocksdb-profile.mode") + || !profile.hasPath("rocksdb-profile.settings")) { + throw new IllegalArgumentException( + "RocksDB profile must define rocksdb-profile.name, mode, and settings"); + } + + String profileName = profile.getString("rocksdb-profile.name"); + String profileMode = profile.getString("rocksdb-profile.mode"); + if (!profileName.matches("[A-Za-z0-9._-]+")) { + throw new IllegalArgumentException( + "RocksDB profile name may only contain letters, digits, dot, underscore, or dash"); + } + if (!"E1".equals(profileMode) && !"E2".equals(profileMode)) { + throw new IllegalArgumentException("RocksDB profile mode must be E1 or E2"); + } + + com.typesafe.config.Config profileSettings = profile.getConfig("rocksdb-profile.settings"); + if (!profileSettings.hasPath("useLegacyOptions")) { + profileSettings = profileSettings.withValue("useLegacyOptions", + ConfigValueFactory.fromAnyRef(false)); + } + com.typesafe.config.Config settings = profileSettings + .withValue("benchmarkProfile", + ConfigValueFactory.fromAnyRef(profileName)) + .withValue("benchmarkMode", + ConfigValueFactory.fromAnyRef(profileMode)) + .withFallback(base.getConfig("storage.dbSettings")); + config = base.withValue("storage.dbSettings", settings.root()).resolve(); + return config; + } + private static void resolveConfigFile(String fileName, File confFile) { if (confFile.exists()) { config = ConfigFactory.parseFile(confFile) @@ -59,4 +117,3 @@ private static void resolveConfigFile(String fileName, File confFile) { } } } - diff --git a/common/src/main/java/org/tron/core/config/args/StorageConfig.java b/common/src/main/java/org/tron/core/config/args/StorageConfig.java index 2c6c3e60a41..eef70d7c142 100644 --- a/common/src/main/java/org/tron/core/config/args/StorageConfig.java +++ b/common/src/main/java/org/tron/core/config/args/StorageConfig.java @@ -84,21 +84,104 @@ public void setSwitch(String v) { @Setter public static class DbSettingsConfig { + private String benchmarkProfile = "default"; + private String benchmarkMode = "E1"; + private boolean useLegacyOptions = true; + private boolean legacySharedBlockCache = false; private int levelNumber = 7; private int compactThreads = 0; // 0 = auto: max(availableProcessors, 1) private int blocksize = 16; + private long blockCacheSize = 1024; + private boolean cacheIndexAndFilterBlocks = true; + private boolean pinL0FilterAndIndexBlocksInCache = true; + private int bloomFilterBitsPerKey = 10; + private List bloomFilterDbAllowList = new ArrayList<>(); + private int perfContextSampleOneIn = 0; + private List perfContextDbAllowList = new ArrayList<>(); + private int blockCacheTraceSampleOneIn = 0; + private List blockCacheTraceDbAllowList = new ArrayList<>(); + private String blockCacheTraceOutputDirectory = ""; + private long blockCacheTraceMaxBytesPerDb = 536870912L; + private String blockCacheTraceNativeLibrary = ""; + private boolean wholeKeyFiltering = true; + private int blockRestartInterval = 16; + private long writeBufferSize = 64; + private int maxWriteBufferNumber = 2; + private int minWriteBufferNumberToMerge = 1; + private int maxBackgroundFlushes = 1; private long maxBytesForLevelBase = 256; private double maxBytesForLevelMultiplier = 10; + private boolean levelCompactionDynamicLevelBytes = true; private int level0FileNumCompactionTrigger = 2; + private int level0SlowdownWritesTrigger = 20; + private int level0StopWritesTrigger = 36; private long targetFileSizeBase = 64; private int targetFileSizeMultiplier = 1; private int maxOpenFiles = 5000; + private String compressionType = "SNAPPY_COMPRESSION"; // Expand 0 → auto-detected processor count. Mirrors develop Args.java:1609-1611. void postProcess() { if (compactThreads == 0) { compactThreads = StrictMathWrapper.max(Runtime.getRuntime().availableProcessors(), 1); } + if (blocksize <= 0 || blockCacheSize < 0 || bloomFilterBitsPerKey < 0 + || blockRestartInterval <= 0 || writeBufferSize <= 0) { + throw new IllegalArgumentException("RocksDB size settings must be positive"); + } + if (maxWriteBufferNumber < 2 || minWriteBufferNumberToMerge < 1 + || minWriteBufferNumberToMerge > maxWriteBufferNumber) { + throw new IllegalArgumentException("Invalid RocksDB write buffer settings"); + } + if (maxBackgroundFlushes < 1 || level0FileNumCompactionTrigger < 1 + || level0SlowdownWritesTrigger < level0FileNumCompactionTrigger + || level0StopWritesTrigger < level0SlowdownWritesTrigger) { + throw new IllegalArgumentException("Invalid RocksDB background or L0 settings"); + } + if (pinL0FilterAndIndexBlocksInCache && !cacheIndexAndFilterBlocks) { + throw new IllegalArgumentException( + "pinL0FilterAndIndexBlocksInCache requires cacheIndexAndFilterBlocks"); + } + if (blockCacheSize == 0 && cacheIndexAndFilterBlocks) { + throw new IllegalArgumentException( + "cacheIndexAndFilterBlocks requires a positive blockCacheSize"); + } + if (legacySharedBlockCache && blockCacheSize <= 0) { + throw new IllegalArgumentException( + "legacySharedBlockCache requires a positive blockCacheSize"); + } + if (bloomFilterDbAllowList.stream().anyMatch( + dbName -> dbName == null || dbName.trim().isEmpty())) { + throw new IllegalArgumentException( + "bloomFilterDbAllowList must contain non-empty database names"); + } + if (perfContextSampleOneIn < 0) { + throw new IllegalArgumentException("perfContextSampleOneIn must not be negative"); + } + if (perfContextDbAllowList.stream().anyMatch( + dbName -> dbName == null || dbName.trim().isEmpty())) { + throw new IllegalArgumentException( + "perfContextDbAllowList must contain non-empty database names"); + } + if (!perfContextDbAllowList.isEmpty() && perfContextSampleOneIn == 0) { + throw new IllegalArgumentException( + "perfContextSampleOneIn must be positive when databases are allowed"); + } + if (blockCacheTraceSampleOneIn < 0 || blockCacheTraceMaxBytesPerDb <= 0) { + throw new IllegalArgumentException("Invalid RocksDB block cache trace settings"); + } + if (blockCacheTraceDbAllowList.stream().anyMatch( + dbName -> dbName == null || dbName.trim().isEmpty())) { + throw new IllegalArgumentException( + "blockCacheTraceDbAllowList must contain non-empty database names"); + } + if (!blockCacheTraceDbAllowList.isEmpty() + && (blockCacheTraceSampleOneIn == 0 + || blockCacheTraceOutputDirectory.trim().isEmpty() + || blockCacheTraceNativeLibrary.trim().isEmpty())) { + throw new IllegalArgumentException("Enabled RocksDB block cache trace requires a positive " + + "sample rate, output directory, and native library"); + } } } diff --git a/common/src/main/resources/reference.conf b/common/src/main/resources/reference.conf index 2fb5c747f9e..2af02cbf131 100644 --- a/common/src/main/resources/reference.conf +++ b/common/src/main/resources/reference.conf @@ -106,15 +106,41 @@ storage { # RocksDB settings (only used when db.engine = "ROCKSDB") # Strongly recommend NOT modifying unless you know every item's meaning clearly. dbSettings = { + benchmarkProfile = "default" // Label emitted in logs for benchmark traceability. + benchmarkMode = "E1" // E1=migration state, E2=materialized steady state. + useLegacyOptions = true // Preserve the pre-profile native Options unless opted out. + legacySharedBlockCache = false // Keep legacy Options but inject one shared block cache. levelNumber = 7 // Number of RocksDB levels. compactThreads = 0 // 0 = auto: max(availableProcessors, 1) blocksize = 16 // n * KB + blockCacheSize = 1024 // Shared block cache, n * MB. 0 disables block cache. + cacheIndexAndFilterBlocks = true + pinL0FilterAndIndexBlocksInCache = true + bloomFilterBitsPerKey = 10 // 0 disables Bloom filter for newly built SSTs. + bloomFilterDbAllowList = [] // Legacy Options: enable Bloom only for listed databases. + perfContextSampleOneIn = 0 // 0 disables sampled per-Get PerfContext collection. + perfContextDbAllowList = [] // Collect only for these databases when sampling is enabled. + blockCacheTraceSampleOneIn = 0 // 0 disables per-level block cache tracing. + blockCacheTraceDbAllowList = [] // Use ["*"] or ["all"] for every RocksDB store. + blockCacheTraceOutputDirectory = "" // Trace files are written outside database directories. + blockCacheTraceMaxBytesPerDb = 536870912 // Hard limit for each database trace file. + blockCacheTraceNativeLibrary = "" // Absolute path to the version-matched JNI bridge. + wholeKeyFiltering = true + blockRestartInterval = 16 + writeBufferSize = 64 // Memtable size, n * MB. + maxWriteBufferNumber = 2 + minWriteBufferNumberToMerge = 1 + maxBackgroundFlushes = 1 maxBytesForLevelBase = 256 // n * MB maxBytesForLevelMultiplier = 10 // Level size multiplier. + levelCompactionDynamicLevelBytes = true level0FileNumCompactionTrigger = 2 // L0 files that trigger compaction. + level0SlowdownWritesTrigger = 20 + level0StopWritesTrigger = 36 targetFileSizeBase = 64 // n * MB targetFileSizeMultiplier = 1 // Target file size multiplier. maxOpenFiles = 5000 // Maximum open files for RocksDB. + compressionType = "SNAPPY_COMPRESSION" } balance.history.lookup = false # Whether to enable historical balance lookup. diff --git a/common/src/test/java/org/tron/common/setting/RocksDbSettingsTest.java b/common/src/test/java/org/tron/common/setting/RocksDbSettingsTest.java new file mode 100644 index 00000000000..f13d74fe518 --- /dev/null +++ b/common/src/test/java/org/tron/common/setting/RocksDbSettingsTest.java @@ -0,0 +1,173 @@ +package org.tron.common.setting; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.util.Collections; +import org.junit.Test; +import org.rocksdb.BlockBasedTableConfig; +import org.rocksdb.CompressionType; +import org.rocksdb.Options; +import org.tron.core.config.args.StorageConfig.DbSettingsConfig; + +public class RocksDbSettingsTest { + + @Test + public void shouldApplyBenchmarkSettingsToNativeOptions() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setUseLegacyOptions(false); + config.setBenchmarkProfile("native-options"); + config.setBenchmarkMode("E2"); + config.setBlocksize(32); + config.setWriteBufferSize(96); + config.setMaxWriteBufferNumber(4); + config.setMinWriteBufferNumberToMerge(2); + config.setMaxBackgroundFlushes(3); + config.setLevel0FileNumCompactionTrigger(5); + config.setLevel0SlowdownWritesTrigger(12); + config.setLevel0StopWritesTrigger(18); + config.setTargetFileSizeBase(128); + config.setCompressionType("LZ4_COMPRESSION"); + + RocksDbSettings settings = RocksDbSettings.initCustomSettings(config); + try (Options options = RocksDbSettings.getOptionsByDbName("benchmark-test")) { + assertEquals("native-options", settings.getBenchmarkProfile()); + assertEquals("E2", settings.getBenchmarkMode()); + assertEquals(96L * 1024 * 1024, options.writeBufferSize()); + assertEquals(4, options.maxWriteBufferNumber()); + assertEquals(2, options.minWriteBufferNumberToMerge()); + assertEquals(3, options.maxBackgroundFlushes()); + assertEquals(5, options.level0FileNumCompactionTrigger()); + assertEquals(12, options.level0SlowdownWritesTrigger()); + assertEquals(18, options.level0StopWritesTrigger()); + assertEquals(128L * 1024 * 1024, options.targetFileSizeBase()); + assertEquals(CompressionType.LZ4_COMPRESSION, options.compressionType()); + + assertTrue(options.tableFormatConfig() instanceof BlockBasedTableConfig); + BlockBasedTableConfig table = (BlockBasedTableConfig) options.tableFormatConfig(); + assertEquals(32L * 1024, table.blockSize()); + assertEquals(16, table.blockRestartInterval()); + assertTrue(table.cacheIndexAndFilterBlocks()); + assertTrue(table.pinL0FilterAndIndexBlocksInCache()); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + + @Test + public void shouldPreserveLegacyNativeOptionsByDefault() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setBenchmarkProfile("legacy-options"); + config.setMaxBackgroundFlushes(3); + config.setWriteBufferSize(96); + config.setBlocksize(32); + config.setCacheIndexAndFilterBlocks(true); + config.setPinL0FilterAndIndexBlocksInCache(true); + config.setBloomFilterBitsPerKey(10); + config.setLevel0FileNumCompactionTrigger(5); + config.setTargetFileSizeBase(128); + config.setCompressionType("LZ4_COMPRESSION"); + + RocksDbSettings settings = RocksDbSettings.initCustomSettings(config); + try (Options options = RocksDbSettings.getOptionsByDbName("legacy-options"); + Options expected = new Options()) { + expected.setIncreaseParallelism(1); + expected.setMaxBackgroundCompactions(settings.getCompactThreads()); + assertTrue(settings.isUseLegacyOptions()); + assertEquals(expected.maxBackgroundFlushes(), options.maxBackgroundFlushes()); + assertEquals(expected.writeBufferSize(), options.writeBufferSize()); + assertEquals(expected.compressionType(), options.compressionType()); + assertEquals(5, options.level0FileNumCompactionTrigger()); + assertEquals(128L * 1024 * 1024, options.targetFileSizeBase()); + assertTrue(options.tableFormatConfig() instanceof BlockBasedTableConfig); + BlockBasedTableConfig table = (BlockBasedTableConfig) options.tableFormatConfig(); + BlockBasedTableConfig defaultTable = new BlockBasedTableConfig(); + assertEquals(defaultTable.blockSize(), table.blockSize()); + assertEquals(defaultTable.blockRestartInterval(), table.blockRestartInterval()); + assertEquals(defaultTable.cacheIndexAndFilterBlocks(), + table.cacheIndexAndFilterBlocks()); + assertEquals(defaultTable.pinL0FilterAndIndexBlocksInCache(), + table.pinL0FilterAndIndexBlocksInCache()); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + + @Test + public void shouldEnableSharedCacheWithoutChangingLegacyTableOptions() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setBenchmarkProfile("b1-shared-cache-1g"); + config.setUseLegacyOptions(true); + config.setLegacySharedBlockCache(true); + config.setBlockCacheSize(1024); + + RocksDbSettings settings = RocksDbSettings.initCustomSettings(config); + try (Options options = RocksDbSettings.getOptionsByDbName("b1-first"); + Options second = RocksDbSettings.getOptionsByDbName("b1-second")) { + assertTrue(settings.isUseLegacyOptions()); + assertTrue(settings.isLegacySharedBlockCache()); + assertTrue(options.tableFormatConfig() instanceof BlockBasedTableConfig); + assertTrue(second.tableFormatConfig() instanceof BlockBasedTableConfig); + BlockBasedTableConfig table = (BlockBasedTableConfig) options.tableFormatConfig(); + BlockBasedTableConfig expected = new BlockBasedTableConfig(); + assertEquals(expected.blockSize(), table.blockSize()); + assertEquals(expected.blockRestartInterval(), table.blockRestartInterval()); + assertEquals(expected.cacheIndexAndFilterBlocks(), + table.cacheIndexAndFilterBlocks()); + assertEquals(expected.pinL0FilterAndIndexBlocksInCache(), + table.pinL0FilterAndIndexBlocksInCache()); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + + @Test + public void shouldEnableLegacyBloomFilterOnlyForAllowedDatabase() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setBenchmarkProfile("account-asset-bloom"); + config.setUseLegacyOptions(true); + config.setBloomFilterBitsPerKey(10); + config.setBloomFilterDbAllowList(Collections.singletonList("account-asset")); + + RocksDbSettings.initCustomSettings(config); + try { + RocksDbSettings settings = RocksDbSettings.getSettings(); + assertTrue(settings.shouldEnableBloomFilter("account-asset")); + assertTrue(!settings.shouldEnableBloomFilter("delegated-resource")); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + + @Test + public void shouldEnablePerfContextOnlyForAllowedDatabase() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setPerfContextSampleOneIn(100); + config.setPerfContextDbAllowList(Collections.singletonList("storage-row")); + + RocksDbSettings settings = RocksDbSettings.initCustomSettings(config); + try { + assertEquals(100, settings.getPerfContextSampleOneIn()); + assertTrue(settings.shouldSamplePerfContext("storage-row")); + assertTrue(!settings.shouldSamplePerfContext("account")); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + + @Test + public void shouldEnableBlockCacheTraceForAllOrAllowedDatabase() { + DbSettingsConfig config = new DbSettingsConfig(); + config.setBlockCacheTraceSampleOneIn(100); + config.setBlockCacheTraceDbAllowList(Collections.singletonList("*")); + config.setBlockCacheTraceOutputDirectory("/tmp/trace"); + config.setBlockCacheTraceNativeLibrary("/tmp/libtrace.so"); + RocksDbSettings settings = RocksDbSettings.initCustomSettings(config); + try { + assertTrue(settings.shouldTraceBlockCache("account")); + assertTrue(settings.shouldTraceBlockCache("storage-row")); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } +} diff --git a/common/src/test/java/org/tron/core/config/args/StorageConfigTest.java b/common/src/test/java/org/tron/core/config/args/StorageConfigTest.java index e3f1925a763..6ae4c732a1c 100644 --- a/common/src/test/java/org/tron/core/config/args/StorageConfigTest.java +++ b/common/src/test/java/org/tron/core/config/args/StorageConfigTest.java @@ -65,16 +65,90 @@ public void testDbSettingsDefaults() { StorageConfig sc = StorageConfig.fromConfig(empty); StorageConfig.DbSettingsConfig ds = sc.getDbSettings(); assertEquals(7, ds.getLevelNumber()); + assertEquals("default", ds.getBenchmarkProfile()); + assertEquals("E1", ds.getBenchmarkMode()); + assertTrue(ds.isUseLegacyOptions()); // compactThreads default is 0 in reference.conf, auto-expanded by postProcess() assertEquals(StrictMathWrapper.max(Runtime.getRuntime().availableProcessors(), 1), ds.getCompactThreads()); assertEquals(16, ds.getBlocksize()); + assertEquals(1024, ds.getBlockCacheSize()); + assertTrue(ds.isCacheIndexAndFilterBlocks()); + assertTrue(ds.isPinL0FilterAndIndexBlocksInCache()); + assertEquals(10, ds.getBloomFilterBitsPerKey()); + assertTrue(ds.getBloomFilterDbAllowList().isEmpty()); + assertEquals(64, ds.getWriteBufferSize()); + assertEquals(2, ds.getMaxWriteBufferNumber()); + assertEquals(1, ds.getMinWriteBufferNumberToMerge()); assertEquals(256, ds.getMaxBytesForLevelBase()); assertEquals(10, ds.getMaxBytesForLevelMultiplier(), 0.01); assertEquals(2, ds.getLevel0FileNumCompactionTrigger()); assertEquals(64, ds.getTargetFileSizeBase()); assertEquals(1, ds.getTargetFileSizeMultiplier()); assertEquals(5000, ds.getMaxOpenFiles()); + assertEquals("SNAPPY_COMPRESSION", ds.getCompressionType()); + } + + @Test + public void testBenchmarkSettingsOverride() { + Config config = withRef("storage.dbSettings { benchmarkProfile = cache-2g, " + + "benchmarkMode = E1, useLegacyOptions = false, blockCacheSize = 2048, " + + "writeBufferSize = 128, " + + "maxWriteBufferNumber = 4, minWriteBufferNumberToMerge = 2 }"); + StorageConfig.DbSettingsConfig settings = StorageConfig.fromConfig(config).getDbSettings(); + assertEquals("cache-2g", settings.getBenchmarkProfile()); + assertFalse(settings.isUseLegacyOptions()); + assertEquals(2048, settings.getBlockCacheSize()); + assertEquals(128, settings.getWriteBufferSize()); + assertEquals(4, settings.getMaxWriteBufferNumber()); + assertEquals(2, settings.getMinWriteBufferNumberToMerge()); + } + + @Test + public void testBloomFilterDbAllowListOverride() { + Config config = withRef( + "storage.dbSettings.bloomFilterDbAllowList = [account-asset, delegated-resource]"); + assertEquals(java.util.Arrays.asList("account-asset", "delegated-resource"), + StorageConfig.fromConfig(config).getDbSettings().getBloomFilterDbAllowList()); + } + + @Test + public void testPerfContextSamplingOverride() { + Config config = withRef("storage.dbSettings { perfContextSampleOneIn = 100, " + + "perfContextDbAllowList = [account-asset, storage-row] }"); + StorageConfig.DbSettingsConfig settings = StorageConfig.fromConfig(config).getDbSettings(); + assertEquals(100, settings.getPerfContextSampleOneIn()); + assertEquals(java.util.Arrays.asList("account-asset", "storage-row"), + settings.getPerfContextDbAllowList()); + } + + @Test(expected = IllegalArgumentException.class) + public void testPerfContextAllowListRequiresSampling() { + StorageConfig.fromConfig(withRef( + "storage.dbSettings.perfContextDbAllowList = [account-asset]")); + } + + @Test + public void testBlockCacheTraceAllDatabasesOverride() { + Config config = withRef("storage.dbSettings { blockCacheTraceSampleOneIn = 100, " + + "blockCacheTraceDbAllowList = [\"*\"], blockCacheTraceOutputDirectory = \"/tmp/t\", " + + "blockCacheTraceNativeLibrary = \"/tmp/libtrace.so\" }"); + StorageConfig.DbSettingsConfig settings = StorageConfig.fromConfig(config).getDbSettings(); + assertEquals(100, settings.getBlockCacheTraceSampleOneIn()); + assertEquals(java.util.Collections.singletonList("*"), + settings.getBlockCacheTraceDbAllowList()); + } + + @Test(expected = IllegalArgumentException.class) + public void testBlockCacheTraceAllowListRequiresCompleteConfiguration() { + StorageConfig.fromConfig(withRef( + "storage.dbSettings.blockCacheTraceDbAllowList = [account]")); + } + + @Test(expected = IllegalArgumentException.class) + public void testPinL0RequiresIndexAndFilterCache() { + StorageConfig.fromConfig(withRef( + "storage.dbSettings.cacheIndexAndFilterBlocks = false")); } @Test diff --git a/framework/src/main/java/org/tron/core/config/args/Args.java b/framework/src/main/java/org/tron/core/config/args/Args.java index 3946f0d764f..994b533d832 100644 --- a/framework/src/main/java/org/tron/core/config/args/Args.java +++ b/framework/src/main/java/org/tron/core/config/args/Args.java @@ -163,7 +163,7 @@ public static void setParam(final String[] args, final String confFileName) { // Resolve config file path configFilePath = StringUtils.isNoneBlank(cmd.shellConfFileName) ? cmd.shellConfFileName : confFileName; - Config config = Configuration.getByFileName(configFilePath); + Config config = Configuration.getByFileName(configFilePath, cmd.rocksDbConfigFileName); // 2. Config overrides defaults (event config bean is read here but not yet applied) applyConfigParams(config); @@ -226,12 +226,7 @@ private static void applyStorageConfig(StorageConfig sc) { // RocksDB settings StorageConfig.DbSettingsConfig dbs = sc.getDbSettings(); - PARAMETER.rocksDBCustomSettings = RocksDbSettings - .initCustomSettings(dbs.getLevelNumber(), dbs.getCompactThreads(), - dbs.getBlocksize(), dbs.getMaxBytesForLevelBase(), - dbs.getMaxBytesForLevelMultiplier(), dbs.getLevel0FileNumCompactionTrigger(), - dbs.getTargetFileSizeBase(), dbs.getTargetFileSizeMultiplier(), - dbs.getMaxOpenFiles()); + PARAMETER.rocksDBCustomSettings = RocksDbSettings.initCustomSettings(dbs); RocksDbSettings.loggingSettings(); // Dynamic nested objects use StorageConfig's raw storage sub-tree @@ -1299,7 +1294,7 @@ private static String getCommitIdAbbrev() { private static Map getOptionGroup() { String[] tronOption = new String[] {"version", "help", "shellConfFileName", "logbackPath", "eventSubscribe", "solidityNode", "keystoreFactory"}; - String[] dbOption = new String[] {"outputDirectory"}; + String[] dbOption = new String[] {"outputDirectory", "rocksDbConfigFileName"}; String[] witnessOption = new String[] {"witness", "privateKey"}; String[] vmOption = new String[] {"debug"}; diff --git a/framework/src/main/java/org/tron/core/config/args/CLIParameter.java b/framework/src/main/java/org/tron/core/config/args/CLIParameter.java index 4f056a32e3a..9284b5ad28f 100644 --- a/framework/src/main/java/org/tron/core/config/args/CLIParameter.java +++ b/framework/src/main/java/org/tron/core/config/args/CLIParameter.java @@ -22,6 +22,10 @@ public class CLIParameter { @Parameter(names = {"-c", "--config"}, description = "Config file (default:config.conf)") public String shellConfFileName; + @Parameter(names = "--rocksdb-config", + description = "RocksDB benchmark profile applied over storage.dbSettings") + public String rocksDbConfigFileName; + @Parameter(names = {"-d", "--output-directory"}, description = "Data directory for the " + "databases (default:output-directory)") public String outputDirectory; diff --git a/framework/src/main/java/org/tron/program/BlockReplay.java b/framework/src/main/java/org/tron/program/BlockReplay.java index a650501bc6e..d6cbfddba54 100644 --- a/framework/src/main/java/org/tron/program/BlockReplay.java +++ b/framework/src/main/java/org/tron/program/BlockReplay.java @@ -8,13 +8,16 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.ArrayList; import java.util.Arrays; +import java.util.List; import java.util.Locale; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.support.DefaultListableBeanFactory; import org.tron.common.application.TronApplicationContext; import org.tron.common.log.LogService; import org.tron.common.prometheus.Metrics; +import org.tron.common.setting.RocksDbSettings; import org.tron.common.utils.BlockFile; import org.tron.core.ChainBaseManager; import org.tron.core.capsule.BlockCapsule; @@ -71,7 +74,7 @@ static int execute(String[] args, PrintStream output, PrintStream error) { } private static ReplayResult verify(Options options) throws Exception { - Args.setParam(new String[] {"-c", options.config}, "config.conf"); + Args.setParam(nodeArgs(options, null), "config.conf"); return replay(Paths.get(options.input), null, null, options.warmupBlocks, options.maxBlocks, false); } @@ -82,8 +85,7 @@ private static ReplayResult apply(Options options) throws Exception { throw new IllegalArgumentException( "Output directory must be an existing D0 snapshot: " + outputDirectory); } - Args.setParam(new String[] {"-c", options.config, "-d", outputDirectory.toString(), - "--p2p-disable", "true"}, "config.conf"); + Args.setParam(nodeArgs(options, outputDirectory), "config.conf"); LogService.load(Args.getInstance().getLogbackPath()); Metrics.init(); @@ -106,6 +108,23 @@ static void startConsensus(TronApplicationContext context) { context.getBean(ConsensusService.class).start(); } + private static String[] nodeArgs(Options options, Path outputDirectory) { + List args = new ArrayList<>(); + args.add("-c"); + args.add(options.config); + if (options.rocksDbConfig != null) { + args.add("--rocksdb-config"); + args.add(options.rocksDbConfig); + } + if (outputDirectory != null) { + args.add("-d"); + args.add(outputDirectory.toString()); + args.add("--p2p-disable"); + args.add("true"); + } + return args.toArray(new String[0]); + } + static ReplayResult replay(Path input, TronNetDelegate tronNetDelegate, ChainBaseManager chainBaseManager, long warmupBlocks, long maxBlocks, boolean apply) throws Exception { @@ -171,6 +190,10 @@ static final class Options { @Parameter(names = {"-c", "--config"}, description = "Node config file.") private String config = "config.conf"; + @Parameter(names = "--rocksdb-config", + description = "RocksDB benchmark profile applied over storage.dbSettings.") + private String rocksDbConfig; + @Parameter(names = "--apply", description = "Apply blocks to D0. Without this flag the command only verifies the file.") private boolean apply; @@ -196,6 +219,9 @@ private void validate() { if (config == null || config.trim().isEmpty()) { throw new ParameterException("--config must not be empty"); } + if (rocksDbConfig != null && !Files.isRegularFile(Paths.get(rocksDbConfig))) { + throw new ParameterException("RocksDB profile does not exist: " + rocksDbConfig); + } if (apply && (outputDirectory == null || outputDirectory.trim().isEmpty())) { throw new ParameterException("--output-directory is required with --apply"); } @@ -229,14 +255,17 @@ private ReplayResult(boolean applied, long start, long end, long processed, } String format() { + RocksDbSettings rocksDbSettings = RocksDbSettings.getSettings(); double elapsedMs = measuredNanos / 1_000_000.0; double blocksPerSecond = measuredNanos == 0 ? 0.0 : measured * 1_000_000_000.0 / measuredNanos; return String.format(Locale.ROOT, - "mode=%s range=[%d,%d] processed=%d warmup=%d measured=%d " + "mode=%s rocksdb_profile=%s rocksdb_experiment=%s " + + "range=[%d,%d] processed=%d warmup=%d measured=%d " + "elapsed_ms=%.3f blocks_per_second=%.3f", - applied ? "apply" : "verify", start, end, processed, warmup, measured, - elapsedMs, blocksPerSecond); + applied ? "apply" : "verify", rocksDbSettings.getBenchmarkProfile(), + rocksDbSettings.getBenchmarkMode(), start, end, processed, warmup, + measured, elapsedMs, blocksPerSecond); } } } diff --git a/framework/src/main/java/org/tron/program/RocksDbBlockCacheTraceAnalyzer.java b/framework/src/main/java/org/tron/program/RocksDbBlockCacheTraceAnalyzer.java new file mode 100644 index 00000000000..470f468db08 --- /dev/null +++ b/framework/src/main/java/org/tron/program/RocksDbBlockCacheTraceAnalyzer.java @@ -0,0 +1,351 @@ +package org.tron.program; + +import com.beust.jcommander.JCommander; +import com.beust.jcommander.Parameter; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +/** Aggregates bounded per-database block-cache trace CSV files. */ +public final class RocksDbBlockCacheTraceAnalyzer { + + private RocksDbBlockCacheTraceAnalyzer() { + } + + public static void main(String[] args) throws Exception { + Arguments options = new Arguments(); + JCommander.newBuilder().addObject(options).build().parse(args); + analyze(Paths.get(options.input), Paths.get(options.output), options.startTimestampUs, + options.endTimestampUs); + } + + static void analyze(Path input, Path output) throws Exception { + analyze(input, output, 0, Long.MAX_VALUE); + } + + static void analyze(Path input, Path output, long startTimestampUs, long endTimestampUs) + throws Exception { + if (startTimestampUs < 0 || endTimestampUs <= startTimestampUs) { + throw new IllegalArgumentException("Invalid trace timestamp range"); + } + Files.createDirectories(output); + Map events = new HashMap<>(); + Map gets = new HashMap<>(); + try (Stream paths = Files.list(input)) { + List files = new ArrayList<>(); + paths.filter(p -> p.toString().endsWith(".csv")).sorted().forEach(files::add); + for (Path path : files) { + read(path, events, gets, startTimestampUs, endTimestampUs); + } + } + writeEvents(output.resolve("block-access.csv"), events); + writeGets(output.resolve("get-path.csv"), gets); + writeGetLevels(output.resolve("get-level.csv"), gets); + } + + private static void read(Path path, Map events, + Map gets, long startTimestampUs, long endTimestampUs) throws Exception { + String db = path.getFileName().toString().replaceFirst("\\.csv$", ""); + try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8)) { + reader.readLine(); + String line; + while ((line = reader.readLine()) != null) { + String[] values = splitCsv(line); + if (values.length != 12) { + throw new IllegalArgumentException("Invalid trace row in " + path + ": " + line); + } + long timestampUs = Long.parseUnsignedLong(values[0]); + if (timestampUs < startTimestampUs || timestampUs >= endTimestampUs) { + continue; + } + long getId = Long.parseUnsignedLong(values[1]); + int level = Integer.parseInt(values[2]); + int caller = Integer.parseInt(values[4]); + int blockType = Integer.parseInt(values[5]); + boolean cacheHit = "1".equals(values[6]); + boolean keyExists = "1".equals(values[8]); + long blockSize = Long.parseLong(values[9]); + EventKey eventKey = new EventKey(db, level, callerName(caller), blockName(blockType), + cacheHit ? "hit" : "miss"); + events.computeIfAbsent(eventKey, ignored -> new Aggregate()).add(blockSize); + if (getId != 0 && caller == 1) { + gets.computeIfAbsent(new GetKey(db, getId), ignored -> new GetAggregate()) + .add(level, blockType, cacheHit, keyExists, blockSize); + } + } + } + } + + private static String[] splitCsv(String line) { + List values = new ArrayList<>(); + StringBuilder value = new StringBuilder(); + boolean quoted = false; + for (int i = 0; i < line.length(); i++) { + char current = line.charAt(i); + if (current == '"') { + if (quoted && i + 1 < line.length() && line.charAt(i + 1) == '"') { + value.append('"'); + i++; + } else { + quoted = !quoted; + } + } else if (current == ',' && !quoted) { + values.add(value.toString()); + value.setLength(0); + } else { + value.append(current); + } + } + values.add(value.toString()); + return values.toArray(new String[0]); + } + + private static void writeEvents(Path path, Map values) throws Exception { + try (BufferedWriter writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) { + writer.write("db,level,caller,block_type,cache_result,accesses,bytes\n"); + values.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach(entry -> write(writer, + entry.getKey().csv() + "," + entry.getValue().count + "," + entry.getValue().bytes)); + } + } + + private static void writeGets(Path path, Map values) throws Exception { + Map summary = new LinkedHashMap<>(); + values.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach(entry -> { + GetAggregate get = entry.getValue(); + long[] counts = summary.computeIfAbsent(entry.getKey().db, ignored -> new long[10]); + counts[0]++; + counts[1] += get.dataCandidates; + counts[2] += get.found ? get.candidateMisses : 0; + counts[3] += get.found ? 1 : 0; + counts[4] += get.dataCacheMisses; + counts[5] += get.found ? 0 : get.candidateMisses; + counts[6] += get.avoidableCandidates(); + counts[7] += get.avoidableCacheMisses(); + counts[8] += get.found ? get.candidateCacheMisses : 0; + counts[9] += get.found ? 0 : get.candidateCacheMisses; + }); + try (BufferedWriter writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) { + writer.write("db,sampled_gets,data_candidates,upper_misses,trace_found_gets," + + "data_cache_misses,final_miss_candidates,avoidable_candidates," + + "avoidable_cache_misses,upper_cache_misses,final_cache_misses\n"); + summary.forEach((db, count) -> write(writer, db + "," + count[0] + "," + count[1] + + "," + count[2] + "," + count[3] + "," + count[4] + "," + count[5] + + "," + count[6] + "," + count[7] + "," + count[8] + "," + count[9])); + } + } + + private static void writeGetLevels(Path path, Map values) + throws Exception { + Map summary = new HashMap<>(); + values.forEach((getKey, get) -> get.candidates.forEach(candidate -> { + String outcome = candidate.keyExists ? "found" : get.found ? "upper_miss" : "final_miss"; + GetLevelKey key = new GetLevelKey(getKey.db, candidate.level, outcome, + candidate.cacheHit ? "hit" : "miss"); + summary.computeIfAbsent(key, ignored -> new Aggregate()).add(candidate.blockSize); + })); + try (BufferedWriter writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) { + writer.write("db,level,path_outcome,cache_result,accesses,bytes\n"); + summary.entrySet().stream().sorted(Map.Entry.comparingByKey()).forEach(entry -> write(writer, + entry.getKey().csv() + "," + entry.getValue().count + "," + entry.getValue().bytes)); + } + } + + private static void write(BufferedWriter writer, String value) { + try { + writer.write(value); + writer.newLine(); + } catch (Exception e) { + throw new IllegalStateException(e); + } + } + + private static String callerName(int caller) { + String[] names = {"reserved", "get", "multiget", "iterator", "approximate_size", + "verify_checksum", "sst_dump", "ingest", "repair", "prefetch", "compaction", + "compaction_refill", "flush", "sst_reader", "uncategorized"}; + return caller >= 0 && caller < names.length ? names[caller] : "caller_" + caller; + } + + private static String blockName(int type) { + switch (type) { + case 7: + return "index"; + case 8: + return "filter"; + case 9: + return "data"; + case 10: + return "compression_dict"; + case 11: + return "range_deletion"; + default: + return "block_" + type; + } + } + + private static final class Aggregate { + private long count; + private long bytes; + + void add(long size) { + count++; + bytes += size; + } + } + + private static final class GetAggregate { + private int dataCandidates; + private int candidateMisses; + private int candidateCacheMisses; + private int dataCacheMisses; + private boolean found; + private final List candidates = new ArrayList<>(); + + void add(int level, int blockType, boolean cacheHit, boolean keyExists, long blockSize) { + if (blockType != 9) { + return; + } + dataCandidates++; + candidates.add(new GetCandidate(level, cacheHit, keyExists, blockSize)); + if (!cacheHit) { + dataCacheMisses++; + } + if (keyExists) { + found = true; + } else { + candidateMisses++; + if (!cacheHit) { + candidateCacheMisses++; + } + } + } + + int avoidableCandidates() { + return candidateMisses; + } + + int avoidableCacheMisses() { + return candidateCacheMisses; + } + } + + private static final class GetCandidate { + private final int level; + private final boolean cacheHit; + private final boolean keyExists; + private final long blockSize; + + GetCandidate(int level, boolean cacheHit, boolean keyExists, long blockSize) { + this.level = level; + this.cacheHit = cacheHit; + this.keyExists = keyExists; + this.blockSize = blockSize; + } + } + + private static final class EventKey implements Comparable { + private final String db; + private final int level; + private final String caller; + private final String block; + private final String result; + + EventKey(String db, int level, String caller, String block, String result) { + this.db = db; + this.level = level; + this.caller = caller; + this.block = block; + this.result = result; + } + + String csv() { + return db + "," + level + "," + caller + "," + block + "," + result; + } + + public int compareTo(EventKey other) { + return csv().compareTo(other.csv()); + } + + public boolean equals(Object other) { + return other instanceof EventKey && compareTo((EventKey) other) == 0; + } + + public int hashCode() { + return csv().hashCode(); + } + } + + private static final class GetKey implements Comparable { + private final String db; + private final long id; + + GetKey(String db, long id) { + this.db = db; + this.id = id; + } + + public int compareTo(GetKey other) { + int comparison = db.compareTo(other.db); + return comparison != 0 ? comparison : Long.compareUnsigned(id, other.id); + } + + public boolean equals(Object other) { + return other instanceof GetKey && compareTo((GetKey) other) == 0; + } + + public int hashCode() { + return 31 * db.hashCode() + Long.hashCode(id); + } + } + + private static final class GetLevelKey implements Comparable { + private final String db; + private final int level; + private final String outcome; + private final String cacheResult; + + GetLevelKey(String db, int level, String outcome, String cacheResult) { + this.db = db; + this.level = level; + this.outcome = outcome; + this.cacheResult = cacheResult; + } + + String csv() { + return db + "," + level + "," + outcome + "," + cacheResult; + } + + public int compareTo(GetLevelKey other) { + return csv().compareTo(other.csv()); + } + + public boolean equals(Object other) { + return other instanceof GetLevelKey && compareTo((GetLevelKey) other) == 0; + } + + public int hashCode() { + return csv().hashCode(); + } + } + + static final class Arguments { + @Parameter(names = "--input", required = true) + private String input; + @Parameter(names = "--output", required = true) + private String output; + @Parameter(names = "--start-timestamp-us") + private long startTimestampUs; + @Parameter(names = "--end-timestamp-us") + private long endTimestampUs = Long.MAX_VALUE; + } +} diff --git a/framework/src/main/java/org/tron/program/RocksDbRebuild.java b/framework/src/main/java/org/tron/program/RocksDbRebuild.java new file mode 100644 index 00000000000..c82aeb4ec0f --- /dev/null +++ b/framework/src/main/java/org/tron/program/RocksDbRebuild.java @@ -0,0 +1,229 @@ +package org.tron.program; + +import com.beust.jcommander.JCommander; +import com.beust.jcommander.Parameter; +import com.beust.jcommander.ParameterException; +import java.io.PrintStream; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.rocksdb.ReadOptions; +import org.rocksdb.RocksDB; +import org.rocksdb.RocksIterator; +import org.tron.common.setting.RocksDbSettings; +import org.tron.common.storage.rocksdb.RocksDbDataSourceImpl; +import org.tron.core.config.args.Args; + +/** Rebuilds one RocksDB database so every output SST uses the active table options. */ +public final class RocksDbRebuild { + + private static final int BATCH_SIZE = 10_000; + + private RocksDbRebuild() { + } + + public static void main(String[] args) { + int exitCode = execute(args, System.out, System.err); + if (exitCode != 0) { + System.exit(exitCode); + } + } + + static int execute(String[] args, PrintStream output, PrintStream error) { + Options options = new Options(); + JCommander commander = JCommander.newBuilder() + .addObject(options) + .programName("RocksDbRebuild") + .build(); + try { + commander.parse(args); + if (options.help) { + commander.usage(); + return 0; + } + options.validate(); + Args.setParam(options.nodeArgs(), "config.conf"); + long start = System.nanoTime(); + long entries = options.compactExisting ? compactExisting(options) : rebuild(options); + double elapsedSeconds = (System.nanoTime() - start) / 1_000_000_000.0; + output.printf("rebuilt database=%s entries=%d elapsed_seconds=%.3f%n", + options.database, entries, elapsedSeconds); + return 0; + } catch (ParameterException | IllegalArgumentException e) { + error.println(e.getMessage()); + commander.usage(); + return 2; + } catch (Exception e) { + error.println("RocksDB rebuild failed: " + e.getMessage()); + return 1; + } finally { + Args.clearParam(); + } + } + + private static long rebuild(Options options) throws Exception { + RocksDbDataSourceImpl target = new RocksDbDataSourceImpl( + options.targetDirectory, options.database); + long entries = 0; + Map batch = new LinkedHashMap<>(BATCH_SIZE); + try (org.rocksdb.Options sourceOptions = RocksDbSettings.getOptionsByDbName(options.database); + RocksDB source = RocksDB.openReadOnly(sourceOptions, options.sourcePath().toString()); + ReadOptions readOptions = new ReadOptions().setFillCache(false); + RocksIterator iterator = source.newIterator(readOptions)) { + for (iterator.seekToFirst(); iterator.isValid(); iterator.next()) { + batch.put(iterator.key(), iterator.value()); + entries++; + if (batch.size() == BATCH_SIZE) { + target.updateByBatch(batch); + batch.clear(); + } + } + if (!batch.isEmpty()) { + target.updateByBatch(batch); + } + iterator.status(); + target.getDatabase().compactRange(); + verifySameContent(source, target, entries); + return entries; + } finally { + target.closeDB(); + } + } + + private static long compactExisting(Options options) throws Exception { + RocksDbDataSourceImpl target = new RocksDbDataSourceImpl( + options.targetDirectory, options.database); + try (org.rocksdb.Options sourceOptions = RocksDbSettings.getOptionsByDbName(options.database); + RocksDB source = RocksDB.openReadOnly(sourceOptions, options.sourcePath().toString())) { + forceCompactBottommost(target); + return verifySameContent(source, target, -1); + } finally { + target.closeDB(); + } + } + + @SuppressWarnings({"rawtypes", "unchecked"}) + private static void forceCompactBottommost(RocksDbDataSourceImpl target) throws Exception { + Class optionsClass = Class.forName("org.rocksdb.CompactRangeOptions"); + Class bottommostClass = (Class) Class.forName( + "org.rocksdb.CompactRangeOptions$BottommostLevelCompaction"); + Object options = optionsClass.getConstructor().newInstance(); + Object force = Enum.valueOf(bottommostClass, "kForce"); + try { + optionsClass.getMethod("setBottommostLevelCompaction", bottommostClass) + .invoke(options, force); + optionsClass.getMethod("setExclusiveManualCompaction", boolean.class) + .invoke(options, true); + optionsClass.getMethod("setMaxSubcompactions", int.class).invoke(options, 8); + Method compactRange = target.getDatabase().getClass().getMethod("compactRange", + Class.forName("org.rocksdb.ColumnFamilyHandle"), byte[].class, byte[].class, + optionsClass); + compactRange.invoke(target.getDatabase(), target.getDatabase().getDefaultColumnFamily(), + null, null, options); + } finally { + ((AutoCloseable) options).close(); + } + } + + private static long verifySameContent(RocksDB source, + RocksDbDataSourceImpl target, long expectedEntries) throws Exception { + long compared = 0; + try (ReadOptions sourceOptions = new ReadOptions().setFillCache(false); + ReadOptions targetOptions = new ReadOptions().setFillCache(false); + RocksIterator sourceIterator = source.newIterator(sourceOptions); + RocksIterator targetIterator = target.getDatabase().newIterator(targetOptions)) { + sourceIterator.seekToFirst(); + targetIterator.seekToFirst(); + while (sourceIterator.isValid() && targetIterator.isValid()) { + if (!Arrays.equals(sourceIterator.key(), targetIterator.key()) + || !Arrays.equals(sourceIterator.value(), targetIterator.value())) { + throw new IllegalStateException("Rebuilt content differs at entry " + compared); + } + compared++; + sourceIterator.next(); + targetIterator.next(); + } + sourceIterator.status(); + targetIterator.status(); + if (sourceIterator.isValid() || targetIterator.isValid() + || (expectedEntries >= 0 && compared != expectedEntries)) { + throw new IllegalStateException("Rebuilt entry count differs: copied=" + expectedEntries + + ", compared=" + compared); + } + return compared; + } + } + + static final class Options { + @Parameter(names = {"-s", "--source-directory"}, + description = "Existing directory containing the source database.") + private String sourceDirectory; + + @Parameter(names = {"-t", "--target-directory"}, + description = "Existing empty directory that will contain the rebuilt database.") + private String targetDirectory; + + @Parameter(names = "--database", description = "Database name to rebuild.") + private String database; + + @Parameter(names = "--compact-existing", + description = "Force-compact an existing target copy, including bottommost SST files.") + private boolean compactExisting; + + @Parameter(names = {"-c", "--config"}, description = "Node config file.") + private String config = "config.conf"; + + @Parameter(names = "--rocksdb-config", + description = "RocksDB profile to materialize into every output SST file.") + private String rocksDbConfig; + + @Parameter(names = {"-h", "--help"}, help = true) + private boolean help; + + private void validate() { + requireDirectory(sourceDirectory, "--source-directory"); + requireDirectory(targetDirectory, "--target-directory"); + if (database == null || database.trim().isEmpty()) { + throw new ParameterException("--database is required"); + } + Path source = Paths.get(sourceDirectory, database).toAbsolutePath().normalize(); + Path target = Paths.get(targetDirectory, database).toAbsolutePath().normalize(); + if (!Files.isRegularFile(source.resolve("CURRENT"))) { + throw new ParameterException("Existing RocksDB source is required: " + source); + } + if (compactExisting && !Files.isRegularFile(target.resolve("CURRENT"))) { + throw new ParameterException("Existing RocksDB target is required: " + target); + } + if (!compactExisting && Files.exists(target)) { + throw new ParameterException("Target database must not exist: " + target); + } + } + + private void requireDirectory(String value, String optionName) { + if (value == null || value.trim().isEmpty() || !Files.isDirectory(Paths.get(value))) { + throw new ParameterException(optionName + " must be an existing directory"); + } + } + + private String[] nodeArgs() { + List args = new ArrayList<>(); + args.add("-c"); + args.add(config); + if (rocksDbConfig != null) { + args.add("--rocksdb-config"); + args.add(rocksDbConfig); + } + return args.toArray(new String[0]); + } + + private Path sourcePath() { + return Paths.get(sourceDirectory, database).toAbsolutePath().normalize(); + } + } +} diff --git a/framework/src/test/java/org/tron/common/storage/metric/DbOperationMetricsTest.java b/framework/src/test/java/org/tron/common/storage/metric/DbOperationMetricsTest.java index 9476db1c6c6..63d840a27db 100644 --- a/framework/src/test/java/org/tron/common/storage/metric/DbOperationMetricsTest.java +++ b/framework/src/test/java/org/tron/common/storage/metric/DbOperationMetricsTest.java @@ -50,9 +50,35 @@ public void preBoundChildrenRecordLatencyAndBytes() { MetricKeys.Histogram.DB_OPERATE_BYTES + "_count", database), 0.0); } + @Test + public void preBoundChildrenRecordGetOutcome() { + CommonParameter.getInstance().setMetricsPrometheusEnable(true); + CommonParameter.getInstance().setMetricsPrometheusDatabaseEnable(true); + + String database = "outcome-test"; + DbOperationMetrics metrics = DbOperationMetrics.create("ROCKSDB", database); + double hitBefore = outcomeSample(database, "hit"); + double missBefore = outcomeSample(database, "miss"); + + metrics.observeGetOutcome(true); + metrics.observeGetOutcome(false); + metrics.observeGetOutcome(false); + + assertEquals(hitBefore + 1, outcomeSample(database, "hit"), 0.0); + assertEquals(missBefore + 2, outcomeSample(database, "miss"), 0.0); + } + private Double sample(String metric, String database) { Double value = CollectorRegistry.defaultRegistry.getSampleValue(metric, new String[]{"type", "db", "op"}, new String[]{"ROCKSDB", database, "get"}); return value == null ? 0.0 : value; } + + private double outcomeSample(String database, String outcome) { + Double value = CollectorRegistry.defaultRegistry.getSampleValue( + MetricKeys.Counter.DB_GET + "_total", + new String[]{"type", "db", "outcome"}, + new String[]{"ROCKSDB", database, outcome}); + return value == null ? 0.0 : value; + } } diff --git a/framework/src/test/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImplTest.java b/framework/src/test/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImplTest.java index af70837a966..d7e719e38e5 100644 --- a/framework/src/test/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImplTest.java +++ b/framework/src/test/java/org/tron/common/storage/rocksdb/RocksDbDataSourceImplTest.java @@ -10,6 +10,7 @@ import java.io.IOException; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.Collections; import org.junit.AfterClass; import org.junit.Assert; import org.junit.BeforeClass; @@ -158,6 +159,7 @@ public void exportsRocksDbStatisticsWhenDatabaseMetricsEnabled() { dataSource.putData(key1, value1); dataSource.getData(key1); + dataSource.getData("missing-key".getBytes()); dataSource.stat(); Double writes = CollectorRegistry.defaultRegistry.getSampleValue( @@ -166,6 +168,8 @@ public void exportsRocksDbStatisticsWhenDatabaseMetricsEnabled() { new String[]{"ROCKSDB", database, "number_keys_written"}); Assert.assertNotNull(writes); Assert.assertTrue(writes >= 1); + Assert.assertEquals(1.0, getOutcome(database, "hit"), 0.0); + Assert.assertEquals(1.0, getOutcome(database, "miss"), 0.0); } finally { if (dataSource != null) { dataSource.closeDB(); @@ -176,6 +180,44 @@ public void exportsRocksDbStatisticsWhenDatabaseMetricsEnabled() { } } + @Test + public void skipsPerfContextSamplingWhenLegacyJniDoesNotExposeIt() { + CommonParameter parameter = CommonParameter.getInstance(); + boolean prometheusEnabled = parameter.isMetricsPrometheusEnable(); + boolean databaseMetricsEnabled = parameter.isMetricsPrometheusDatabaseEnable(); + RocksDbSettings settings = RocksDbSettings.getSettings(); + int sampleOneIn = settings.getPerfContextSampleOneIn(); + java.util.List allowList = settings.getPerfContextDbAllowList(); + RocksDbDataSourceImpl dataSource = null; + try { + parameter.setMetricsPrometheusEnable(true); + parameter.setMetricsPrometheusDatabaseEnable(true); + settings.withPerfContextSampleOneIn(1) + .withPerfContextDbAllowList(Collections.singletonList("legacy-perf-context")); + dataSource = new RocksDbDataSourceImpl( + Args.getInstance().getOutputDirectory(), "legacy-perf-context"); + + dataSource.putData(key1, value1); + Assert.assertArrayEquals(value1, dataSource.getData(key1)); + } finally { + if (dataSource != null) { + dataSource.closeDB(); + } + settings.withPerfContextSampleOneIn(sampleOneIn) + .withPerfContextDbAllowList(allowList); + parameter.setMetricsPrometheusDatabaseEnable(databaseMetricsEnabled); + parameter.setMetricsPrometheusEnable(prometheusEnabled); + } + } + + private double getOutcome(String database, String outcome) { + Double value = CollectorRegistry.defaultRegistry.getSampleValue( + MetricKeys.Counter.DB_GET + "_total", + new String[]{"type", "db", "outcome"}, + new String[]{"ROCKSDB", database, outcome}); + return value == null ? 0.0 : value; + } + private void makeExceptionDb(String dbName) { RocksDbDataSourceImpl dataSource = new RocksDbDataSourceImpl( Args.getInstance().getOutputDirectory(), "test_initDb"); diff --git a/framework/src/test/java/org/tron/core/config/ConfigurationTest.java b/framework/src/test/java/org/tron/core/config/ConfigurationTest.java index b066bc1e6be..39efa970b52 100644 --- a/framework/src/test/java/org/tron/core/config/ConfigurationTest.java +++ b/framework/src/test/java/org/tron/core/config/ConfigurationTest.java @@ -26,11 +26,17 @@ import com.typesafe.config.Config; import java.lang.reflect.Field; import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Arrays; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; import org.bouncycastle.util.encoders.Hex; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; import org.tron.common.TestConstants; import org.tron.common.crypto.ECKey; import org.tron.common.utils.ByteArray; @@ -39,6 +45,9 @@ @Slf4j public class ConfigurationTest { + @Rule + public final TemporaryFolder temporaryFolder = new TemporaryFolder(); + @Before public void resetSingleton() throws SecurityException, NoSuchFieldException, IllegalArgumentException, @@ -91,4 +100,65 @@ public void getConfigurationWhenOnlyConfFileName() { assertTrue(config.hasPath("seed.node")); assertTrue(config.hasPath("genesis.block")); } + + @Test + public void shouldMergeOnlyRocksDbProfileSettings() throws Exception { + Path profile = temporaryFolder.newFile("cache-profile.conf").toPath(); + Files.write(profile, Arrays.asList( + "rocksdb-profile {", + " name = cache-2g", + " mode = E1", + " settings { blockCacheSize = 2048, maxOpenFiles = 3000 }", + "}", + "node.listen.port = 1"), StandardCharsets.UTF_8); + + Config config = Configuration.getByFileName(TestConstants.TEST_CONF, profile.toString()); + + assertEquals("cache-2g", + config.getString("storage.dbSettings.benchmarkProfile")); + assertEquals("E1", config.getString("storage.dbSettings.benchmarkMode")); + assertFalse(config.getBoolean("storage.dbSettings.useLegacyOptions")); + assertEquals(2048, config.getInt("storage.dbSettings.blockCacheSize")); + assertEquals(3000, config.getInt("storage.dbSettings.maxOpenFiles")); + assertEquals(16, config.getInt("storage.dbSettings.blocksize")); + assertTrue(config.getInt("node.listen.port") != 1); + } + + @Test + public void shouldAllowProfileToSelectLegacyOptions() throws Exception { + Path profile = temporaryFolder.newFile("legacy-profile.conf").toPath(); + Files.write(profile, Arrays.asList( + "rocksdb-profile {", + " name = a1-legacy-options", + " mode = E1", + " settings.useLegacyOptions = true", + "}"), StandardCharsets.UTF_8); + + Config config = Configuration.getByFileName(TestConstants.TEST_CONF, profile.toString()); + + assertTrue(config.getBoolean("storage.dbSettings.useLegacyOptions")); + assertEquals("a1-legacy-options", + config.getString("storage.dbSettings.benchmarkProfile")); + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectIncompleteRocksDbProfile() throws Exception { + Path profile = temporaryFolder.newFile("incomplete-profile.conf").toPath(); + Files.write(profile, Arrays.asList( + "rocksdb-profile.name = incomplete", + "rocksdb-profile.settings.blockCacheSize = 2048"), StandardCharsets.UTF_8); + + Configuration.getByFileName(TestConstants.TEST_CONF, profile.toString()); + } + + @Test(expected = IllegalArgumentException.class) + public void shouldRejectUnknownRocksDbExperimentMode() throws Exception { + Path profile = temporaryFolder.newFile("unknown-mode.conf").toPath(); + Files.write(profile, Arrays.asList( + "rocksdb-profile.name = candidate", + "rocksdb-profile.mode = E3", + "rocksdb-profile.settings.blockCacheSize = 2048"), StandardCharsets.UTF_8); + + Configuration.getByFileName(TestConstants.TEST_CONF, profile.toString()); + } } diff --git a/framework/src/test/java/org/tron/program/BlockReplayTest.java b/framework/src/test/java/org/tron/program/BlockReplayTest.java index 156839316fa..ce073e6d741 100644 --- a/framework/src/test/java/org/tron/program/BlockReplayTest.java +++ b/framework/src/test/java/org/tron/program/BlockReplayTest.java @@ -13,17 +13,21 @@ import com.google.protobuf.ByteString; import java.io.ByteArrayOutputStream; import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import java.nio.file.Path; import org.junit.Rule; import org.junit.Test; import org.junit.rules.TemporaryFolder; import org.mockito.ArgumentCaptor; import org.tron.common.application.TronApplicationContext; +import org.tron.common.setting.RocksDbSettings; import org.tron.common.utils.BlockFile; import org.tron.common.utils.Sha256Hash; import org.tron.core.ChainBaseManager; import org.tron.core.capsule.BlockCapsule; import org.tron.core.capsule.BlockCapsule.BlockId; +import org.tron.core.config.args.StorageConfig.DbSettingsConfig; import org.tron.core.consensus.ConsensusService; import org.tron.core.net.TronNetDelegate; @@ -61,6 +65,29 @@ public void shouldVerifyThroughCommandLine() throws Exception { assertTrue(output.toString().contains("processed=2")); } + @Test + public void shouldApplyRocksDbProfileThroughCommandLine() throws Exception { + BlockId parent = new BlockId(Sha256Hash.ZERO_HASH, 9); + Path input = write(blocks(parent, 10, 1)); + Path profile = temporaryFolder.getRoot().toPath().resolve("profile.conf"); + Files.write(profile, ("rocksdb-profile { name = replay-profile, mode = E1, " + + "settings.blockCacheSize = 1024 }").getBytes(StandardCharsets.UTF_8)); + ByteArrayOutputStream output = new ByteArrayOutputStream(); + ByteArrayOutputStream error = new ByteArrayOutputStream(); + + try { + int exitCode = BlockReplay.execute(new String[] {"--input", input.toString(), + "--rocksdb-config", profile.toString()}, + new PrintStream(output), new PrintStream(error)); + + assertEquals(error.toString(), 0, exitCode); + assertTrue(output.toString().contains("rocksdb_profile=replay-profile")); + assertTrue(output.toString().contains("rocksdb_experiment=E1")); + } finally { + RocksDbSettings.initCustomSettings(new DbSettingsConfig()); + } + } + @Test public void shouldApplyBlocksThroughSyncPath() throws Exception { BlockId parent = new BlockId(Sha256Hash.ZERO_HASH, 9); diff --git a/framework/src/test/java/org/tron/program/RocksDbBlockCacheTraceAnalyzerTest.java b/framework/src/test/java/org/tron/program/RocksDbBlockCacheTraceAnalyzerTest.java new file mode 100644 index 00000000000..ae0177397a9 --- /dev/null +++ b/framework/src/test/java/org/tron/program/RocksDbBlockCacheTraceAnalyzerTest.java @@ -0,0 +1,52 @@ +package org.tron.program; + +import static org.junit.Assert.assertTrue; + +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class RocksDbBlockCacheTraceAnalyzerTest { + @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder(); + + @Test + public void aggregatesLevelsAndGetPaths() throws Exception { + Path input = temporaryFolder.newFolder("input").toPath(); + Path output = temporaryFolder.newFolder("output").toPath(); + Files.write(input.resolve("account.csv"), ("timestamp_us,get_id,level,sst_file,caller," + + "block_type,cache_hit,no_insert,key_exists,block_size,cf_id,cf_name\n" + + "1,42,1,10,1,9,0,0,0,4096,0,default\n" + + "2,42,3,11,1,9,1,0,1,4096,0,default\n" + + "3,0,2,12,10,9,0,0,0,8192,0,default\n").getBytes(StandardCharsets.UTF_8)); + RocksDbBlockCacheTraceAnalyzer.analyze(input, output); + String gets = new String(Files.readAllBytes(output.resolve("get-path.csv")), + StandardCharsets.UTF_8); + assertTrue(gets.contains("account,1,2,1,1,1,0,1,1,1,0")); + String blocks = new String(Files.readAllBytes(output.resolve("block-access.csv")), + StandardCharsets.UTF_8); + assertTrue(blocks.contains("account,1,get,data,miss,1,4096")); + assertTrue(blocks.contains("account,2,compaction,data,miss,1,8192")); + String levels = new String(Files.readAllBytes(output.resolve("get-level.csv")), + StandardCharsets.UTF_8); + assertTrue(levels.contains("account,1,upper_miss,miss,1,4096")); + assertTrue(levels.contains("account,3,found,hit,1,4096")); + } + + @Test + public void filtersByHalfOpenTimestampRange() throws Exception { + Path input = temporaryFolder.newFolder("range-input").toPath(); + Path output = temporaryFolder.newFolder("range-output").toPath(); + Files.write(input.resolve("account.csv"), ("timestamp_us,get_id,level,sst_file,caller," + + "block_type,cache_hit,no_insert,key_exists,block_size,cf_id,cf_name\n" + + "9,41,1,10,1,9,0,0,0,4096,0,default\n" + + "10,42,2,11,1,9,0,0,1,4096,0,default\n" + + "20,43,3,12,1,9,0,0,1,4096,0,default\n").getBytes(StandardCharsets.UTF_8)); + RocksDbBlockCacheTraceAnalyzer.analyze(input, output, 10, 20); + String gets = new String(Files.readAllBytes(output.resolve("get-path.csv")), + StandardCharsets.UTF_8); + assertTrue(gets.contains("account,1,1,0,1,1,0,0,0,0,0")); + } +}