Skip to content

feat(database): Supports CSV import and semantic layer analysis - #183

Merged
mengnankkkk merged 10 commits into
mainfrom
feature/add_csv
Nov 14, 2026
Merged

mengnankkkk merged 10 commits into
mainfrom
feature/add_csv

Conversation

@mengnankkkk

Copy link
Copy Markdown
Member
  • 区分 CSV 托管来源与 JDBC 引擎,复用 MySQL 数据源和分析链路
  • 增加上传预览、字段类型确认、异步批量导入及原子语义发布
  • 支持失败重试、删除补偿、任务恢复和过期文件清理
  • 在查询及元数据入口校验表归属和就绪状态,沿用角色表权限
  • 增加前端 CSV 分析空间、MySQL 5.7 兼容迁移及 Docker 配置说明

验证:

  • 后端主代码及既有测试编译通过,Checkstyle 和本次 Java 改动的 Spotless 检查通过
  • 前端类型检查、生产构建及本次改动的 ESLint 检查通过
  • MySQL 8.0 独立临时库验证导入、精确聚合、权限隔离、失败重试和删除
  • 迁移脚本重复执行通过,未新增或执行单测,仅同步既有测试构造依赖
  • MySQL 5.7 尚未实机验证;启用步骤见 docs/csv-import.md

Refs #165

- 区分 CSV 托管来源与 JDBC 引擎,复用 MySQL 数据源和分析链路
- 增加上传预览、字段类型确认、异步批量导入及原子语义发布
- 支持失败重试、删除补偿、任务恢复和过期文件清理
- 在查询及元数据入口校验表归属和就绪状态,沿用角色表权限
- 增加前端 CSV 分析空间、MySQL 5.7 兼容迁移及 Docker 配置说明

验证:
- 后端主代码及既有测试编译通过,Checkstyle 和本次 Java 改动的 Spotless 检查通过
- 前端类型检查、生产构建及本次改动的 ESLint 检查通过
- MySQL 8.0 独立临时库验证导入、精确聚合、权限隔离、失败重试和删除
- 迁移脚本重复执行通过,未新增或执行单测,仅同步既有测试构造依赖
- MySQL 5.7 尚未实机验证;启用步骤见 docs/csv-import.md

Refs #165
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

Thank you for your contribution! We will review your request as soon as possible. Please review the code yourself using ponytail https://github.com/DietrichGebert/ponytail.

Preserve the application executor and align schema inference with empty-value handling. Isolate import attempts, bind retries to the submitting user, and guard publishing, deletion, and orphan cleanup against stale workers.

Validate metric table references before deletion, share semantic conversion, and ignore stale upload and pagination responses. Make concurrent confirmations idempotent and return HTTP 413 for oversized multipart uploads.

Document dedicated CSV database setup and upgrade requirements. Verified with 57 live integration checks and a successful backend build.
Extract shared datasource validation and inject the CSV executor by name. Generate preview metadata mappings with MapStruct while keeping task lifecycle initialization in the service.

Remove the exception factory from the schema converter and introduce CSV-specific error codes through the existing exception handling flow. Document error codes and HTTP status mappings.

Validated with a successful backend build and 24 temporary Spring context checks.
Centralize date parsing, the default timezone, and MySQL date range validation. Reuse these rules in CSV inference and conversion, the date tool, and agent prompts.

Extract physical naming rules, organize worker execution and cleanup into focused methods, and tighten frontend column and task status types.

Validated backend and frontend builds, lint, 9 existing date tool tests, 34 date and naming checks, 24 Spring context checks, and 47 live API checks.
Move shared table and column snapshots into model.bo and update semantic sync consumers. Deep-clone dialog input to keep nested edits independent of the parent list.

Persist the final row count and DATA_READY state in one guarded update. Reuse UTF-8 byte counts from record validation when sizing import batches.

Validated backend and frontend builds, lint, the existing semantic sync test, and focused checks for editor isolation, stale attempts, atomic completion, and byte limits.
Rename shared schema snapshots with the Bo suffix and use MapStruct for mechanical metadata mappings while keeping business defaults in services. Reuse existing datasource enums and retain contextual asynchronous failure diagnostics before compensation.
Separate upload forms and cleanup responsibilities, reuse JSON utilities, and preserve local date and time values. Keep cleanup failures polling, serialize deletion and reference writes, and protect semantic refreshes from concurrent publication.
Accept Tab delimiters and reject cross-database metric references. Add datasource-aware semantic navigation and isolate state when switching sources.

Remove the single-datasource chat selector, clear CSV configuration defaults, and update portable setup documentation.
@mengnankkkk
mengnankkkk merged commit ddcc6c2 into main Nov 14, 2026
5 checks passed
@mengnankkkk
mengnankkkk deleted the feature/add_csv branch November 14, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant