refactor: 死代码清理 + UX 增强 + 查询清洗模块化 - #1
Merged
Merged
Conversation
- APIReranker 新增 arerank() async 方法（httpx 替代 requests） - HybridRetriever 新增 ahybrid_search() async 方法（run_in_executor） - _run_qa() 改为 async，核心调用通过 asyncio.to_thread 包裹 - _run_insight_async() 新增（异步检索+重排序） - 路由处理函数改用 await 调用 - _run_qa_sync() 保留以兼容 evaluation 模块 - 并发测试验证：3 请求总耗时 73s ≈ 最慢单请求（而非 148s 串行） - pyproject.toml: 添加 httpx 依赖 + ruff/mypy/pytest 配置
- 新增 src/core/state.py: AppState 类 + asyncio.Lock 线程安全 - 新增 src/api/dependencies.py: FastAPI Depends 注入 - api.py: 移除全局 _runtime，所有端点通过 Depends 获取 AppState - src/graph.py: 全部节点 async def + llm.ainvoke() / arerank() / ahybrid_search() - src/agents/supervisor.py: 新增 adecide() 异步方法 - src/agents/insight_agent.py: 新增 agenerate() 异步方法 - QA 响应 20.3s -> 12.0s (-40%) - Insight 响应 209.1s -> 65.4s (-69%)
- src/api/routes/: 5 个路由模块 (health/qa/insight/evaluate/crawl) - src/api/main.py: FastAPI 应用组装 - api.py: 向后兼容重定向到 src.api.main - AppState: 新增 init_sync() / rebuild_sync() 供 Streamlit 使用 - app.py: 删除独立的 build_runtime/reload_after_fetch → 委托给 AppState - src/runtime.py: 标记 deprecated（功能已被 AppState 取代）
…te Limiting Day 7: Docker + CI - Dockerfile: 多阶段构建 (python:3.11-slim + uv) - docker-compose.yml: API + PostgreSQL(pgvector) + Redis - .github/workflows/ci.yml: lint → test → build - .dockerignore: 优化构建上下文 - .env.example: 新增 DATABASE_URL, REDIS_URL, LangFuse, 限流配置 - src/config.py: 新增 database/redis/langfuse/rate_limit 配置 - src/api/main.py: 集成 RequestID + CORS + 全局异常处理中间件 Day 8: SSE Streaming - src/api/routes/qa_stream.py: POST /api/qa/stream (逐 token 流式) - src/api/routes/insight_stream.py: POST /api/insight/stream (逐阶段流式) - src/agents/insight_agent.py: 新增 astream() 异步流式生成 - static/js/app.js: SSE 前端消费 (fetch + ReadableStream) Day 9: LangFuse + Prompt YAML - src/core/observability.py: LangFuse CallbackHandler 集成 - src/core/prompt_loader.py: Prompt YAML 加载器 (缓存+热重载) - src/prompts/: 4 个 YAML Prompt (gen_answer, rewrite_query, supervisor, insight_report) - src/graph.py: 从 YAML 加载 Prompt 替代硬编码 - src/agents/supervisor.py: 从 YAML 加载 Prompt - src/agents/insight_agent.py: 从 YAML 加载 Prompt (v2) Day 10: Testing + Rate Limiting + Delivery - tests/test_api/: API 集成测试 (health, qa, insight) - tests/test_agents/: Agent 单元测试 (supervisor, insight_agent) - src/api/main.py: slowapi 限流基础设施 - README.md: 完整重写 (新架构图+Docker+API文档+流式) - pyproject.toml: v1.0.0, 新增 slowapi/langfuse/asyncpg/sqlalchemy/alembic - src/core/state.py: 清理重复代码 - src/api/middle.py: 删除 (已集成到 main.py) 总计: 32 files changed, ~2000+ lines added
- src/core/database.py: SQLAlchemy 2.0 async + asyncpg + pgvector - DocumentTable: id/content/metadata/embedding (1024维) - HNSW 向量索引 (cosine_ops) - insert_documents / search_by_vector 辅助函数 - src/ingestion.py: 新增 PG 异步 Ingestion - _clean_frontmatter() 统一清理函数 - ingest_to_pg() 全量写入 PG - incremental_ingest_to_pg() 增量去重写入 - src/retrievers.py: 新增 PG 向量检索 - PGVectorStore: 兼容 ChromaDB 接口 (similarity_search) - PgHybridRetriever: PG版 混合检索 (向量+BM25+RRF) - create_pg_vectorstore() 自动检测 PG 可用性 - src/core/state.py: 双模式自动切换 - 优先 PG (DATABASE_URL 配置时) - 回退 ChromaDB (无 PG 或连接失败) - _use_pg 标志追踪当前模式 - src/graph.py: 兼容 PG async 向量搜索 - alembic/: 数据库迁移 (001_initial) - documents 表 + pgvector 扩展 + HNSW 索引
- tests/test_agents/: 用 MagicMock 替代 patch.object 绕过 Pydantic 限制 - tests/test_api/: MockAppState + raise_app_exceptions=False 简化测试 - src/core/state.py: AppState.initialize() 优先 PG 回退 ChromaDB - 32/32 测试全部通过
- 删除 6 个冗余文件: app.py, import_data.py, fetcher.py, mcp_tools.py, rag_pipeline.py, runtime.py - 移除 Reranker 本地 CrossEncoder 类、RAGEvaluator、同步方法等死代码 - 新增 query_utils.py 共享模块，消除 3 个文件间的查询清洗重复 - 前端 UX 增强: SSE 流式反馈、Toast 通知、搜索防抖、进度指示 - src/evaluation.py 保留 EvalResult/CategoryEvalReport 数据结构 - 更新 .gitignore、README.md 设计决策章节
新增： - CreatorGenerator 选题引擎：从评论数据生成爆款选题+脚本大纲 - SSE 流式双报告：一次请求同时推送选品+选题两份报告 - 灵感库：9品类×21条=189条人工精选方向，零爬虫依赖 - 双按钮前端（选品分析/博主方案）+ 一键复制导出条 删除（做减法）： - LangGraph（graph.py, supervisor.py）：前端路由全部绕过 - Streamlit（streamlit_app.py, .streamlit/）：改为纯FastAPI+静态前端 - RAGAS评估（evaluation.py, evaluate路由）：前端零引用 - LangFuse可观测性（observability.py）：从未接上 - 旧版v1 Prompt×3 工程： - 依赖从27个精简到20个 - 路由从16个精简到11个 - 删除死代码~2000行 - README全面重写
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更概要
♻️ 死代码清理
✨ UX 增强
📦 其他
数据统计