fix(parser): PDF/DOCX 切片层级结构修复与 MinerU V2 解析增强

核心修复:
- _post_process_chunks: 用 _buffer_has_body 标志替代 buffer.text_level=0,
  标题+正文合并后保留 text_level 不置零,修复层级信息丢失
- heading_rules numeric_level2: 正则从 ^\d+\.\d+[\.、\s] 改为
  ^\d+\.\d+(?!\.\d),修复无空格标题如"2.1运行调度"无法匹配
- V2 title handler: heading_rules 优先(模式匹配可靠),VLM 仅兜底
  (VLM 常给所有标题 level=1)

MinerU V2 解析增强:
- 两轮 TOC 过滤:多行目录块检测 + 孤立标题/单字符残留清理
- 封面 logo 过滤、重复标题去重
- chart VLM 描述和 Markdown 数据表提取
- ChromaDB metadata 新增 text_level/bbox/table_type/sub_type 字段

配置调整:
- CLUSTER_SECTION_PREFIX_LEVELS 1→2(两级 section 聚类更精确)
- MINERU_LOCAL_BACKEND 默认改为 pipeline

文档:
- RAG检索流程逻辑.md 更新 MinerUChunk 字段、ChromaDB metadata、
  MinerU 解析策略等章节
- 新增 RAG引用跳转-优化计划.md、云端MinerU输出分析与优化方案.md
This commit is contained in:
lacerate551
2026-06-19 23:56:13 +08:00
parent e205eda6a8
commit ee5295cc97
7 changed files with 1193 additions and 33 deletions

View File

@@ -335,6 +335,7 @@ class KnowledgeBaseManager(
"section": section_path,
"status": "active",
"version": "v1",
"text_level": getattr(chunk, 'text_level', 0), # 标题级别(0=正文,1=h1,2=h2,3=h3),供检索管线层次感知
}
if extra_metadata:
@@ -347,6 +348,25 @@ class KnowledgeBaseManager(
if hasattr(chunk, 'image_path') and chunk.image_path:
metadata['image_path'] = chunk.image_path
# bbox 坐标PDF 有DOCX 无,需 None 保护)
# _build_citation() 从 metadata 读取 bbox 做引用定位
chunk_bbox = getattr(chunk, 'bbox', None)
if chunk_bbox:
metadata['bbox'] = json.dumps(chunk_bbox)
# MinerU 结构化元数据(表格类型、嵌套层级、图片子类型)
chunk_table_type = getattr(chunk, 'table_type', '')
if chunk_table_type:
metadata['table_type'] = chunk_table_type
chunk_nest_level = getattr(chunk, 'table_nest_level', '')
if chunk_nest_level:
metadata['table_nest_level'] = str(chunk_nest_level)
chunk_sub_type = getattr(chunk, 'sub_type', '')
if chunk_sub_type:
metadata['sub_type'] = chunk_sub_type
# 生成向量
try:
embedding = embedding_model.encode(semantic_content).tolist()