jhzhang/rag
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(boundary): 修复多库边界问题、版本管理及删除清理

Browse Source 
多库检索与存储修复:
- RRF 融合去重改用 (collection, chunk_id) 复合键，修复同名文件结果被吞
- DocStore 存储路径加 collection 前缀，修复跨库同名切片数据覆盖
- search_multiple 去重改用复合键
- chunk_id 解析改用 rsplit 兼容下划线文件名

上传与版本管理修复:
- 同名文件上传改为覆盖模式，自动清理旧切片
- 修复首次上传不创建版本记录
- 修复覆盖上传版本号回退到 v1
- sync ADDED 分支改用动态版本号生成
- _generate_version_id 改为基于全部版本递增
- 废止/恢复操作同步 SQLite 版本记录
- mark_document_as_superseded 改为仅更新 SQLite

删除清理修复:
- 删除文档时同步清理 SQLite 版本记录和变更日志
- 删除向量库时同步清理该库所有版本记录
- cleanup 改为清理 SQLite 记录而非 ChromaDB

测试:
- test_version_management.py: 27 条版本管理单元测试
- test_edge_cases.py: 28 条边界用例测试
- test_upload_dedup.py: 5 条上传去重测试
- e2e_risk_test.py: 27 条端到端风险测试

文档:
- 新增风险边界问题修复注意事项.md（面向后端的对接文档）
- 新增向量库边界风险分析.md
- 更新多篇现有文档
This commit is contained in:
lacerate551
2026-06-04 23:58:44 +08:00
parent a1a0814633
commit cb75b9b274
50 changed files with 6385 additions and 6248 deletions
Download Patch File Download Diff File Expand all files Collapse all files

467
docs/MinerU模型部署指南.md
View File

@@ -35,7 +35,11 @@ MinerU 配置文件查找顺序：
**Windows**：`C:\Users\<username>\mineru.json`
**Linux**：`/root/mineru.json` 或 `/home/<user>/mineru.json`
### 1.3 当前项目使用方式
### 1.3 MinerU 在线 API 模式
项目同时支持 MinerU 在线 API 解析，通过 `.env.production` 中的 `MINERU_API_TOKEN` 环境变量配置。当设置了该 Token 时，可直接调用 OpenDataLab 云端 API 进行文档解析，无需在本地部署模型。
### 1.4 当前项目使用方式
查看 `parsers/mineru_parser.py` 第 186-197 行：
@@ -52,9 +56,10 @@ cmd = [
```
**关键发现**：
- ✅ 代码中**没有硬编码路径**
- ✅ 使用命令行调用 `mineru` 可执行文件
- ✅ MinerU 自动读取配置文件或环境变量
- 代码中**没有硬编码路径**
- 使用命令行调用 `mineru` 可执行文件
- MinerU 自动读取配置文件或环境变量
- 所有配置均通过 `.env.production` 环境变量注入，不依赖 `config.py` 硬编码
---
@@ -81,7 +86,7 @@ cmd = [
## 三、解决方案
### 方案 A：本地模型模式（推荐）✅
### 方案 A：本地模型模式（推荐）
**适用场景**：
- 服务器无法访问 HuggingFace
@@ -118,60 +123,90 @@ mineru-models-download -s huggingface -m all -d models\mineru
"models-dir": {
"pipeline": "/app/models/mineru/pipeline",
"vlm": "/app/models/mineru/vlm"
}
},
"config_version": "1.3.1"
}
```
**注意**：路径使用 Docker 容器内的路径 `/app/`。
#### Step 3：修改 Dockerfile
#### Step 3：生产环境 Dockerfile
当前项目使用 `deploy/Dockerfile.prod`（基于 Python 3.10-slim，CPU-only PyTorch）：
```dockerfile
# Dockerfile
# Dockerfile.prod - 生产环境优化版
# ================================
# 特点：CPU-only、精简依赖、最小化镜像
FROM python:3.10-slim
WORKDIR /app
# 系统依赖
RUN apt-get update && apt-get install -y \
# 使用阿里云镜像源
RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list.d/debian.sources \
&& sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list.d/debian.sources
# 系统依赖（精简版）
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
poppler-utils \
libmagic1 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Python依赖
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install gunicorn>=21.0.0
# ==================== PyTorch（CPU 模式） ====================
# 先从阿里云安装 PyTorch 依赖（避免从 PyPI 下载超时）
RUN pip install --no-cache-dir networkx sympy mpmath typing-extensions \
-i https://mirrors.aliyun.com/pypi/simple/
# 再从 PyTorch 官方源下载 CPU-only 版本（约 200MB），跳过依赖解析
RUN pip install --no-cache-dir --no-deps torch --index-url https://download.pytorch.org/whl/cpu
# 应用代码
# 设置 PyTorch 使用 CPU 模式
ENV CUDA_VISIBLE_DEVICES=""
# ==================== Python 依赖 ====================
COPY requirements-prod.txt .
RUN pip install --no-cache-dir -r requirements-prod.txt \
-i https://mirrors.aliyun.com/pypi/simple/
# ==================== 应用代码 ====================
COPY . .
# 复制模型文件（重要！）
COPY models /app/models
# ==================== MinerU 配置 ====================
# 生产环境使用本地模型
RUN mkdir -p /root && echo '{\n\
"models-dir": {\n\
"pipeline": "/app/models/mineru/pipeline",\n\
"vlm": "/app/models/mineru/vlm"\n\
},\n\
"config_version": "1.3.1"\n\
}' > /root/mineru.json
# 复制配置文件到容器内用户目录
COPY mineru.json /root/mineru.json
# 设置环境变量
ENV MINERU_MODEL_SOURCE=local
ENV MINERU_TOOLS_CONFIG_JSON=/root/mineru.json
# 创建数据目录
RUN mkdir -p data knowledge/vector_store .data documents
# ==================== 数据目录 ====================
RUN mkdir -p knowledge/vector_store documents models .data
# ==================== 环境变量 ====================
ENV APP_ENV=prod
ENV PYTHONUNBUFFERED=1
EXPOSE 5001
# 生产模式启动
CMD ["gunicorn", "-c", "gunicorn.conf.py", "wsgi:app"]
CMD ["gunicorn", "-c", "deploy/gunicorn.conf.py", "deploy.wsgi:app"]
```
#### Step 4：构建镜像
```bash
# 构建镜像（会包含模型文件）
docker build -t rag-service:latest .
# 从项目根目录执行构建
docker-compose -f deploy/docker-compose.prod.yml up -d --build
# 或单独构建镜像
docker build -f deploy/Dockerfile.prod -t rag-service:latest .
# 查看镜像大小
docker images rag-service
@@ -213,42 +248,67 @@ mineru-models-download -s huggingface -m all -d /data/mineru-models
"models-dir": {
"pipeline": "/models/pipeline",
"vlm": "/models/vlm"
}
},
"config_version": "1.3.1"
}
```
#### Step 3：Docker Compose 配置
当前项目使用 `deploy/docker-compose.prod.yml`：
```yaml
# docker-compose.yml
# docker-compose.prod.yml - 生产环境部署配置
version: '3.8'
services:
rag-service:
image: rag-service:latest
build:
context: ..
dockerfile: deploy/Dockerfile.prod
container_name: rag-service
env_file:
- .env.production
ports:
- "5001:5001"
volumes:
# 挂载模型目录
- /data/mineru-models:/models:ro
# 挂载配置文件
- /data/mineru.json:/root/mineru.json:ro
# 挂载数据目录
- ./data:/app/data
- ./documents:/app/documents
- ./knowledge:/app/knowledge
environment:
- MINERU_MODEL_SOURCE=local
- MINERU_TOOLS_CONFIG_JSON=/root/mineru.json
- APP_ENV=prod
- ENABLE_SESSION=false
# 数据目录挂载（代码在镜像内，不挂载）
- ../knowledge/vector_store:/app/knowledge/vector_store
- ../documents:/app/documents
- ../models:/app/models
- ../.data:/app/.data
- ../data:/app/data
restart: unless-stopped
shm_size: '256m'
deploy:
resources:
limits:
memory: 4G
reservations:
memory: 2G
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5001/health || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
```
**关键配置说明**：
- `env_file: .env.production`：通过 `.env.production` 文件注入所有环境变量（包括 `MINERU_API_TOKEN`、`DASHSCOPE_API_KEY` 等），不使用 `config.py` 硬编码
- `../models:/app/models`：将宿主机模型目录挂载到容器内，Dockerfile.prod 中已配置 `MINERU_MODEL_SOURCE=local` 和 `MINERU_TOOLS_CONFIG_JSON=/root/mineru.json`
- 端口映射 `5001:5001`，容器名为 `rag-service`
#### Step 4：启动服务
```bash
docker-compose up -d
# 从项目根目录执行
docker-compose -f deploy/docker-compose.prod.yml up -d
```
---
@@ -285,9 +345,51 @@ volumes:
---
## 四、验证部署
## 四、环境变量配置（.env.production）
### 4.1 检查模型路径
生产环境所有配置通过 `deploy/.env.production` 文件注入，不依赖 `config.py` 硬编码。
### 4.1 .env.production 示例
```bash
# .env.production - 生产环境配置
# 部署到服务器时复制到 deploy/.env.production
# 环境标识
APP_ENV=prod
# LLM API
DASHSCOPE_API_KEY=<your-api-key>
DASHSCOPE_BASE_URL=<your-base-url>
DASHSCOPE_MODEL=mimo-v2.5
RAG_CHAT_MODEL=mimo-v2.5
INTENT_MODEL=mimo-v2.5
VLM_MODEL=qwen-vl-plus
# MinerU 在线 API（可选，设置后无需本地模型）
MINERU_API_TOKEN=<your-mineru-api-token>
# 网络搜索（按需开启）
ENABLE_WEB_SEARCH=false
SERPER_API_KEY=<your-serper-key>
# Rerank ONNX 加速（CPU 服务器建议关闭）
RERANK_USE_ONNX=false
```
### 4.2 MinerU 模型路径配置方式
| 方式 | 配置位置 | 说明 |
|------|----------|------|
| `MINERU_API_TOKEN` 环境变量 | `.env.production` | 使用 MinerU 云端 API，无需本地模型 |
| `mineru.json` 配置文件 | `/root/mineru.json`（容器内） | 指定本地模型路径，Dockerfile.prod 已自动生成 |
| `MINERU_MODEL_SOURCE` 环境变量 | Dockerfile.prod 中设置 | `local` 使用本地模型，`huggingface` 自动下载 |
---
## 五、验证部署
### 5.1 检查模型路径
进入容器检查：
@@ -306,7 +408,7 @@ ls -lh /app/models/mineru/vlm/
python -c "from mineru.utils.config_reader import read_config; print(read_config())"
```
### 4.2 测试解析
### 5.2 测试解析
```bash
# 在容器内测试
@@ -314,7 +416,7 @@ cd /app
python parsers/mineru_parser.py documents/test.pdf
```
### 4.3 查看日志
### 5.3 查看日志
```bash
# 查看容器日志
@@ -327,7 +429,7 @@ docker logs -f rag-service
---
## 五、模型文件清单
## 六、模型文件清单
### Pipeline 模型（必需）
@@ -368,7 +470,7 @@ models/mineru/vlm/
---
## 六、常见问题
## 七、常见问题
### Q1: 镜像太大怎么办？
@@ -401,282 +503,47 @@ models/mineru/vlm/
3. 配置文件格式是否正确（JSON 语法）
4. 模型目录路径是否存在
### Q6: 如何使用 MinerU 在线 API 代替本地模型？
**A**: 在 `.env.production` 中设置 `MINERU_API_TOKEN=<your-token>`，无需在本地部署模型。Token 可从 OpenDataLab 平台获取。
---
## 七、推荐方案总结
## 八、推荐方案总结
| 方案 | 优点 | 缺点 | 适用场景 |
|------|------|------|----------|
| **方案 A：打包到镜像** | 部署简单、启动快 | 镜像大、更新麻烦 | 单机部署、离线环境 |
| **方案 B：挂载目录** | 灵活、易更新、多容器共享 | 需要管理宿主机文件 | 多节点、生产环境 |
| **方案 C：自动下载** | 镜像小 | 首次启动慢、依赖网络 | 测试环境 |
| **在线 API** | 无需本地模型、镜像最小 | 依赖网络、有调用限制 | 轻量部署、测试环境 |
**生产环境推荐**：方案 B（挂载模型目录）
**生产环境推荐**：方案 B（挂载模型目录）+ `.env.production` 环境变量注入
---
## 八、部署检查清单
## 九、部署检查清单
部署前检查：
- [ ] 模型文件已下载到 `models/mineru/` 目录
- [ ] 创建了 `mineru.json` 配置文件
- [ ] Dockerfile 中添加了 `COPY models` 和环境变量
- [ ] 创建了 `deploy/.env.production` 配置文件（含 `MINERU_API_TOKEN` 等环境变量）
- [ ] 确认使用 `deploy/Dockerfile.prod` 和 `deploy/docker-compose.prod.yml`
- [ ] 测试了本地解析功能
- [ ] 确认模型文件大小（3-8GB）
部署后检查：
- [ ] 容器启动成功
- [ ] 容器启动成功：`docker ps | grep rag-service`
- [ ] 配置文件存在：`docker exec rag-service cat /root/mineru.json`
- [ ] 模型目录存在：`docker exec rag-service ls /app/models/mineru`
- [ ] 环境变量正确：`docker exec rag-service env | grep MINERU`
- [ ] 健康检查通过：`curl http://localhost:5001/health`
- [ ] 测试解析功能：上传一个 PDF 测试
- [ ] 查看日志无错误
- [ ] 查看日志无错误：`docker logs -f rag-service`
---
**文档版本**: v1.0
**最后更新**: 2026-04-20
**文档版本**: v1.1
**最后更新**: 2026-06-04
**维护者**: RAG 服务开发组
---
## 附加篇：MinerU安装深度解析与Windows框架全景 (Technical Analysis and Deployment Framework for MinerU)
Comprehensive Technical Analysis and Deployment Framework for MinerU Document Parsing Systems on Windows Platforms
The rapid proliferation of large language models (LLMs) has fundamentally transformed the landscape of data engineering, necessitating high-fidelity document parsing tools that can bridge the gap between unstructured PDF, image, and office formats and machine-readable structured data. MinerU, an open-source document parsing toolkit born from the pre-training requirements of the InternLM series, has emerged as a preeminent solution for converting complex documents into Markdown and JSON formats while preserving structural integrity, mathematical formulas, and tabular data.[1, 2] As an orchestrator of multiple deep learning components—including layout detection, optical character recognition (OCR), and vision-language model (VLM) reasoning—MinerU provides a robust pipeline for downstream retrieval-augmented generation (RAG) and agentic workflows.[3, 4] However, the deployment of such a sophisticated system on Windows operating systems presents a unique set of architectural and dependency-related challenges, primarily due to its heavy reliance on libraries traditionally optimized for Linux environments. This report provides an exhaustive investigation into the installation methodologies, operational configurations, and troubleshooting frameworks required to successfully implement MinerU on Windows platforms.
Architectural Evolution and Component Logic
Understanding the implementation of MinerU requires a nuanced appreciation of its underlying architecture and how its components interact within a Windows-based environment. Originally conceived as the Magic-PDF project, the tool has evolved into a comprehensive suite capable of handling not only PDFs but also DOCX, PPTX, and XLSX inputs.[5, 6] The system is designed to remove semantic noise—such as headers, footers, and page numbers—while accurately reconstructing the human reading order in multi-column or complex layouts.[3, 4]
Inference Backends and Hardware Specificity
The versatility of MinerU is largely attributed to its support for diverse inference backends, which allow users to balance accuracy against available hardware resources. On Windows, the choice of backend is the most significant predictor of installation complexity and operational stability.
Backend Type
Core Technologies
Hardware Optimization
Minimum VRAM
Accuracy (OmniDocBench)
Pipeline
Layout detection + OCR
CPU (MKL/AVX) or GPU (Volta+)
4 GB
86.2 [5, 7]
Hybrid
Layout + OCR + VLM Reasoning
NVIDIA GPU (Volta+)
8 GB
90+ [5]
VLM-Transformers
Transformers-based Vision models
NVIDIA GPU (Ampere+)
8 GB
90+ [8]
VLM-SGLang
SGLang Optimized Inference
NVIDIA GPU (Ampere+)
24 GB
90+ [8]
The "Pipeline" backend is the most compatible with Windows systems, especially those lacking high-end GPUs, as it supports pure CPU inference through Intel MKL and AVX instruction sets.[5, 9] The recent release of version 3.0.0 introduced a systematic upgrade to this architecture, replacing several AGPLv3-licensed models with high-precision alternatives that achieve higher accuracy on the OmniDocBench benchmark.[5] This version also pioneered native DOCX parsing, which bypasses the traditional PDF-to-Markdown conversion, improving end-to-end speed by tens of times.[5]
The Role of OCR and Vision Models
A fundamental feature of the MinerU ecosystem is its dual-engine approach to content extraction. For text-based PDFs, it utilizes sophisticated reading order analysis; for scanned or "garbled" documents, it automatically triggers an OCR pipeline supporting 109 languages.[1, 5] In Windows environments, the OCR functionality is primarily powered by PaddleOCR, which necessitates a specific version of the PaddlePaddle framework that matches the system's CUDA environment.[9, 10] This requirement is often a source of friction during installation, as version mismatches between CUDA, cuDNN, and PaddlePaddle can lead to silent failures or runtime errors.[11, 12]
Foundational Requirements and Environment Preparation
A successful Windows installation is predicated on the rigorous preparation of the software toolchain and hardware drivers. Windows lacks the integrated development headers found in most Linux distributions, making the manual configuration of the C++ build environment mandatory.
Hardware Prerequisites and Resource Allocation
MinerU is a resource-intensive application, particularly during the model initialization and inference stages. Windows users must ensure their systems adhere to the following hardware guidelines to avoid performance bottlenecks or out-of-memory (OOM) conditions.
Component
Minimum Specification
Recommended Specification
Rationale
RAM
16 GB
32 GB or more
Required for loading multiple model weights simultaneously.[5, 7]
GPU
NVIDIA Volta (e.g., V100)
Ampere (30-series) or later
Newer architectures support advanced quantization and faster inference.[7, 8]
VRAM
4 GB (Pipeline mode)
24 GB (VLM mode)
VLM backends require substantial memory for high-resolution document imagery.[8]
Storage
20 GB (SSD)
50 GB+ (SSD)
Models and intermediate rendering files require fast I/O and significant space.[7, 13]
Processor
64-bit x86 with AVX
Multi-core Intel/AMD
AVX support is critical for the CPU version of PaddlePaddle.[9, 14]
The Windows C++ Build Toolchain
One of the most common points of failure in the installation process is the absence of Microsoft Visual C++ Build Tools. Many of the underlying Python libraries, most notably detectron2 and pycocotools, involve C++ and CUDA extensions that must be compiled during installation if pre-built wheels are unavailable.[15, 16]
To resolve this, the user must install the Visual Studio Community edition and select the "Desktop development with C++" workload.[15, 17] Within this workload, it is essential to verify the presence of:
MSVC v142 or v143 C++ x64/x86 build tools.
Windows 10 or 11 SDK (matching the host OS version).[15, 16]
C++/CLI support for the respective build tools.[15]
Following the installation of these tools, a system restart is strongly recommended to ensure that the compiler paths are correctly registered in the Windows environment.[16]
Python Version Strategy
MinerU currently supports Python versions 3.10 through 3.12, with some documentation suggesting the beginning of support for 3.13.[8, 18] However, for Windows users, Python 3.10 remains the recommended version. This is because many critical dependencies, such as the precompiled wheels for detectron2 provided by the community, are specifically built for Python 3.10.[13, 19, 20] Using newer versions of Python often forces the system to attempt a source compilation of detectron2, which is notoriously difficult to complete successfully on Windows without extensive manual patching.[21]
Detailed Installation Procedures
There are three primary avenues for installing MinerU on Windows: the pip/uv method, installation from source, and Docker-based deployment.
Path A: Streamlined Installation via uv
The uv package manager has become the preferred tool for MinerU installation due to its superior dependency resolution and execution speed compared to standard pip.[6, 8]
Environment Setup: It is a best practice to use a clean Conda environment to avoid library version conflicts.
Manager Installation: Ensure pip and uv are up to date.
Core Package Installation: The mineru[all] extra ensures that all components, including those for VLM and pipeline acceleration, are fetched.
Path B: Addressing the Detectron2 Hurdle
The detectron2 library is a frequent obstacle as it lacks official Windows support from Meta.[21, 22] If the standard installation fails, users should manually install a precompiled wheel tailored for Windows and Python 3.10.
pip install detectron2 --extra-index-url https://myhloli.github.io/wheels/
``` [13, 23]
This wheel bypasses the need for local compilation. If this index is inaccessible or the user is on a different Python version, they must clone the repository and build it manually, which requires the `CUDA_HOME` environment variable to be set correctly to the location of the installed CUDA Toolkit.[21] Failure to match the CUDA version used to compile `detectron2` with the CUDA version used for PyTorch will result in a runtime crash.[21]
### Path C: GPU and OCR Acceleration Setup
To enable GPU acceleration on Windows, the default PyTorch installation must often be overwritten with a version specifically linked to the system's CUDA version.[10, 13]
| CUDA Version | PyTorch Installation Command |
| :--- | :--- |
| **11.8** | `pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118` [10, 13] |
| **12.1+** | Refer to the PyTorch official website for the matching `torch` and `torchvision` versions.[24] |
Furthermore, to activate OCR acceleration, the `paddlepaddle-gpu` library must be installed. For users with CUDA 11.8, version 2.6.1 or 3.0.0b1 is recommended.[10] For users on newer CUDA versions (e.g., 12.6 or 12.9), specific GPU-enabled wheels from the PaddlePaddle repository must be used to ensure compatibility with the GPU driver.[9, 25]
```bash
# Example for CUDA 12.6
python -m pip install paddlepaddle-gpu==3.2.2 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
``` [25]
## Configuration and Model Management
Post-installation, MinerU requires the presence of several model weights to function. The management of these weights and the system's global configuration is handled through a JSON-based framework.
### The Configuration Shift: magic-pdf.json to mineru.json
Earlier versions of the project (v1.x) utilized a configuration file named `magic-pdf.json`. With the transition to version 2.0 and above, the system now prioritizes `mineru.json`.[24] This file is automatically generated in the user's home directory (e.g., `C:\Users\YourUsername`) the first time the model download utility is run.[8, 10, 18]
Key configuration parameters within `mineru.json` include:
* **models-dir**: This specifies the absolute path to the directory where model weights are stored. On Windows, it is critical to use forward slashes (e.g., `D:/models`) or escaped backslashes (`D:\\models`) to prevent JSON parsing errors.[26, 27]
* **device-mode**: Set to `cuda` for GPU acceleration or `cpu` for standard inference.[10, 13]
* **latex-delimiter-config**: Allows customization of the symbols used to mark LaTeX formulas in the output Markdown.[8, 24]
* **llm-aided-config**: Configures an external LLM (via OpenAI-compatible API) to assist in determining the title hierarchy and logical structure of the document.[24]
### Automated and Manual Model Acquisition
The system provides a built-in utility, `mineru-models-download`, to facilitate model acquisition. By default, this tool attempts to fetch models from HuggingFace. For users in mainland China or other regions with network restrictions, the model source can be switched to ModelScope.[24, 28]
This change is implemented via an environment variable:
```cmd
set MINERU_MODEL_SOURCE=modelscope
mineru-models-download
``` [24, 28]
If models are moved to a different disk after download, the `models-dir` path in `mineru.json` must be updated accordingly. It is important to note that the `mineru-models-download` tool does not currently support custom download paths; it will always download to a default location and then update the JSON configuration with that path.[28]
## Operational Framework and Usage Scenarios
MinerU provides three primary modes of operation: the Command Line Interface (CLI), the Python API (SDK), and the Web-based User Interface (WebUI).
### Command Line Excellence
The CLI is the primary tool for batch processing and integration into automated shell scripts. The command structure has transitioned from `magic-pdf` in older versions to the unified `mineru` command in version 2.x and later.[5, 8]
**Basic Syntax**:
`mineru -p <input_path> -o <output_path> -m auto` [6, 8]
**Advanced Parameters**:
* `-m` / `--method`: Determines the parsing logic. `txt` is optimized for text-based PDFs, `ocr` for images and scans, and `auto` allows the system to intelligently switch based on document characteristics.[8, 26]
* `-b` / `--backend`: Specifies the inference engine, such as `pipeline` or `vlm-transformers`.[8]
* `-s` / `--start` and `-e` / `--end`: Enables partial parsing of long documents by specifying a range of page numbers (0-indexed).[8]
* `-f` / `--formula` and `-t` / `--table`: Boolean flags to enable or disable the recognition of mathematical formulas and tables.[8]
### Integration via Python SDK
For sophisticated developers, the Python SDK offers the `Dataset` class, allowing for programmatic control over the parsing stages.[8] This is particularly useful for building custom RAG pipelines where documents need to be processed and then immediately ingested into a vector database. The new `Stage` architecture allows users to define custom processing steps and combine them creatively.[8]
### WebUI and API Services
The project includes a Gradio-based demo for interactive testing. This can be launched by running `python demo.py` from the demo directory or using the `mineru-gradio` command.[3, 24] For production deployments, `mineru-api` provides a FastAPI service that supports both synchronous (`/file_parse`) and asynchronous (`/tasks`) endpoints.[5] Windows users can also integrate MinerU into tools like Dify via its official plugin, which supports both the cloud-hosted API and local deployments.[4]
## Troubleshooting and Issue Mitigation on Windows
The Windows environment introduces specific failure modes that are often absent on Linux. These typically involve DLL dependencies, path handling, and memory management.
### Dependency and DLL Failures
* **libiomp5md.dll Missing**: This is the Intel OpenMP Runtime library, essential for parallelizing CPU-based computations. This error occurs if the Intel CPU library dependencies are out of sync. Solutions include installing the Intel Fortran runtime or ensuring that the DLL is present in the `System32` directory or the Python environment's library path.[29, 30]
* **zlib.dll Errors**: A "missing" `zlib.dll` can prevent the application from starting. This is often caused by accidental deletion or incomplete package installation. Reinstalling the program that uses the library or running `sfc /scannow` to repair the Windows system files are common remedies.[31]
* **ModuleNotFoundError: No module named 'package'**: This deceptive error is often triggered by an incompatible version of the `ultralytics` package. The verified fix is to downgrade `ultralytics` to version 8.3.43 or 8.3.40.[32]
### Path and JSON Parsing Issues
The Windows backslash (`\`) is a frequent source of "Unexpected Token" errors in the JSON configuration files. Because the backslash is an escape character in JSON, it must be escaped with another backslash (e.g., `D:\\MinerU\\models`) or replaced with a forward slash (`D:/MinerU/models`).[26, 27] Furthermore, Windows users with non-ASCII characters or spaces in their usernames (e.g., `C:\Users\User Name`) may encounter issues with the default configuration path. Setting the `MINERU_TOOLS_CONFIG_JSON` environment variable to a custom path on a different drive can bypass these restrictions.[33]
### Optical Character Recognition (OCR) Degradation
In documents with a high density of inline formulas, users may observe that some text lines or symbols are missing from the output. This is often caused by overly aggressive binarization thresholds in the layout detector or OCR engine.[34] Adjusting the `det_db_thresh` and `det_db_box_thresh` parameters in the OCR initialization configuration can mitigate this. Lowering `det_db_box_thresh` from the default 0.6 to 0.3 allows the engine to retain more bounding boxes that might otherwise be filtered out as noise.[34]
## Performance Optimization and Stability
Moving from experimental parsing to production-scale document processing on Windows requires the application of several optimization strategies.
### Memory Optimization for Long Documents
Historically, parsing ultra-long documents (thousands of pages) led to massive memory spikes and eventual crashes. MinerU has addressed this by introducing a "sliding-window" mechanism that processes the document in smaller, manageable chunks.[5]
The parameter `MINERU_PROCESSING_WINDOW_SIZE` (defaulting to 64) can be adjusted via environment variables to fine-tune the memory usage.[35] On systems with limited RAM, reducing this window size ensures stability, while on high-end workstations, increasing it can improve throughput. Additionally, the system now supports streaming writes to disk, allowing results to be persisted as they are completed rather than waiting for the entire document to finish.[5]
### VRAM and GPU Tuning
For users leveraging GPU acceleration, the `MINERU_HYBRID_BATCH_RATIO` environment variable allows for the dynamic adjustment of VRAM usage. This is critical for fitting the VLM and OCR models into consumer-grade GPUs with limited memory.[35]
| Available VRAM | Recommended Batch Ratio |
| :--- | :--- |
| **<= 2 GB** | 1 |
| **<= 3 GB** | 2 |
| **<= 4 GB** | 4 |
| **<= 6 GB** | 8 [35] |
For large-scale deployments, the introduction of `mineru-router` in version 3.0.0 allows for unified entry deployment across multiple services and GPUs. It provides automatic task load balancing and is fully compatible with the standard `mineru-api` interfaces.[5]
## Conclusion and Strategic Outlook
The implementation of MinerU on Windows platforms provides a powerful mechanism for high-precision document extraction, despite the inherent complexity of the installation process. By navigating the nuances of C++ build toolchains, specialized `detectron2` wheels, and CUDA-aligned PaddlePaddle versions, professional users can establish a stable and efficient parsing environment.
The transition toward version 3.0.0 represents a significant milestone in the project's maturity, emphasizing not just raw accuracy but also engineering stability and architectural flexibility. Features like native DOCX parsing and the sliding-window mechanism for long documents demonstrate a commitment to production-readiness. As MinerU continues to integrate with the broader AI ecosystem—serving as a foundational layer for RAG frameworks and agentic platforms—its role in the document intelligence domain is poised to expand significantly. For Windows-based enterprises and researchers, following the structured deployment framework detailed in this report ensures that they can capitalize on these advancements while maintaining the data sovereignty and performance benefits of a local installation.
---
1. MinerU, [https://opendatalab.github.io/MinerU/](https://opendatalab.github.io/MinerU/)
2. papayalove/Magic-PDF - GitHub, [https://github.com/papayalove/Magic-PDF](https://github.com/papayalove/Magic-PDF)
3. Extract Any PDF with MinerU 2.5 (Easy Tutorial) - Sonusahani.com, [https://sonusahani.com/blogs/mineru](https://sonusahani.com/blogs/mineru)
4. MinerU - Dify Marketplace, [https://marketplace.dify.ai/plugin/langgenius/mineru](https://marketplace.dify.ai/plugin/langgenius/mineru)
5. GitHub - opendatalab/MinerU: Transforms complex documents like PDFs into LLM-ready markdown/JSON for your Agentic workflows., [https://github.com/opendatalab/mineru](https://github.com/opendatalab/mineru)
6. Quick Start - MinerU, [https://opendatalab.github.io/MinerU/quick_start/](https://opendatalab.github.io/MinerU/quick_start/)
7. FAQ - MinerU - OpenDataLab | Data-Centric AI Research, [https://opendatalab.github.io/MinerU/faq/](https://opendatalab.github.io/MinerU/faq/)
8. mineru - PyPI, [https://pypi.org/project/mineru/2.0.0/](https://pypi.org/project/mineru/2.0.0/)
9. Install on Windows via PIP-Document-PaddlePaddle Deep Learning Platform, [https://www.paddlepaddle.org.cn/documentation/docs/en/install/pip/windows-pip_en.html](https://www.paddlepaddle.org.cn/documentation/docs/en/install/pip/windows-pip_en.html)
10. Installation - Boost With Cuda - 《MinerU v1.1 Documentation》 - 书栈网 · BookStack, [https://www.bookstack.cn/read/mineru-1.1-en/e60b2c33b3945c15.md](https://www.bookstack.cn/read/mineru-1.1-en/e60b2c33b3945c15.md)
11. [R] Struggle with PaddlePaddle OCR Vision Language installation - Reddit, [https://www.reddit.com/r/MachineLearning/comments/1p5d1gn/r_struggle_with_paddlepaddle_ocr_vision_language/](https://www.reddit.com/r/MachineLearning/comments/1p5d1gn/r_struggle_with_paddlepaddle_ocr_vision_language/)
12. Can't install paddle · PaddlePaddle PaddleOCR · Discussion #14556 - GitHub, [https://github.com/PaddlePaddle/PaddleOCR/discussions/14556](https://github.com/PaddlePaddle/PaddleOCR/discussions/14556)
13. AiBotLab/MinerU - Gitee, [https://gitee.com/yuzhu_yang/MinerU/blob/master/README.md](https://gitee.com/yuzhu_yang/MinerU/blob/master/README.md)
14. Installation Guide-Document-PaddlePaddle Deep Learning Platform, [https://www.paddlepaddle.org.cn/documentation/docs/en/2.2/install/index_en.html](https://www.paddlepaddle.org.cn/documentation/docs/en/2.2/install/index_en.html)
15. Step-by-Step Guide: How to Install Detectron2 from Scratch on Windows - Medium, [https://medium.com/@lukmanshaikh/step-by-step-guide-how-to-install-detectron2-from-scratch-on-windows-748b69cf18f1](https://medium.com/@lukmanshaikh/step-by-step-guide-how-to-install-detectron2-from-scratch-on-windows-748b69cf18f1)
16. How to install Detectron2 on Your Windows local system? - Shreyas Kulkarni, [https://helloshreyas.com/how-to-install-detectron2-on-windows-machine](https://helloshreyas.com/how-to-install-detectron2-on-windows-machine)
17. Detectron 2 Installation on Windows | by Vigneshwara - Medium, [https://medium.com/@rockingstarvic/detectron-2-installation-on-windows-3c8e717b44fd](https://medium.com/@rockingstarvic/detectron-2-installation-on-windows-3c8e717b44fd)
18. magic-pdf - PyPI, [https://pypi.org/project/magic-pdf/](https://pypi.org/project/magic-pdf/)
19. How to install Detectron2 - Stack Overflow, [https://stackoverflow.com/questions/75357936/how-to-install-detectron2](https://stackoverflow.com/questions/75357936/how-to-install-detectron2)
20. fail to install detectron2 · Issue #199 · opendatalab/MinerU - GitHub, [https://github.com/opendatalab/MinerU/issues/199](https://github.com/opendatalab/MinerU/issues/199)
21. Installation — detectron2 0.6 documentation, [https://detectron2.readthedocs.io/tutorials/install.html](https://detectron2.readthedocs.io/tutorials/install.html)
22. Detectron2 & Python Wheels Cache - Medium, [https://medium.com/data-science/detectron2-python-wheels-cache-bfb94a0267ef](https://medium.com/data-science/detectron2-python-wheels-cache-bfb94a0267ef)
23. It is impossible to start magic-pdf --version using the command line, and a TypeError is reported. · Issue #232 · opendatalab/MinerU - GitHub, [https://github.com/opendatalab/MinerU/issues/232](https://github.com/opendatalab/MinerU/issues/232)
24. Quick Usage - MinerU, [https://opendatalab.github.io/MinerU/usage/quick_usage/](https://opendatalab.github.io/MinerU/usage/quick_usage/)
25. Install on Windows via PIP, [https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/install/pip/windows-pip_en.html](https://www.paddlepaddle.org.cn/en/install/quick?docurl=/documentation/docs/en/install/pip/windows-pip_en.html)
26. lm_tool/MinerU - Gitee, [https://gitee.com/lm_tool/MinerU/blob/master/README.md](https://gitee.com/lm_tool/MinerU/blob/master/README.md)
27. Error in parsing backslash escape sequence in JSON - Stack Overflow, [https://stackoverflow.com/questions/52343137/error-in-parsing-backslash-escape-sequence-in-json](https://stackoverflow.com/questions/52343137/error-in-parsing-backslash-escape-sequence-in-json)
28. Model Source - MinerU, [https://opendatalab.github.io/MinerU/usage/model_source/](https://opendatalab.github.io/MinerU/usage/model_source/)
29. Libiomp5md.dll Not Found? Here's the Fast Fix! - YouTube, [https://www.youtube.com/watch?v=IltGCJLbW5I](https://www.youtube.com/watch?v=IltGCJLbW5I)
30. Missing DLL problem - CoPS, [https://www.copsmodels.com/gpmissingdll.htm](https://www.copsmodels.com/gpmissingdll.htm)
31. Download.... Missing Zlib.dll from ur computer ?? - Microsoft Q&A, [https://learn.microsoft.com/en-us/answers/questions/2456917/download-missing-zlib-dll-from-ur-computer](https://learn.microsoft.com/en-us/answers/questions/2456917/download-missing-zlib-dll-from-ur-computer)
32. pip installation successful, but 'magic-pdf --version' cannot display the version · Issue #1219 · opendatalab/MinerU - GitHub, [https://github.com/opendatalab/MinerU/issues/1219](https://github.com/opendatalab/MinerU/issues/1219)
33. MinerU customized weight folder · Issue #2955 - GitHub, [https://github.com/opendatalab/MinerU/issues/2955](https://github.com/opendatalab/MinerU/issues/2955)
34. Layout Bug · Issue #450 · opendatalab/MinerU - GitHub, [https://github.com/opendatalab/MinerU/issues/450](https://github.com/opendatalab/MinerU/issues/450)
35. CLI Tools - MinerU, [https://opendatalab.github.io/MinerU/usage/cli_tools/](https://opendatalab.github.io/MinerU/usage/cli_tools/)