周末项目：AI 时代真正流式解析+渲染双重优化的 Incremark

故事的开始

故事的开始是我想要为我的 tiptap 编辑器增加 AI 流式内容输出的功能，但 AI 输出的是 markdown，我需要将其解析为 prosemirror JSONContent，但我又想尽可能节省性能，每次已经稳定的内容避免重复解析，正在生成的块不断进行更新，因此有了 incremark 这个小工具。

问题分析

传统的 Markdown 解析器（marked、remark 等）设计用于处理完整文档。但在流式输出场景中，每次收到新内容都需要重新解析全部文本：

▼
text
复制代码
收到 chunk 1: 解析 "# Hello"
收到 chunk 2: 解析 "# Hello\n\nWorld"
收到 chunk 3: 解析 "# Hello\n\nWorld ..."
...

这导致了 O(n²) 的时间复杂度。文档越长，问题越严重。

解决方案

Incremark 的核心思想是增量解析：

识别已完成的 Markdown 块（标题、段落、代码块等）
将已完成的块"锁定"，不再重新解析
只解析新增内容和未完成的块

这样将复杂度从 O(n²) 降到 O(n)。

性能测试

为了验证效果，我写了一个 Benchmark，模拟不同长度文档的流式输入：

文档大小	加速比	适用场景
~1KB	2-3x	短回复
~5KB	9-11x	中等回复
~10KB	17-23x	长回复
~20KB	37-46x	超长回复

说明：

加速比与 chunk 大小有关，chunk 越小（如 10 字符），加速比越高
真实 AI 输出通常是小 chunk（几个到几十个字符），接近测试中的高加速比场景
短文档场景下加速比不明显，这是正常的（O(n²) 在 n 小时与 O(n) 差距不大）

测试环境：Node.js 20+，可以通过 pnpm benchmark 自行验证。

技术挑战

增量解析的核心难点是块边界检测：如何判断一个 Markdown 块已经完成？

比如代码块：

▼
markdown
复制代码
```javascript
function hello() {

这时候还不能认为代码块完成，因为没有闭合的 ```。

Incremark 针对不同块类型实现了边界检测逻辑：

代码块：检测配对的 ``` 或 ~~~
列表：检测缩进变化和列表模式中断
引用：检测 > 前缀的连续性
表格：检测表格行模式
段落：遇到空行或块级元素时完成

实际 benchmark 测试

小型 markdown 解析性能可提高 2-10 倍，中型 markdown 解析性能提高 10-20 倍，长 markdown 解析性能提高 20-46 倍，当前更长的测试是没有必要的。

原始测试输出结果：

▼
plain
复制代码
============================================================
Incremark Benchmark
============================================================
Markdown length: 771 chars
Chunk size: 10 chars
Total chunks: 78
Iterations: 100
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 2608.41 ms
   Parse count: 7800
   Avg time per parse: 0.3344 ms
   Total chars parsed: 3,080,100

⚡ Incremark (incremental)
   Total time: 638.36 ms
   Parse count: 7800
   Avg time per parse: 0.0818 ms
   Total chars parsed: 77,100

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 75.5%
   Chars parsing saved: 97.5%
   Speedup: 4.09x faster

============================================================

🔬 Running Incremark Benchmark Suite


============================================================
📄 Document Size: Short (~1KB) (1000 chars)
============================================================

📦 Chunk size: 10 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 1000 chars
Chunk size: 10 chars
Total chunks: 100
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 435.85 ms
   Parse count: 2000
   Avg time per parse: 0.2179 ms
   Total chars parsed: 1,010,000

⚡ Incremark (incremental)
   Total time: 171.50 ms
   Parse count: 2000
   Avg time per parse: 0.0858 ms
   Total chars parsed: 20,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 60.7%
   Chars parsing saved: 98.0%
   Speedup: 2.54x faster

============================================================

📦 Chunk size: 50 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 1000 chars
Chunk size: 50 chars
Total chunks: 20
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 92.33 ms
   Parse count: 400
   Avg time per parse: 0.2308 ms
   Total chars parsed: 210,000

⚡ Incremark (incremental)
   Total time: 43.77 ms
   Parse count: 400
   Avg time per parse: 0.1094 ms
   Total chars parsed: 20,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 52.6%
   Chars parsing saved: 90.5%
   Speedup: 2.11x faster

============================================================

============================================================
📄 Document Size: Medium (~5KB) (5000 chars)
============================================================

📦 Chunk size: 10 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 5000 chars
Chunk size: 10 chars
Total chunks: 500
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 10335.94 ms
   Parse count: 10000
   Avg time per parse: 1.0336 ms
   Total chars parsed: 25,050,000

⚡ Incremark (incremental)
   Total time: 916.48 ms
   Parse count: 10000
   Avg time per parse: 0.0916 ms
   Total chars parsed: 100,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 91.1%
   Chars parsing saved: 99.6%
   Speedup: 11.28x faster

============================================================

📦 Chunk size: 50 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 5000 chars
Chunk size: 50 chars
Total chunks: 100
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 2120.47 ms
   Parse count: 2000
   Avg time per parse: 1.0602 ms
   Total chars parsed: 5,050,000

⚡ Incremark (incremental)
   Total time: 223.64 ms
   Parse count: 2000
   Avg time per parse: 0.1118 ms
   Total chars parsed: 100,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 89.5%
   Chars parsing saved: 98.0%
   Speedup: 9.48x faster

============================================================

============================================================
📄 Document Size: Long (~10KB) (10000 chars)
============================================================

📦 Chunk size: 10 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 10000 chars
Chunk size: 10 chars
Total chunks: 1000
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 40596.85 ms
   Parse count: 20000
   Avg time per parse: 2.0298 ms
   Total chars parsed: 100,100,000

⚡ Incremark (incremental)
   Total time: 1781.89 ms
   Parse count: 20000
   Avg time per parse: 0.0891 ms
   Total chars parsed: 200,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 95.6%
   Chars parsing saved: 99.8%
   Speedup: 22.78x faster

============================================================

📦 Chunk size: 50 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 10000 chars
Chunk size: 50 chars
Total chunks: 200
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 8095.40 ms
   Parse count: 4000
   Avg time per parse: 2.0239 ms
   Total chars parsed: 20,100,000

⚡ Incremark (incremental)
   Total time: 473.23 ms
   Parse count: 4000
   Avg time per parse: 0.1183 ms
   Total chars parsed: 200,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 94.2%
   Chars parsing saved: 99.0%
   Speedup: 17.11x faster

============================================================

============================================================
📄 Document Size: Very Long (~20KB) (20000 chars)
============================================================

📦 Chunk size: 10 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 20000 chars
Chunk size: 10 chars
Total chunks: 2000
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 183844.78 ms
   Parse count: 40000
   Avg time per parse: 4.5961 ms
   Total chars parsed: 400,200,000

⚡ Incremark (incremental)
   Total time: 3997.77 ms
   Parse count: 40000
   Avg time per parse: 0.0999 ms
   Total chars parsed: 400,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 97.8%
   Chars parsing saved: 99.9%
   Speedup: 45.99x faster

============================================================

📦 Chunk size: 50 chars

============================================================
Incremark Benchmark
============================================================
Markdown length: 20000 chars
Chunk size: 50 chars
Total chunks: 400
Iterations: 20
============================================================

Warming up...

Running benchmark...

Results:
------------------------------------------------------------

📊 Traditional (re-parse all)
   Total time: 37400.52 ms
   Parse count: 8000
   Avg time per parse: 4.6751 ms
   Total chars parsed: 80,200,000

⚡ Incremark (incremental)
   Total time: 1001.10 ms
   Parse count: 8000
   Avg time per parse: 0.1251 ms
   Total chars parsed: 400,000

------------------------------------------------------------

🎯 Performance Improvement:
   Time saved: 97.3%
   Chars parsing saved: 99.5%
   Speedup: 37.36x faster

============================================================


================================================================================
📈 Complete Benchmark Summary
================================================================================

| Document Size    | Chunk | Time Saved | Chars Saved | Speedup |
|------------------|-------|------------|-------------|---------|
| Short (~1KB)     | 10    |      60.7% |       98.0% |   2.54x |
| Short (~1KB)     | 50    |      52.6% |       90.5% |   2.11x |
| Medium (~5KB)    | 10    |      91.1% |       99.6% |  11.28x |
| Medium (~5KB)    | 50    |      89.5% |       98.0% |   9.48x |
| Long (~10KB)     | 10    |      95.6% |       99.8% |  22.78x |
| Long (~10KB)     | 50    |      94.2% |       99.0% |  17.11x |
| Very Long (~20KB) | 10    |      97.8% |       99.9% |  45.99x |
| Very Long (~20KB) | 50    |      97.3% |       99.5% |  37.36x |

--------------------------------------------------------------------------------

📊 Average by Document Size:

   Short (~1KB): 2.33x faster, 56.7% time saved
   Medium (~5KB): 10.38x faster, 90.3% time saved
   Long (~10KB): 19.94x faster, 94.9% time saved
   Very Long (~20KB): 41.67x faster, 97.5% time saved

--------------------------------------------------------------------------------

🎯 Overall Average:
   Time Saved: 84.8%
   Chars Saved: 98.0%
   Speedup: 18.58x

================================================================================

框架支持

由于从解析层面上实现了流式解析，并使用 mdast 进行解析结果的记录，因此当前工具很容易迁移到各种框架中，目前实现了 vue 与 react 版本，svelte 与 solid 等小众框架实现也很简单（但暂未做）。 vue demo 地址：https://incremark-vue.vercel.app/ react demo 地址：https://incremark-react.vercel.app/

devtools

除了功能实现，还增加了 devtools 可以查看当前解析内容的一些状态

markdown 内容统计

markdown 块统计

mdast 结果

chunk append 记录

deepwiki 查看代码偶性能提升原因回答

使用方式

提供 Vue 和 React 官方集成：

▼
bash
复制代码
# Vue
pnpm add @incremark/core @incremark/vue

# React
pnpm add @incremark/core @incremark/react

▼
vue
复制代码
<script setup>
import { useIncremark, Incremark } from '@incremark/vue'

const { blocks, append, finalize } = useIncremark()

async function handleStream(stream) {
  for await (const chunk of stream) {
    append(chunk)
  }
  finalize()
}
</script>

<template>
  <Incremark :blocks="blocks" />
</template>

局限性

额外的边界检测开销：增量解析需要做块边界检测，这本身有一定开销。对于极短的文档（几百字符），可能收益不明显。
复杂嵌套结构：某些复杂的嵌套场景（如列表中的多层引用）边界检测可能不够精确，会导致部分块被重新解析。
不是万能的：如果你的场景是一次性渲染完整文档，传统解析器可能更简单直接。

适用场景

✅ AI 聊天应用的流式输出
✅ 实时 Markdown 预览
✅ 流式文档生成
⚠️ 短文档场景收益有限
❌ 一次性渲染完整文档（直接用 marked/remark 即可）