使用 Kernel Memory 的 TextChunker 來幫我們切 Chunk

註: Kernel Memory 0.97.250211.1 已改用 PlainTextChunker

前言

使用 RAG(Retrieval-Augmented Generation)需要將內容切成 Chunk,
而 Kernel Memory 的 TextChunker 正可以幫我們來做這件事。

實作

1.加入Microsoft.KernelMemory Nuget 套件

2.讀取文件內容後，交給TextChunker 來處理

using Microsoft.KernelMemory.AI;
using Microsoft.KernelMemory.DataFormats.Text;

#pragma warning disable KMEXP00
string contents = System.IO.File.ReadAllText(@"你的文字檔");
TextChunker.TokenCounter tokenCounter = new CL100KTokenizer().CountTokens;

//計算字串的Token數
int tokenCount = tokenCounter(contents);
Console.WriteLine($"The text contains {tokenCount} tokens.");

//每行最多100個Token，超過就換到下一行
int maxTokensPerLine = 100;
var sentences = TextChunker.SplitPlainTextLines(contents, maxTokensPerLine: maxTokensPerLine, tokenCounter: tokenCounter);
Console.WriteLine("======= SplitPlainTextLines =======");
int i = 0;
foreach (var sentence in sentences) {
    Console.WriteLine($"{++i}=>{sentence}");
}


//每個段落最多400個Token，超過就換到下一個段落
int maxTokensPerParagraph = 400;
int overlapTokens = 10;
var partitions = TextChunker.SplitPlainTextParagraphs(sentences,
    maxTokensPerParagraph: maxTokensPerParagraph, overlapTokens: overlapTokens, tokenCounter: tokenCounter);
Console.WriteLine("======= SplitPlainTextParagraphs =======");
i = 0;
foreach (var partition in partitions)
{
    Console.WriteLine($"{++i}=>{partition}");
}

//最後將這些 partition 呼叫 Embedding Model ，轉成 Vector 一併儲起來

• 註: maxTokensPerLine, maxTokensPerParagraph, overlapTokens 請依文件內容來進行調整。

參考資源

Kernel Memory