TextChunker.SplitPlainTextParagraphs Method

Definition

Split plain text into paragraphs.

public static System.Collections.Generic.List<string> SplitPlainTextParagraphs (System.Collections.Generic.IEnumerable<string> lines, int maxTokensPerParagraph, int overlapTokens = 0, string? chunkHeader = default, Microsoft.SemanticKernel.Text.TextChunker.TokenCounter? tokenCounter = default);
static member SplitPlainTextParagraphs : seq<string> * int * int * string * Microsoft.SemanticKernel.Text.TextChunker.TokenCounter -> System.Collections.Generic.List<string>
Public Shared Function SplitPlainTextParagraphs (lines As IEnumerable(Of String), maxTokensPerParagraph As Integer, Optional overlapTokens As Integer = 0, Optional chunkHeader As String = Nothing, Optional tokenCounter As TextChunker.TokenCounter = Nothing) As List(Of String)

Parameters

lines
IEnumerable<String>

Lines of text.

maxTokensPerParagraph
Int32

Maximum number of tokens per paragraph.

overlapTokens
Int32

Number of tokens to overlap between paragraphs.

chunkHeader
String

Text to be prepended to each individual chunk.

tokenCounter
TextChunker.TokenCounter

Function to count tokens in a string. If not supplied, the default counter will be used.

Returns

List of paragraphs.

Applies to