1. AI Vector Search User's Guide
  2. Use SQL Functions for Vector Operations
  3. Chunking and Vector Generation Functions
  4. VECTOR_CHUNKS

VECTOR_CHUNKS

Use VECTOR_CHUNKS to split plain text into smaller chunks in preparation for the generation of vector embeddings, to be used with vector indexes or hybrid vector indexes.

Syntax


Description of vector_chunks.eps follows
Description of the illustration vector_chunks.eps

chunks_table_arguments::=


Description of chunks_table_arguments.eps follows
Description of the illustration chunks_table_arguments.eps

chunking_spec::=


Description of chunking_spec.eps follows
Description of the illustration chunking_spec.eps

split_characters_list::=


Description of split_characters_list.eps follows
Description of the illustration split_characters_list.eps

custom_split_characters_list


Description of custom_split_characters_list.eps follows
Description of the illustration custom_split_characters_list.eps

normalization_spec


Description of normalization_spec.eps follows
Description of the illustration normalization_spec.eps

custom_normalization_spec


Description of custom_normalization_spec.eps follows
Description of the illustration custom_normalization_spec.eps

normalization_mode


Description of normalization_mode.eps follows
Description of the illustration normalization_mode.eps

chunking_mode::=


Description of chunking_mode.eps follows
Description of the illustration chunking_mode.eps

Purpose

VECTOR_CHUNKS takes a row of textual input and returns the output as one or more chunks (pieces of words, sentences, or paragraphs) with metadata.

The chunking output includes:

  • chunk_offset: Position of each chunk (NUMBER) in the source document, relative to the start of document which has a position of 1

  • chunk_length: Character length (NUMBER) of each chunk

  • chunk_text: Text pieces from each chunk

VECTOR_CHUNKS takes as input one of the following data types: VARCHAR2, CHAR, CLOB, NVARCHAR2, NCLOB, NCHAR. It is aware of the database character set and the national language character set.

It returns as output a text chunk as VARCHAR2 or NVARCAHR2.

Table 7-1 Input and Output Data Type Details

Input Data Type Database NLS Parameter Input Encoding Output Data Type Output Offset

VARCHAR2

NLS_CHARACTERSET

Any

VARCHAR2

byte

CHAR

NLS_CHARACTERSET

Any

VARCHAR2

 byte

CLOB

NLS_CHARACTERSET

Note: CLOB is a special case, as it stores the data in UTF-16 (when this character set is multi-byte) but provides access in this NLS character set. The chunking table function uses the NLS_CHARACTER_SET so it can output in a VARCHAR2. However, the returned offset and length are in terms of characters.

Any

VARCHAR2

character

NVARCHAR2

NLS_NCHAR_CHARACTERSET

AL16UTF16

(also AL32UTF8)

NVARCHAR2

byte

NCLOB

NLS_NCHAR_CHARACTERSET

AL16UTF16

(also AL32UTF8)

NVARCHAR2

character

NCHAR

NLS_NCHAR_CHARACTERSET

AL16UTF16

(also AL32UTF8)

NVARCHAR2

byte

Note:

  • The SQL NCHAR, NVARCHAR2, and NCLOB data types support Unicode data only. You can use either UTF8 or AL16UTF16 character set. The default is AL16UTF16.

  • The VARCHAR2 input data type is limited to 4000 bytes unless the MAX_STRING_SIZE parameter is set to EXTENDED, which increases the limit to 32767.

Parameters

All chunking parameters are optional, and the default chunking specifications are automatically applied to your chunk data.

When specifying chunking parameters for this API, ensure that you provide these parameters only in the listed order.

Table 7-2 Chunking Parameters Table

Parameter Description and Acceptable Values

BY

Specifies the mode for splitting your data, that is, to split by counting the number of characters, words, or vocabulary tokens.

Valid values:

  • BY CHARACTERS (or BY CHARS):

    Splits by counting the number of characters.

  • BY WORDS:

    Splits by counting the number of words.

    Words are defined as sequences of alphabetic characters, sequences of digits, individual punctuation marks, or symbols. For segmented languages without whitespace word boundaries (such as Chinese, Japanese, or Thai), each native character is considered a word (that is, unigram).

  • BY VOCABULARY:

    Splits by counting the number of vocabulary tokens.

    Vocabulary tokens are words or word pieces, recognized by the vocabulary of the tokenizer that your embedding model uses. You can load your vocabulary file using the VECTOR_CHUNKS helper API DBMS_VECTOR_CHAIN.CREATE_VOCABULARY.

    Note: For accurate results, ensure that the chosen model matches the vocabulary file used for chunking. If you are not using a vocabulary file, then ensure that the input length is defined within the token limits of your model.

Default value: BY WORDS

MAX

Specifies a limit on the maximum size of each chunk. This setting splits the input text at a fixed point where the maximum limit occurs in the larger text. The units of MAX correspond to the BY mode, that is, to split data when it reaches the maximum size limit of a certain number of characters, words, numbers, punctuation marks, or vocabulary tokens.

Valid values:

  • BY CHARACTERS: 50 to 4000 characters

  • BY WORDS: 10 to 1000 words

  • BY VOCABULARY: 10 to 1000 tokens

Default value: 100

SPLIT [BY]

Specifies where to split the input text when it reaches the maximum size limit. This helps to keep related data together by defining appropriate boundaries for chunks.

Valid values:

  • NONE:

    Splits at the MAX limit of characters, words, or vocabulary tokens.

  • NEWLINE, BLANKLINE, and SPACE:

    These are single-split character conditions that split at the last split character before the MAX value.

    Use NEWLINE to split at the end of a line of text. Use BLANKLINE to split at the end of a blank line (sequence of characters, such as two newlines). Use SPACE to split at the end of a blank space.

  • RECURSIVELY:

    This is a multiple-split character condition that breaks the input text using an ordered list of characters (or sequences).

    RECURSIVELY is predefined as BLANKLINE, NEWLINE, SPACE, NONE in this order:

    1. If the input text is more than the MAX value, then split by the first split character.

    2. If that fails, then split by the second split character.

    3. And so on.

    4. If no split characters exist, then split by MAX wherever it appears in the text.

  • SENTENCE:

    This is an end-of-sentence split condition that breaks the input text at a sentence boundary.

    This condition automatically determines sentence boundaries by using knowledge of the input language's sentence punctuation and contextual rules. This language-specific condition relies mostly on end-of-sentence (EOS) punctuations and common abbreviations.

    Contextual rules are based on word information, so this condition is only valid when splitting the text by words or vocabulary (not by characters).

    Note: This condition obeys the BY WORD and MAX settings, and thus may not determine accurate sentence boundaries in some cases. For example, when a sentence is larger than the MAX value, it splits the sentence at MAX. Similarly, it includes multiple sentences in the text only when they fit within the MAX limit.

  • CUSTOM:

    Splits based on custom split characters list. You can provide custom sequences up to a limit of 16 split character strings, with a maximum length of 10 each.

    Provide valid text literals as follows:
    VECTOR_CHUNKS(c. doc, BY character SPLIT CUSTOM ('<html>' , '</html>')) vc

    You can omit sequences only for tab (\t), newline (\n), and linefeed (\r).

Default value: RECURSIVELY

OVERLAP

Specifies the amount (as a positive integer literal or zero) of the preceding text that the chunk should contain, if any. This helps in logically splitting up related text (such as a sentence) by including some amount of the preceding chunk text.

The amount of overlap depends on how the maximum size of the chunk is measured (in characters, words, or vocabulary tokens). The overlap begins at the specified SPLIT condition (for example, at NEWLINE).

Valid value: 5% to 20% of MAX

Default value: 0

LANGUAGE

Specifies the language of your input data.

This clause is important, especially when your text contains certain characters (for example, punctuations or abbreviations) that may be interpreted differently in another language.

Valid values:

You must use double quotation marks (") for any language name with spaces. For example:

LANGUAGE "simplified chinese"

For one-word language names, quotation marks are not needed. For example:

LANGUAGE american

Default value: NLS_LANGUAGE from session

NORMALIZE

Automatically pre-processes or post-processes issues (such as multiple consecutive spaces and smart quotes) that may arise when documents are converted into text. Oracle recommends you to use a normalization mode to extract high-quality chunks.

Valid values:

  • NONE:

    Applies no normalization.

  • ALL:

    Normalizes common multi-byte (unicode) punctuation to standard single-byte.

  • Normalization options such as:

    • PUNCTUATION:

      Includes smart quotes, smart hyphens, and other multi-byte equivalents to simple single-byte punctuation.

      For example:

      • 2018u 'map to 0027'

      • 2019u 'map to 0027'

      • 201Bu 'map to 0027'

    • WHITESPACE:

      Minimizes whitespace by eliminating unnecessary characters.

      For example, retain blanklines, but remove any extra newlines and interspersed spaces or tabs: " \n \n " => "\n\n"

    • WIDECHAR:

      Normalizes wide, multi-byte digits and (a-z) letters to single-byte.

      These are multi-byte equivalents for 0-9 and a-z A-Z, which can show up in ZH/JA formatted text.

    Note: You must specify a comma-separated list of WHITESPACE, PUNCTUATION, and WIDECHAR in parentheses.

Default value: None

EXTENDED

Increases the output limit of a VARCHAR2 string to 32767 bytes, without requiring you to set the MAX_STRING_SIZE parameter to EXTENDED.

Default value: 4000 or 32767 (when MAX_STRING_SIZE=EXTENDED)

Example

CREATE TABLE documentation_tab (
  id   NUMBER,
  text VARCHAR2(2000));

INSERT INTO documentation_tab 
   VALUES(1, 'sample');

COMMIT;

SET LINESIZE 100;
SET PAGESIZE 20;
COLUMN pos FORMAT 999;
COLUMN siz FORMAT 999;
COLUMN txt FORMAT a60;

PROMPT SQL VECTOR_CHUNKS
SELECT D.id id, C.chunk_offset pos, C.chunk_length siz, C.chunk_text txt
FROM documentation_tab D, VECTOR_CHUNKS(D.text 
                                  BY words
                                  MAX 200
                                  OVERLAP 10
                                  SPLIT BY recursively
                                  LANGUAGE american
                                  NORMALIZE all) C;

See Also:

Parent topic: Chunking and Vector Generation Functions