tokenizer

noun
UK /ˈtəʊkənaɪzə/
US /ˈtoʊkənaɪzər/

A tool that breaks something into small pieces.

The tokenizer split the sentence into separate words.

SYNONYMS

splitterparserdividersegmenter

RELATED WORDS

tokentokenizelexercompilertext processing

OTHER MEANINGS

2. A program that splits text into tokens for a computer to understand.

The compiler's tokenizer turned the code into keywords and symbols.

3. A device or routine that separates a data stream into meaningful units.

The network tokenizer divided the incoming bytes into packets.

4. In language study, a tool that divides writing into words or characters.

Researchers used a tokenizer to count the words in the corpus.