TAAFT
Free mode
100% free
Freemium
Free Trial
Prompts Deals
Glossary
Artificial Intelligence
Tokens

Tokens

[ˈtoʊkənz]
New Artificial Intelligence
Last updated: 2026-06-05

Definition

The basic units of text or data that AI models process, typically words, subwords, or characters.

Detailed Explanation

Discrete elements that represent text or data in a format processable by AI models. In NLP, tokens can be words, subwords, or characters, created through various tokenization algorithms. The choice of tokenization strategy affects model performance and efficiency.

Use Cases

Language model processing, Text generation, Machine translation, Code completion

Related Terms

Inference Engines

The component of an AI system that applies logical rules to a knowledge base to deduce new information.

Artificial Intelligence

Autonomous Navigation

The ability of robots to plan and execute movement through environments without human intervention.

Artificial Intelligence

Linear Algebra

The branch of mathematics dealing with linear equations, matrices, and vector spaces that forms the backbone of many AI algorithms.

Artificial Intelligence