2.2 Tokens, Context Windows, and Inference Parameters
Explores how LLMs process language through tokenization, context window limits, and generation parameters like Temperature and Top-P.
🎯 Key AWS AI Exam Takeaways
- Tokens are subword fragments; ~1,000 English tokens corresponds to approximately 750 words.
- Context windows define hard boundaries for combined prompt and completion tokens.
- Temperature controls output randomness: 0.0–0.2 for deterministic factual output; 0.7–1.0 for creative generation.
Language models do not perceive words as whole entities. Instead, text is decomposed into numerical subword tokens using algorithms like Byte-Pair Encoding. In standard English, one token averages four characters or roughly 0.75 words. Both API billing and hardware memory consumption are measured strictly in tokens.
Every model operates within a bounded 'context window'—the maximum total token count it can ingest and generate within a single inference call. Exceeding this boundary results in request truncation or API validation errors, necessitating intelligent document chunking strategies.
Developers tune generation dynamics using inference hyperparameters. Temperature scales output probability dispersion: low temperature (near 0.0) yields deterministic, greedy token selection, whereas higher values (0.8+) introduce creative diversity. Top-P (nucleus sampling) dynamically restricts the token selection pool to those comprising the top cumulative probability mass.
⚠️ Common Pearson VUE Exam Traps
- Do not set high temperature for compliance, financial calculations, or code generation.
- Remember that Top-P (nucleus sampling) cuts off low-probability tokens based on cumulative probability mass.
Knowledge Checkpoint
A banking compliance application requires an LLM to extract numerical financial metrics from earnings reports. The answers must be strictly factual, deterministic, and repeatable with minimal randomness. Which inference parameter setting should the engineering team configure?