Staff Writer Amanda

What are parameters in LLMs? Find out the differences between weights, biases, and hyperparameters, and how they define your LLM’s capability.

What are Parameters in LLMs? Understanding Model DNA







Category
Hyperparameter
What it Does




Training Controls
Learning Rate
Dictates how large a step the model takes to correct its errors during training. Too high, and it overshoots; too low, and training takes forever.


Model Architecture
Number of Layers
Defines the "depth" of the model. More layers mean a more complex model capable of learning richer patterns, but at a higher computational cost.


Output Generation
Temperature
Controls the randomness and creativity of the generated tex







Category
Hyperparameter
What it Does




Training Controls
Learning Rate
Dictates how large a step the model takes to correct its errors during training. Too high, and it overshoots; too low, and training takes forever.


Model Architecture
Number of Layers
Defines the "depth" of the model. More layers mean a more complex model capable of learning richer patterns, but at a higher computational cost.


Output Generation
Temperature
Controls the randomness and creativity of the generated text. Higher values lead to more creative, unpredictable responses, while lower values make output focused and factual.


Output Generation
Context Window
The maximum number of tokens the model can "remember" or process in a single go to maintain coherence.


Output Generation
Top-P (Nucleus)
Controls output diversity by limiting token selection to a cumulative probability threshold p (e.g., 90%). This limits the model to the most likely words in each step for better fluency and coherence.


Output Generation
Top-K
Filters word choices to the k most probable words. Also controls diversity, but by a fixed number of tokens.


Output Generation
Max Tokens
Sets a hard limit on the length of the model's response, preventing overly long or costly outputs.


Output Generation
Presence Penalty
Discourages the model from introducing new topics or tokens it has already used, preventing topic drift or repetition.


Output Generation
Frequency Penalty
Discourages the model from repeating words already frequently used in the generated text, promoting diversity.


Output Generation
Stop Sequence
A custom string (like \n\n or Best regards,) that immediately stops generation when encountered, useful for structured output.





AMD Instinct™ MI355X GPUs Now Available

What are Parameters in LLMs? Understanding Model DNA

Subscribe to our Blog

Stay ahead of the curve with the latest in AI, AMD accelerators, and all things TensorWave.

Product

Resources

Company

© 2025 TensorWave Inc. - All rights reserved.