What is Top-P?

Top-P, also known as nucleus sampling, is a configuration setting used in Large Language Models (LLMs) to restrict the predicted next token to come from tokens with the top predicted probabilities^[1]. This sampling setting helps to control the randomness and diversity of generated text^[1]. Top-P sampling selects the top tokens whose cumulative probability does not exceed a certain value (P)^[1]. Values for P range from 0 (greedy decoding) to 1 (all tokens in the LLM’s vocabulary)^[1].

The best way to choose between top-K and top-P is to experiment with both methods (or both together) and see which one produces the results you are looking for^[1].