Why can the same prompt give different answers, and what levers control that variability in a backend service?

Generation samples from a probability distribution over next tokens, so unless sampling is fully greedy there is inherent randomness, and even at temperature zero you can see variation across model versions, hardware, or batching.

The main levers are temperature (lower is more focused and repeatable, higher is more creative), top-p/top-k sampling limits, and a seed parameter where the provider supports it, plus pinning the exact model version so a silent upgrade does not change behavior. For backend reliability you typically run extraction and classification tasks at low temperature for consistency, and you never assume bit-for-bit reproducibility even so. Interviewers are checking that you know determinism is not guaranteed, which is exactly why you validate outputs, pin models, and build evals rather than trusting a single golden response.