Reach for an LLM when the task involves open-ended natural language, ambiguity, or synthesis that is hard to express in rules, and avoid it when you need determinism, sub-millisecond latency, exact arithmetic, or auditable correctness.
A regex, a lookup table, a SQL query, or a small classical ML model is cheaper, faster, testable, and reproducible, so using an LLM to, say, validate an email format or sum a column is strictly worse. The honest framing for a PM is a cost/latency/reliability tradeoff: every LLM call adds money, tens to thousands of milliseconds, and a non-zero hallucination rate, so it should earn its place. A strong candidate also proposes hybrid designs -- classic code for the deterministic core, the LLM only for the genuinely fuzzy part -- rather than making the model responsible for correctness it cannot guarantee.
- Using an LLM for deterministic logic like tax rules.
- Not weighing cost and latency against a simple rule.