What is the difference between a system prompt and a user prompt, and how do you reliably get structured output like JSON out of a model?

The system prompt sets durable instructions, role, tone, and constraints that should govern the whole conversation, while user prompts carry the actual per-turn request; keeping instructions in the system prompt makes them harder to override and easier to reuse.

To get structured output reliably you do not just ask nicely -- you use the provider's JSON mode or structured-output feature, or supply a function/tool schema, so the model is constrained to emit valid JSON matching a shape. In the Java world Spring AI's entity mapping or a BeanOutputConverter and LangChain4j's typed AiServices let you deserialize directly into a POJO. The senior nuance is that you still validate and handle parse failures, because 'usually valid JSON' is not a contract, and you keep the schema small since every field description is tokens you pay for.