What happens behind the scenes when we change effort for same LLM models?
What really happens behind the scenes when we select high effort vs medium effort for the same LLM models?
What really happens behind the scenes when we select high effort vs medium effort for the same LLM models?
Discussion Highlights (3 comments)
boveyking
You have a restaurant and have 2 types of chef. One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge. One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.
cyanydeez
llamacpp has the basics: reasoning-budget limits thinking token output and reasoning-message thats injected when budget is exhausted. client can set these per request so dynamics are possible. models dont seem to care if you cut them off.
neongreen
This is a very good question and the answers are different for different models. --- Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main... > When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked. --- Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token": https://magazine.sebastianraschka.com/p/controlling-reasonin... > During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.