大多数

LLM/LLMs do not guess. They optimize. Here is the difference and the math behind it.md

@@ -0,0 +1,146 @@
+---
+title: "LLMs do not guess. They optimize. Here is the difference and the math behind it"
+source: "https://x.com/0xDeliriumm/status/2091560754122014781"
+author:
+  - "[[@0xDeliriumm]]"
+published: 2026-08-24
+created: 2026-09-28
+description: "Most people think an LLM is guessing.You type a question. It produces an answer. It feels random, probabilistic, sometimes wrong. It must be..."
+tags:
+  - "clippings"
+---
+![图像](https://pbs.twimg.com/media/HQa3aE0W4AAhPXk?format=jpg&name=large)
+
+Most people think an LLM is guessing.
+
+You type a question. It produces an answer. It feels random, probabilistic, sometimes wrong. It must be guessing.
+
+It is not. Every word an LLM generates is the result of an optimization. Every weight in the model was set by an optimization. The entire field of machine learning is, at its core, the application of one mathematical idea: find the input that makes a function as large or as small as possible.
+
+The word optimization sounds technical. The idea is not. You do it every day. You pick the fastest route to work. You choose the cheapest flight that still arrives on time. You decide how to spend an hour to get the most done. All of these are optimization problems, and the math that solves them formally is the same math running inside every AI system in the world.
+
+Here is that math, from the simplest version to the one that trains a model with trillions of parameters.
+
+What optimization actually means
+
+A function takes inputs and produces an output. Optimization asks: which input produces the smallest output? Or the largest?
+
+Draw a curve on a piece of paper. Optimization finds the bottom of the valley or the top of the hill. That is the entire idea.
+
+For a smooth curve with one variable, you already know how to do this from calculus. The bottom of a valley is where the slope is zero. Find where the derivative equals zero and you have your candidate. Check whether it is a minimum or a maximum by looking at the second derivative. Done.
+
+The difficulty is that real problems are never this clean. The function has millions of variables. It is not smooth. It has constraints. You cannot write down a formula for it. You can only evaluate it at specific points.
+
+The history of optimization is the history of finding ways around each of these difficulties, one by one.
+
+The oldest trick: Newton's method
+
+Suppose you have a smooth function of one variable and you want to find where it equals zero. You cannot solve the equation algebraically. But you can evaluate the function and its derivative anywhere.
+
+Newton's method is this: start somewhere. Draw the tangent line at that point. See where the tangent line crosses zero. Move there. Repeat.
+
+Each step replaces the actual function with its linear approximation at the current point, finds the zero of that approximation, and uses it as the next guess. If the function is well-behaved, each step roughly doubles the number of correct decimal places. Start within a reasonable distance of the answer and you converge extremely fast.
+
+The same idea works for optimization. Instead of finding where a function equals zero, you find where its derivative equals zero, which is the condition for a minimum or maximum. Apply Newton's method to the derivative. Now each step uses the second derivative, the rate of change of the slope, to decide how far to move and in which direction.
+
+This is called Newton's method for optimization, and it is the ancestor of almost every training algorithm used in modern machine learning. The version used to train LLMs is called gradient descent, which is a stripped-down Newton where you ignore the second derivative and just follow the slope downhill. You lose the fast convergence but gain the ability to handle functions with trillions of variables without computing a matrix the size of the parameter space.
+
+Training a language model is gradient descent applied to a function that takes 175 billion or 2.8 trillion weights as input and outputs how wrong the model's predictions are. Every training step moves the weights slightly downhill on that surface. Run it for trillions of steps and the weights settle somewhere that produces useful predictions.
+
+Finding the best without a formula: golden section search
+
+Newton's method requires a derivative. Sometimes you do not have one.
+
+Suppose you are optimizing a function you can only evaluate at specific points. No formula. No derivative. Just: give me an input, I will give you an output.
+
+If the function has a single valley with no flat regions, the golden section search handles this cleanly.
+
+![图像](https://pbs.twimg.com/media/HQa0CtBWgAEIASl?format=png&name=large)
+
+Take an interval you know contains the minimum. Place two probe points inside it, positioned so that whatever you learn from the function values, you can eliminate a fixed fraction of the interval. Evaluate the function at both points. The higher one is further from the minimum. Discard the region beyond it. Now you have a smaller interval and one of your two probes is already inside it, perfectly positioned for the next step.
+
+The fraction you eliminate each step is determined by the requirement that the probe positions stay optimal after the discard. This leads to an equation whose solution is the golden ratio, approximately 0.618. Each step eliminates about 38 percent of the remaining interval.
+
+The result is exponentially fast convergence without ever computing a derivative. The golden ratio appears because it is the unique number that makes the probe positions self-consistent across steps, not because the problem was designed to produce it.
+
+This kind of search runs inside hyperparameter tuning for machine learning models. When you want to find the best learning rate, or the best regularization strength, and evaluating each option requires training a full model, you cannot afford to try everything. Golden section search finds the optimum in logarithmic time.
+
+When the answer lives at a corner: the simplex method
+
+Now suppose the problem is linear. The function you are maximizing is a weighted sum of variables, and the constraints are also linear inequalities.
+
+This is called a linear program, and it describes an enormous range of real problems. Allocating resources across a supply chain. Scheduling flights to minimize delays. Deciding how much of each product to manufacture given limited raw materials. Every time you hear that a company is "optimizing operations," this is usually what they mean.
+
+Linear programs have a remarkable property. The set of points satisfying all the constraints forms a shape with flat faces and sharp edges, a polytope. And the maximum of a linear function over this shape is always achieved at one of its vertices.
+
+This is not obvious but it follows directly from linearity. Moving along any edge of the polytope changes the objective function at a constant rate. If the rate is positive, keep moving until you hit a corner. If negative, you already went the wrong way. If zero, you are on an edge where all points are equally good. Either way, a vertex is optimal.
+
+The simplex method exploits this. Start at any vertex. Look at all the edges leaving it. Move along the one that improves the objective function most. At the new vertex, repeat. Stop when no edge improves things.
+
+George Dantzig invented this in 1947. He was working on military logistics for the US Air Force and needed a systematic way to allocate resources. The algorithm he found became one of the most widely used computational methods in history.
+
+There is an uncomfortable fact about it. In theory, the simplex method can be slow. You can construct pathological examples where it visits an exponential number of vertices before finding the optimum. For decades, mathematicians looked for a provably efficient alternative.
+
+They found one in 1984. Interior point methods, which move through the interior of the feasible region rather than along its boundary, are provably polynomial time.
+
+And yet the simplex method is still widely used, often faster in practice than the theoretically superior alternative. Nobody fully understands why. The pathological cases that make it slow do not seem to appear in real problems. This gap between theory and practice remains one of the genuinely open questions in computational mathematics.
+
+When constraints complicate everything: KKT conditions
+
+Newton and gradient descent handle unconstrained problems. Simplex handles linear constraints. What about nonlinear constraints?
+
+Suppose you want to minimize a function subject to the constraint that you stay on a curve, or inside a region defined by a nonlinear inequality. The minimum of the unconstrained function is probably outside the feasible region, so the answer is somewhere on the boundary.
+
+The classical approach is Lagrange multipliers. At the constrained optimum, the gradient of the objective function must be parallel to the gradient of the constraint. If they were not parallel, you could move along the constraint surface in a direction that improves the objective, contradicting optimality. The condition that the gradients are parallel gives you extra equations to solve alongside the constraint.
+
+![图像](https://pbs.twimg.com/media/HQa0RuLWgAABTc-?format=jpg&name=large)
+
+This works for equality constraints. For inequality constraints, the situation is more subtle. You might be at a point where the constraint is not active, where you are strictly inside the feasible region rather than on its boundary. In that case the constraint is irrelevant and the unconstrained optimum applies. Or the constraint might be active, meaning you are on the boundary, and the Lagrange condition holds.
+
+The Karush-Kuhn-Tucker conditions, developed by William Karush in his 1939 master's thesis and independently by Harold Kuhn and Albert Tucker in 1951, handle both cases. They say: at the optimum, either the constraint is inactive and the gradient condition does not apply, or the constraint is active and a version of the Lagrange condition holds, with the additional requirement that the multiplier has the right sign to push you toward the feasible region rather than away from it.
+
+These conditions are necessary for optimality in constrained problems. Under additional assumptions about the shape of the function and constraints, they are also sufficient. They form the theoretical foundation for a large class of modern optimization algorithms, including those used in training AI models with regularization constraints, in support vector machines, and in reinforcement learning.
+
+The hardest problem: approximating optimally
+
+All of the above assumes you can evaluate the function precisely. Sometimes you cannot. You want to represent a complicated function, say, the exponential, using a simple one, say, a polynomial, and you want the approximation to be as accurate as possible.
+
+What does as accurate as possible mean? You could minimize the average error. Or the sum of squared errors, which is what least squares regression does. Or you could minimize the maximum error, the worst case over all points in the domain.
+
+Minimizing the maximum error is called the minimax problem, and it is significantly harder than minimizing an average. The optimal solution has a special structure discovered by Chebyshev in the 19th century.
+
+The Chebyshev polynomials are a specific family of functions defined by a simple property: among all polynomials of a given degree with leading coefficient one, they have the smallest maximum value on the interval from negative one to one. Their maximum is as small as it can possibly be.
+
+This matters because when you use them as a basis for approximation, the error distributes itself evenly across the interval. Instead of being small in some places and large in others, the error equioscillates, reaching its maximum magnitude at multiple points with alternating signs. Chebyshev proved that this equioscillation is the signature of the optimal approximation. If your error equioscillates, you cannot do better.
+
+This is the theoretical basis for how calculators and computers evaluate transcendental functions like sine, cosine, and logarithm. They do not use Taylor series, which are accurate near one point but deteriorate away from it. They use Chebyshev approximations, which spread the error evenly and achieve the best possible worst-case accuracy with a given number of terms.
+
+The same idea appears in control theory, in signal processing, and in the training of neural networks where you want guarantees on worst-case behavior rather than average behavior.
+
+What all of this has to do with LLMs
+
+Training a language model is gradient descent, a descendant of Newton's method, applied to a function with trillions of variables.
+
+The architecture decisions, how many layers, how large the attention heads, how many experts in a mixture-of-experts model, are tuned using methods related to golden section search and Bayesian optimization.
+
+The constraints imposed during training, such as keeping weights small to prevent overfitting, or ensuring certain fairness properties, are handled by KKT-style conditions embedded in the loss function.
+
+The approximations used in efficient inference, such as quantizing weights to 4-bit integers or approximating attention, are evaluated using minimax criteria, minimizing worst-case degradation rather than average degradation.
+
+None of this is new mathematics. Dantzig invented the simplex method to allocate military supplies. Chebyshev developed his polynomials to study the motion of steam engines. Karush wrote his conditions in a master's thesis that went uncited for a decade.
+
+![图像](https://pbs.twimg.com/media/HQa0wwYXcAAIT04?format=png&name=large)
+
+The math was waiting. The scale of the problems changed. The algorithms did not.
+
+What guessing would look like
+
+An LLM that guessed would produce outputs sampled uniformly at random from its vocabulary. It would say "the capital of France is" and follow it with a random word, with equal probability of Paris, Antarctica, seventeen, or any other token.
+
+It does not do this. It produces a probability distribution shaped by 175 billion or 2.8 trillion weights set by optimization over trillions of examples. The distribution concentrates probability where the data said the answer should be. Paris gets high probability not because of luck but because every training step moved the weights slightly toward making Paris more likely after that prompt.
+
+That is not guessing. That is optimization with a very large function and a very large dataset.
+
+The difference matters, because guessing implies randomness without structure, while optimization implies a search for the best, constrained by what is possible. LLMs are the latter. The output feels unpredictable because the function being optimized is complex enough that its outputs surprise us. But surprise is not the same as randomness.
+
+/ If this was useful - follow /