Why Use Squared Error Rather Than Absolute Error?

Squared error and absolute error answer different estimation questions. Squared error emphasizes large residuals and targets a conditional mean; absolute error is less sensitive to large response errors and targets a conditional median. Neither is universally the correct loss.

First distinguish the quantities

For observations $(x_i,y_i)$ and predictions $f_\theta(x_i)$ , define residuals $r_i=y_i-f_\theta(x_i)$ . Then

\|r\|_1=\sum_{i=1}^{N}|r_i|,\qquad \|r\|_2=\sqrt{\sum_{i=1}^{N}r_i^2},\qquad \mathrm{MSE}=\frac1N\|r\|_2^2.

The common regression loss is the squared $L^2$ norm, up to a constant, rather than the norm itself. Without other objective terms, minimizing MSE, the sum of squared errors, RMSE, or $\|r\|_2$ gives the same set of minimizers. Their derivatives differ, and replacing one by another inside a regularized objective need not preserve the minimizer.

A likelihood justification

Suppose the observation model is

y_i=f_\theta(x_i)+\varepsilon_i, \qquad\varepsilon_i\overset{\mathrm{ind}}\sim\mathcal N(0,\sigma^2),

with common fixed variance $\sigma^2>0$ , conditional on the inputs. The negative log likelihood is

-\log p(y_1,\ldots,y_N\mid x_1,\ldots,x_N,\theta) =\frac N2\log(2\pi\sigma^2) +\frac1{2\sigma^2}\sum_{i=1}^{N}r_i^2.

The first term does not depend on $\theta$ , so maximum likelihood selects the parameters minimizing squared error. We optimize over $\theta$ , with the observed $y_i$ held fixed. Likelihood is the data density regarded as a function of the parameters, not a probability distribution over those parameters. Ng's CS229 notes give this derivation.

Independent Laplace errors with fixed scale $b>0$ instead have density proportional to $e^{-|\varepsilon|/b}$ , producing an absolute-error objective. Neither noise assumption is automatic. In particular, the central limit theorem does not establish that every measurement or prediction error is Gaussian.

Mean versus median

There is also a justification that does not assume Gaussian noise. If $Y$ has finite second moment and $\mu=\mathbb E[Y]$ , then

\mathbb E[(Y-a)^2]=\operatorname{Var}(Y)+(a-\mu)^2.

Thus the expected squared error is minimized uniquely at the mean. Applying the same argument conditionally on $X=x$ gives the conditional mean as the unrestricted optimal prediction.

For absolute error, assuming a finite first moment, a minimizer is any median: a value $m$ with

\Pr(Y\le m)\ge\frac12,\qquad\Pr(Y\ge m)\ge\frac12.

For a distribution with a continuous density, differentiating $\mathbb E|Y-a|$ with respect to $a$ gives $2F(a)-1$ , so a minimum occurs where $F(a)=1/2$ . The median condition above also handles atoms and nonunique medians.

For the sample $0,0,10$ , the squared-error constant prediction is $10/3$ , while the absolute-error prediction is $0$ . The distinction reflects the loss we chose, rather than an algebraic superiority of one answer.

Robustness and optimization

A residual of magnitude $10$ contributes $100$ to squared loss but $10$ to absolute loss. For squared loss, the derivative with respect to the residual is $2r$ ; for absolute loss away from zero, it is $\operatorname{sign}(r)$ . This explains why a large response residual exerts more influence under squared loss. Absolute-error regression can still be vulnerable to unusual input points with high leverage; “more robust” is not “immune to outliers.”

Squared loss is smooth and leads to a quadratic objective for linear regression. Absolute loss is convex but nondifferentiable at zero; subgradient methods or linear programming can handle it. With a nonlinear model $f_\theta$ , neither loss guarantees a convex optimization problem in $\theta$ .

What “best” means in linear regression

For $y=X\beta+\varepsilon$ , assume $X$ has full column rank, $\mathbb E[\varepsilon\mid X]=0$ , and $\operatorname{Cov}(\varepsilon\mid X)=\sigma^2I$ . Ordinary least squares is the best linear unbiased estimator of $\beta$ , where “best” means minimum covariance within that class. Gaussian errors are not required for this Gauss–Markov result, and the result does not compare OLS with every biased or nonlinear estimator.

Classification also requires a loss suited to its target. Cross-entropy corresponds to a categorical likelihood, while squared error on predicted probabilities gives the Brier score. It is inaccurate to say that squared loss is inherently invalid for classification.

Choose the loss according to the quantity to predict, the cost of errors, the data assumptions, and the optimization problem. The likelihood derivation is one justification for squared error, not a rule that all errors should be squared.