a marketing mix model to a senior director. The slide showed two beta coefficients side by side: Linear TV at +2.4, Digital TV at +1.8. He nodded, satisfied. Then he asked the question I was dreading.
“Let’s assume we run this with last week’s refreshed data. Same channels, same model, same everything, but one extra week of observations. Will these numbers move?”
They moved. A lot. Linear TV slid to +0.9, Digital TV jumped to +3.2. Same channels. Same model. Slightly different data. Wildly different story about who’s driving sales and how much. To a non-technical stakeholder, this looks like a broken model. To anyone who has spent time inside marketing analytics, it looks painfully familiar. The model isn’t broken. It’s sick, sick with something specific, with a name, and with a deep geometric reason behind it: multicollinearity.
Most data science courses and articles treat multicollinearity as a checkbox: run the VIF, drop a feature, move on. But that diagnostic-and-discard approach hides what’s actually happening inside the matrix. Why do the coefficients explode? Why does the model suddenly stand on quicksand?
In this article, we are going deep. We’ll peel back the layers of linear algebra to see multicollinearity not as a statistical nuisance, but as a geometric collapse. We’ll use Linear TV vs. Digital TV (two channels that real-world marketing models constantly fight to disentangle) as our running example, all the way from the OLS equation to the moment the determinant hits zero and everything falls apart.
Contents
- Section 1: Solving for the betas
- Section 2: The Textbook Extreme
- The Real-World Case: When Features Are Almost the Same
- Section 3: The Hidden Geometry — Why Volume Is Everything
- Section 4: From Determinant to Variance
- Section 5: Same Story, Two More Languages
- Section 6: Watch It Happen – Code Demo
- Section 7: The Last Lesson
Section 1: Solving for the betas
Before we dive into multicollinearity’s mathematical headache, let’s start with the basics. Linear regression boils down to this equation:
Breaking it down: y is what we’re trying to predict, X holds all our input features, β are the coefficients we want to find (basically, how much each feature contributes), and ε is the error, the part we can’t explain. Simple enough!
The Loss Function
We define our error as the difference between the observed values (y) and our model’s predictions (Xβ), using the Sum of Squared Residuals (SSR). In matrix notation, our loss function L(β) is expressed as:
The Scalar Nature and Expansion
A critical point that often confuses beginners is the nature of this equation. The loss function L(β) results in a scalar (a single number representing total error). When we expand the quadratic form, we get:
Optimization: Setting the Derivative to Zero
To find the minimum of this loss function, we take the partial derivative with respect to the vector β. According to the rules of matrix calculus, we are looking for the point where the gradient is zero:
hence we can conclude:
as the final expression that will compute our betas.
Let’s pause and zoom out. The closed-form solution of is beautifully compact, but it hides a fragile assumption; the one that actually behaves! To see what happens when it doesn’t, let’s return to our two TV channels and watch a linear regression model produce, with mathematical confidence, complete nonsense.
Section 2: The Textbook Extreme
When Features Are Truly Identical
Imagine, for a moment, a data engineering bug. Your pipeline accidentally creates two columns from the same source. Let’s call them linear_TV1 and linear_TV2 and both end up in your feature matrix. Numerically, they’re identical: every row, every week, exactly the same spend.
What happens when you fit a regression?
If the duplication is perfect, the model breaks outright: the matrix becomes singular, its determinant collapses to zero, and the inverse simply does not exist. Python throws an error, and you fix the bug before lunch.
But suppose the duplication is almost perfect. For example one of the columns has tiny floating-point noise from a unit conversion. Now the matrix is technically invertible, but only because of rounding errors. The model fits and the predictions look reasonable. And suddenly the coefficients are absurd.
Why? Because there is no unique way to split a single signal into two. If both channels carry exactly the same information, the model can assign +5 to one and 0 to the other, or +2.5 to each, or +100 to one and -95 to the other. All of these produce the same prediction. The model has no mathematical reason to prefer any particular split rather than another. It picks one, essentially at random, dictated by numerical noise and reports it back to you with full confidence, just to surprise your director and undermine your credibility!
This is the textbook extreme of multicollinearity. Easy to spot once you know what you’re looking for, and easy to fix: just remove the duplicate column.
But this is not the case that gets you in trouble in practice. The case that gets you in trouble looks much more innocent.
The Real-World Case: When Features Are Almost the Same
In a real marketing dataset, nobody is duplicating columns. The features are genuinely distinct; Linear TV spend and Digital TV spend really are two different media investments, on two different platforms, measured separately. They are not identical.
They are, however, siblings.
Both serve the same campaign objectives, often with identical creative assets adapted for different screens. Both follow the same quarterly budget cycles, the same product launch windows, the same seasonal pushes. When the marketing team commits to a Q4 brand campaign, both Linear and Digital TV spend rise together. When the budget is trimmed in January, both fall together. They are coordinated by the same media planners, executed by the same agency, and aimed at largely overlapping audiences.
The result, in the data, is a correlation between the two channels that typically sits somewhere between 0.85 and 0.95. They are not duplicates, but week after week, they move together with remarkable fidelity.
Mathematically, this means the matrix is no longer singular. Its determinant is not zero. The inverse exists. The model returns a unique β, with standard errors, with p-values, with everything you’d expect from a healthy regression.
And yet, the moment you add a week of data, remove a week, or perturb a single observation, the coefficients lurch. Linear TV slides from +2.4 to +0.9. Digital TV jumps from +1.8 to +3.2. The total contribution of the two channels combined remains roughly stable, but the split between them (which one is “driving” what) is a number written in sand.
This is the disease that the director in our opening scene was looking at. The model wasn’t broken. The math wasn’t wrong. It’s that the data didn’t contain enough independent information to tell the two channels apart, and the regression, being mathematically obedient, gave back the only kind of answer it could: a confident one that happens to be unstable.
In statistical language, we say the two features are near-collinear, or that the matrix is ill-conditioned. In plain language: the math works, but the answer doesn’t mean what you think it means.
The Geometric Picture Coming
Why does this happen? Why is perfect collinearity easy to catch but near-collinearity the actual practical disaster?
And why does a near-zero determinant produce such wild instability, when a strictly positive determinant should (at least in principle) give us a unique, well-defined answer?
To understand this, we need to stop thinking of features as columns of numbers and start thinking of them as vectors in space. Once we do, multicollinearity stops being a statistical accident. It reveals itself as something else entirely: a geometric collapse, hiding in plain sight inside every regression you’ve ever run.
Section 3: The Hidden Geometry — Why Volume Is Everything
We’ve seen the math, and we’ve seen the model’s identity crisis with Linear and Digital TV. But, as said before, to truly grasp why this happens, we need to stop looking at rows of data and start looking at vectors in space.
Multicollinearity isn’t a statistical glitch. It’s a physical collapse of the space our model lives in. And that is what we’ll try to uncover now.
Before we visualize this collapse, let’s quickly align on a few fundamental tools from linear algebra. They act as the mechanics of our regression, and we need them to follow the rest of the story.
To follow the geometry of a failing model, we need to keep four concepts in mind:
The Transpose
Think of this as “flipping” our data matrix. It’s a reorientation that allows us to perform the next, most critical step:
The Gram Matrix
When we multiply the transpose of X by itself, we aren’t just doing algebra. We are measuring inner products. In plain English, this is a map of correlations. It tells us how much each feature points in the same direction as every other:
For our two TV channels, the off-diagonal entries of this matrix are essentially a scaled correlation: how aligned Linear TV and Digital TV are across the weeks of our dataset.
The Determinant
In linear algebra, the determinant of a matrix is more than just a complex calculation. It is a scaling factor and specifically, when we look at our Gram matrix, the determinant represents the volume of the parallelepiped formed by our feature vectors in n-dimensional space.
The Inverse Constraint
For a unique solution of to exist, our Gram matrix must be invertible. This is only possible if the determinant is non-zero:
The closer this determinant gets to zero, the more our solution loses its footing.
The Meaning in Our Model
In the context of OLS regression, the determinant tells us how much unique space our variables actually occupy.
High Determinant (Stability)
If our features are independent thus pointing in completely different directions in their high-dimensional space then the box they form is wide open, with maximum volume. Each feature contributes clear, distinct information to the model. The system has enough independent dimensions to pin every coefficient down to a single, meaningful value.
Low Determinant (Instability)
As correlation between features grows, the vectors start to align. The box begins to tilt and flatten. Its volume shrinks. The model technically still has a unique solution, but it’s a solution sitting on a thin sliver of space. It is fragile, sensitive, and easy to topple.
Zero Determinant (Collapse)
When collinearity is perfect (remember our hypothetical linear_TV_v1 and linear_TV_v2 duplicate) the vectors lie exactly on top of each other. The n-dimensional volume collapses into a lower dimension. A 3D box becomes a 2D plane. A 2D plane becomes a 1D line. The model has no unique direction left to choose, and the math gives up.
The Dead End
Why does a collapsing volume break the math? Recall our solution for from Part 1:
To compute the inverse of the Gram matrix, we have to divide the adjoint matrix by the determinant:
If the determinant is zero, we are quite literally dividing by zero and thus no solution exists. But even if the determinant is just very small (the realistic case of strongly correlated channels), this division produces massive, unstable numbers. Tiny perturbations in the data get amplified into enormous swings in .
This is the precise reason your coefficients explode. The model isn’t malfunctioning. It is trying to balance itself on a geometrically vanishing surface and small ripples in the data send it sliding from +2.4 to +0.9 and back, every time you refresh the dataset.
A Visual Proof of the Collapse
To make this concrete, imagine three marketing channels. Let’s say Linear TV, Digital TV, and a genuinely independent third channel like Out-of-Home (billboards, transit). When all three contribute independent information, their vectors point in different directions, and together they span a full three-dimensional volume. The model can comfortably distinguish their contributions.
But now suppose Digital TV is essentially a scaled copy of Linear TV and so, they always move together, week after week. Geometrically, the Digital TV vector lies on the same line as the Linear TV vector. Add the independent OOH vector, and instead of a 3D volume we are left with a flat 2D plane: two of our three “directions” are really the same direction.
This is the exact moment the determinant collapses. The matrix loses an effective dimension. The model loses its grip.
The plot below shows both scenarios side by side. In Scenario A, three independent feature vectors span a full 3D cube. In Scenario B, the third vector is a linear combination of the first two vectors, geometrically forced onto the same 2D gray plane as the others. The 3D volume vanishes. This is what near-collinearity looks like in the feature space and what zero determinant stands for.
From Geometry to Statistics
We’ve now seen, geometrically, why everything falls apart when feature vectors crowd together: the volume of the space our model lives in shrinks toward zero, and the inverse of an almost-singular matrix produces wild, unstable answers.
But this geometric picture, beautiful as it is, lives one level of abstraction away from the diagnostics we actually use in practice. When you fit a regression in Python, you don’t see a determinant. You see standard errors. You see VIF scores. You see condition numbers in a model summary.
In the next section, we’ll translate this geometric collapse into the language of statistics — and watch the same story unfold in the numbers that every data scientist reads every day.
Section 4: From Determinant to Variance
We’ve seen the geometric collapse. We’ve watched the volume of our feature space shrink toward zero as Linear TV and Digital TV align. But here’s a question that should bother us: in our opening scene, I said the director watched coefficients swing from +2.4 to +0.9. That’s not what a “broken” model looks like. A broken model would crash, throw a numerical error, refuse to fit. Our model fits perfectly. It returns clean numbers. It just returns different clean numbers every time the data shifts slightly.
What does that mean, mathematically? Where exactly does the instability live?
The answer is hiding in a formula that almost no one talks about, but every statistician quietly knows:
This is the variance-covariance matrix of our OLS estimator. It tells us, for any given dataset, how much our estimated coefficients would fluctuate if we drew slightly different samples from the same underlying world. Its diagonal entries are the squared standard errors of each individual , exactly the SE column you see in any regression summary.
Look at the structure of this formula carefully. The variance of is proportional to the inverse of the Gram matrix. The very same inverse we just spent an entire section showing collapses when features become correlated.
So the chain is clean and uncompromising:
- Linear TV and Digital TV are nearly collinear in our data.
- The columns of X are nearly parallel vectors in observation space.
- The volume of the parallelepiped they span is tiny.
- is very close to zero.
- has entries that blow up meaning division by a near-zero number.
- The diagonal of which is exactly up to a constant becomes enormous.
- The standard errors of our coefficients explode.
This is what your model summary is trying to tell you, every time you see a regression output where the coefficient is +2.4 but the standard error is ±2.1. The point estimate hasn’t moved by itself. What’s happening is that the confidence interval around that estimate is so wide that it comfortably contains +2.4 today, +0.9 tomorrow and -0.3 next week and all of them statistically consistent with the same underlying data.
Your model didn’t change its mind. It never had a mind made up in the first place. It just had a confidence band wide enough to swallow whatever specific number the latest dataset happened to pop out.
This is the precise statistical translation of geometric collapse. The volume of the feature space and the variance of the estimator are not two separate phenomena. They are the same fact, told in different languages. One uses parallelepipeds while the other uses standard errors. Both are saying: we don’t have enough independent information to pin this down.
And once you see this, every diagnostic in your regression output stops being mysterious. The standard errors. The wide confidence intervals. The high p-values on coefficients you know should matter. None of these are bugs. They are the model, honestly reporting back to you that the data isn’t telling it what you wished it would.
In the next section, we’ll see how this same story gets translated one more time into the two diagnostics that data scientists actually reach for in practice: VIF and condition number.
Section 5: Same Story, Two More Languages
By now, you’ve watched the same phenomenon described in two different languages.
In geometric language: the feature vectors collapse toward a common line, and the volume of the space they span shrinks toward zero.
In statistical language: the variance of the estimator explodes, and standard errors become so wide that today’s coefficient of +2.4 and tomorrow’s coefficient of +0.9 are equally consistent with the data.
VIF: How Much of This Feature Is Already Said by the Others?
But there is a third language. This is the one your model summary actually speaks when you fit a regression in Python. It speaks in VIF scores and condition numbers. Most data science courses teach you to glance at these numbers, apply some rule of thumb (VIF above 10 is bad), and move on. What rarely gets explained is that these two diagnostics are not new information at all. They are the exact same story — the volume collapse, the variance explosion — translated one more time, into a form you can read off a single line of code.
The Variance Inflation Factor for a feature is defined as:
Where is the coefficient of determination you’d get if you took a feature i — say, Linear TV — and regressed it against all the other features in your model such as Digital TV, OOH, digital search, social media, and so on.
Take some seconds to read that definition again. It is asking a remarkably honest question: how much of this feature can be predicted from the others? If Linear TV can be reconstructed almost perfectly from the rest of the channels (because, say, Digital TV always moves with it), then is close to 1, the denominator approaches zero, and VIF explodes toward infinity.
When the features are genuinely independent, , the denominator is 1, and VIF equals 1 — its lower bound, indicating no inflation at all.
The name itself tells the rest of the story. Variance Inflation Factor. It is literally measuring how much the variance of gets multiplied because of the feature’s overlap with the others. A VIF of 10 means the standard error of that coefficient is roughly times wider than it would be if the feature was independent. A VIF of 100 means the standard error is 10 times wider. The whole sad chain we walked through in the previous section gets reduced to a single ratio you can compute per feature.
VIF, then, is not a separate concept. It is the per-feature shadow of the geometric collapse: how much each individual variable suffers from the fact that the others already cover its territory.
Condition Number: The Global Health of the Matrix
Where VIF speaks to each feature individually, the condition number speaks to the matrix as a whole.
It comes from the Singular Value Decomposition (SVD) of X. Any matrix can be decomposed into a set of singular values , which measure how much “stretch” the matrix applies along each of its principal directions. Geometrically, these singular values are the lengths of the axes of the n-dimensional ellipsoid that X would carve out of a unit sphere.
The condition number is defined as:
When the features are independent, the ellipsoid is roughly spherical thus all the singular values are similar in size, and their ratio is close to 1. The matrix is well-rounded, balanced and healthy.
When the features become nearly collinear, the ellipsoid flattens. One direction stretches enormously while another collapses toward nothing. shrinks toward zero, and the condition number explodes.
Notice that this is the same geometric collapse we already described, just measured along a different axis. We previously said: the volume of the parallelepiped goes to zero. Now we are saying: the shortest axis of the ellipsoid goes to zero. These are not two different facts. The volume of a parallelepiped is the product of all the singular values; if one of them collapses, the product collapses.
A common rule of thumb: a condition number above 30 suggests serious multicollinearity. Above 100 is severe. But the more useful intuition for me is this: the condition number tells you, in a single number, how much numerical instability is baked into your design matrix before you even start fitting.
One Story, Three Languages
Step back for a moment, and look at what we’ve built.
We started with a single phenomenon: when two features move together, a regression’s individual coefficients become unstable. We then watched this phenomenon get described, in turn, by three different mathematical communities:
- Geometers
- Statisticians
- Numerical Analysts
VIF and condition number are not extra diagnostics. They are the operational summaries of the geometric truth we already understood. Once you see this, the regression output stops being a mysterious table of numbers and becomes a transparent report on the geometric health of your data.
The next time you see a VIF of 47 next to a coefficient, you’ll know exactly what it means: that feature shares roughly a huge portion of its variation with the others. The volume of independent information it brings to the model is almost zero. The coefficient you see is the model’s best guess from a vanishingly thin slice of space.
Section 6: Watch It Happen – Code Demo
Let’s stop describing and start watching. The cleanest way to feel multicollinearity is to construct a tiny synthetic case where we control the correlation directly, fit an OLS regression, and read the standard errors off the model summary.
The code below simulates a marketing-style dataset with three channels: Linear TV and Digital TV (which we will deliberately make correlated to varying degrees), and OOH as a genuinely independent third channel. We then sweep the correlation between the two TV channels from mild to near-perfect, fit the same regression each time, and watch what happens.
import numpy as np
import pandas as pd
import statsmodels.api as sm
np.random.seed(42)
n_weeks = 200
# True coefficients we're trying to recover
true_linear_tv = 2.0
true_digital_tv = 3.0
true_ooh = 1.5
results = []
for corr in [0.3, 0.7, 0.9, 0.95, 0.99]:
# Linear TV: random weekly spend
linear_tv = np.random.uniform(0, 100, n_weeks)
# Digital TV: correlated with Linear TV, controlled by `corr`
noise = np.random.normal(0, 1, n_weeks)
digital_tv = corr * linear_tv + np.sqrt(1 - corr**2) * noise * linear_tv.std()
# OOH: independent
ooh = np.random.uniform(0, 50, n_weeks)
# Sales: true linear combination + noise
sales = (true_linear_tv * linear_tv +
true_digital_tv * digital_tv +
true_ooh * ooh +
np.random.normal(0, 30, n_weeks))
X = sm.add_constant(np.column_stack([linear_tv, digital_tv, ooh]))
model = sm.OLS(sales, X).fit()
results.append({ 'correlation': corr, 'beta_linear_tv': round(model.params[1], 2),
'SE_linear_tv': round(model.bse[1], 2),
'beta_digital_tv': round(model.params[2], 2),
'SE_digital_tv': round(model.bse[2], 2),
'beta_ooh': round(model.params[3], 2),
'SE_ooh': round(model.bse[3], 2),
})
print(pd.DataFrame(results))
As the correlation between Linear TV and Digital TV climbs from 0.3 to 0.99, the standard errors of both TV coefficients inflate dramatically from 0.08 to 0.56, roughly a sevenfold increase. This is the variance explosion we derived analytically.
The point estimates themselves are interesting to watch, too. At low correlation (0.3), the model recovers the true values almost perfectly: against a truth of 2.0, and against the truth of 3.0. But by correlation 0.99, the estimates start to drift visibly as Linear TV slides to 1.52 while Digital TV climbs to 3.54. The point estimates haven’t gone insane, but they have started to slosh: the two coefficients are absorbing each other’s “credit” because the model can no longer cleanly attribute sales to one channel versus the other. The total contribution stays roughly faithful; the individual split does not.
And here is the part that the standard errors quietly confirm: at correlation 0.99, a standard error of 0.56 means the 95% confidence interval for Linear TV’s coefficient roughly spans from 0.4 to 2.6 which is wide enough to contain both the true value and the drifted estimate, and almost anything in between. The model is not lying. It is telling you, in its honest statistical voice, that with this data it cannot pin down individual contributions with confidence.
Crucially, look at the OOH column. Across the entire sweep — from correlation 0.3 to 0.99 — its coefficient stays anchored near its true value of 1.5, and its standard error barely budges. The disease is local to the correlated pair. This is exactly what the geometry predicted: the volume of the feature space collapses only in the directions where features pile on top of each other; everywhere else, the model breathes freely.
Try editing one number at the top, either the random seed, the noise level, the sample size and rerun the code. The story is the same every time. As correlation rises, the standard errors of the correlated coefficients explode, and confidence in any individual contribution dissolves. This is multicollinearity, no longer described but observed.
Section 7: The Last Lesson
Multicollinearity is rarely a question of broken math. Your model fits, your code runs, your output looks clean. The disease hides in the standard errors and in the way coefficients quietly slosh from one number to another between data refreshes.
What we’ve seen is that this is not a statistical accident. It is a geometric truth, dressed up in three different languages. When two features carry nearly the same information, the volume of the space your model lives in shrinks toward zero. The variance of the estimator inflates. VIF and condition number explode. These are not three separate problems. They are one phenomenon, reported by three different communities of mathematicians who never quite realized they were all looking at the same thing.
The practical responses are familiar. Drop a feature, combine features, regularize. Each deserves its own story. But none of them really solves multicollinearity. They manage it. They acknowledge that the data has not given us enough independent information, and they find principled ways to fill the gap.
So the next time a senior stakeholder asks you why your Linear TV coefficient was +2.4 yesterday and +0.9 today, you’ll have a better answer than “the model is sensitive.” The truth is more honest, and more interesting. The feature space has flattened beneath your model, and the coefficient you reported was the model’s best guess from a thin sliver of room. The number didn’t change because the world did. It changed because there was never quite enough room for it to stand still.