Multiple regression analysis

When one $x$ variable is not enough

Fernando Rios-Avila

Levy Economics Institute

August 26, 2024

Multivariate Regression

Whenever you start modeling an outcome $y$ , there will always be two factors that will determine that outcome:
- Factors that you can control (e.g., education, experience, etc.)
- Factors that you cannot control (e.g., errors)
Multiple regression analysis uncovers average $y$ as a function of more than one $x$ variable: $y^E = f(x_1, x_2, ...)$ .
It can lead to better predictions $\hat{y}$ by considering more explanatory variables.
It may improve the interpretation of slope coefficients by comparing observations that are similar in terms of other $x's$ variables.

MR: Ommited Variable Bias

So it turns out that:

$\alpha_1 = \beta_1 + \beta_2 \delta_1 \rightarrow \beta_1-\alpha_1 = -\beta_2 \delta_1$

By “ignoring” $x_2$ in the first regression, we are actually estimating a biased coefficient for $x_1$ .
- Assuming the second model is the “true” model
This is what is known as the Omitted variable bias OBV
- Note: You could also have a bias because you are including a variable that should not be there. This is known as bad control.

Simple example

TS regression: Regress month-to-month change in log quantity sold of Beer ( $y$ ) on month-to-month change in log price ( $x_1$ ).
- $\beta = -0.5$ : sales tend to decrease by 0.5% when our price increases by 1%.
Robustness: $x_2$ : change in ln average price charged by our competitors
- New Results: $\hat{\beta}_1 = -3$ and $\hat{\beta}_2 = 3$
There is a OBV (Model 1 is flatter than Model 2)
Possibly the result of two things:
- a positive association between the two price changes ( $\delta_1$ ) and
- a positive association between competitor price and our own sales ( $\beta_2$ ).

MR: Some language

Setup: Multiple regression with two explanatory variables ( $x_1$ and $x_2$ ),
Technicallity:: We measure differences in expected $y$ across observations that differ in $x_1$ but are similar in terms of $x_2$ .
Interpretation: Difference in $y$ by $x_1$ , conditional on $x_2$ . OR controlling for $x_2$ .
- We condition on $x_2$ , or control for $x_2$ , when we include $x_2$ in a multiple regression that focuses on average differences in $y$ by $x_1$ .
What we care is $x_1$ ’s effect on $y$ , but we control for $x_2$ to get a better estimate of this effect.
Confounding: $x_2$ is a confounder if $x_2$ is correlated with $x_1$ and $y$ .
- Thus, we have a problem if we omit $x_2$ from the regression.

MR: Collinearity

Perfectly collinearity is when $x_i$ is a linear function of $x_{-i}$ .
- Consequence: cannot calculate coefficients.
- One will be dropped by software (but you should know which one).
Strong but imperfect correlation between explanatory is sometimes called multicollinearity.
- Consequence: We can get the slope coefficients and their standard errors,
- But, the standard errors may be large.

MR: Collinearity and SE

Strong multicollinearity is a problem because it increases the standard errors of the coefficients.
- It is typically a problem when the sample size is small.
Numerically, it could make the coefficient estimates unstable. (rare)
More often, you may need to either drop one of the variables, or
Combine them into a single variable. (index)

How to know how strong is the multicollinearity problem ??

Estimate $R^2$ of the regression of $x_i$ on all other $x$ variables. For all cases!
- or use the estat vif command in Stata. (only with OLS)

MR: Testing Single hypotheses

Same as before, but now we have more than one $x$ variable to test.
- $H_0: \beta_1 = 0$
You may want to be careful with multiple testing.
- testing each coefficient separately with the same $\alpha$ level
There is also testing single hypotheses on combinations of coefficients.
- $H_0: _1-2*_2=0 $
As before, you just need to know the point estimate and the standard error to calculate the t-statistic.

MR: Testing Joint hypotheses

Testing joint hypotheses: null hypotheses that contain statements about more than one regression coefficient: $H_0: \beta_1 = \beta_2 = 0$ vs $H_1: H_0$ is false
This kind of test is used to evaluate a subset of the coefficients (such as all geographical variables) are all zero.
But for doing this you need a new test statistic: the F-test.

Difference with the t-test:

In contrast with the t-test, the F-test follows an F-distribution.
This distribution is not symmetric! And you need to know the degrees of freedom.
- How many restrictions are you imposing? and how many coefficients did you estimate?
Also, all test are on-sided

MR: Non-linear patterns

Surprise! you can use the same tools as with single regression
- Uses splines, polynomials, other non-linear functions of $x$ variables.
Non-linear function of various $x_i$ variables may be combined.
As show before, using non-linear functions will increase multicollinearity, but worry not about that type of collinearity.
Be more careful with the interpretation of the coefficients.

Variables	ln wage	ln wage	age
female	-0.195**	-0.185**	-1.484**
	(0.008)	(0.008)	(0.159)
age		0.007**
		(0.000)
Constant	3.514**	3.198**	44.630**
	(0.006)	(0.018)	(0.116)
Observations	18,241	18,241	18,241
R-squared	0.028	0.046	0.005

Variable	Model 1	Model 2	Model 3	Model 4
female	-0.195**	-0.185**	-0.183**	-0.183**
	(0.008)	(0.008)	(0.008)	(0.008)
age		0.007**	0.063**	0.572**
		(0.000)	(0.003)	(0.116)
age2			-0.001**	-0.017**
			(0.000)	(0.004)
age3				0.000**
				(0.000)
Observations	18,241	18,241	18,241	18,241
R-squared	0.028	0.046	0.060	0.062

MR: Qualitative variables

MR can also handle using qualitative variables as explanatory variables.
Two ways to include qualitative variables:
- Create a dummy for each category.
- Let the software create the binary variables for you.
You can only include $k-1$ dummies (dummy variable trap)
- Left out category is the reference category (Base).
Stata Corner
- i. in front of a variable tells Stata to treat it as a categorical variable. (makes dummies on the background): reg lnwage i.educ
- But, the categorical variable cannot be negative.

Variables	ln wage	ln wage	ln wage
female	-0.195**	-0.182**	-0.182**
	(0.008)	(0.009)	(0.009)
ed_Profess		0.134**	-0.002
		(0.015)	(0.018)
ed_PhD		0.136**
		(0.013)
ed_MA			-0.136**
			(0.013)
Constant	3.514**	3.473**	3.609**
	(0.006)	(0.007)	(0.013)
Observations	18,241	18,241	18,241
R-squared	0.028	0.038	0.038

MR: Interaction with two continuous variable

Interactions can also be used with continuous variables, $x_1$ and $x_2$ : $y_E = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2$
Example:
- $y$ is change in revenue $x_1$ is change in global demand, $x_2$ is firm’s financial health
- The interaction can capture that drop in demand can cause financial problems in firms, but less so for firms with better balance sheet.
Perhaps biggest challenge is to interpret a model with interactions.

MR: Interaction with two continuous variable

The interaction term $x_1 x_2$ captures how this two variables affect each others effect on $y$ .
- Typically we assume this is zero.
The coefficient $\beta_3$ captures the magnitude of the interaction effect.
However, if you are interested in the relationship betwee $x_1$ or $x_2$ with $y$ , you need extra care.
- $x_1$ on $y$ is $\frac{dy}{dx_1}=\beta_1 + \beta_3 x_2$ .
- $x_2$ on $y$ is $\frac{dy}{dx_2}=\beta_2 + \beta_3 x_1$ .
- So you need to “fix” $x_2$ to see the effect of $x_1$ on $y$ . (typically at the mean)

CS: Interaction between gender and age

Separate, Earning for men rises faster with age
With interaction, Same result!.
Female dummy is close to zero. Does this mean no gender gap?
- No, Cumulative effect: $-0.036-0.003*age$

Variables	ln wage (Women)	ln wage (Men)	ln wage (All)
female			-0.036
			(0.035)
age	0.006**	0.009**	0.009**
	(0.001)	(0.001)	(0.001)
female × age			-0.003**
			(0.001)
Constant	3.081**	3.117**	3.117**
	(0.023)	(0.026)	(0.026)
Observations	9,685	8,556	18,241
R-squared	0.011	0.028	0.047

MR: Causal analysis

One main reason to estimate multiple regressions is to get closer to a causal interpretation.
By conditioning on other observable variables, we can get closer to comparing similar objects – “apples to apples” – even in observational data.
But getting closer is not the same as getting there.
In principle, one could try conditioning on every potential confounder: variables that would affect $y$ and the causal variable $x_1$ at the same time.
Ceteris paribus = conditioning on every such relevant variable. (everything else constant).

Variables	ln wage (Model 1)	ln wage (Model 2)	ln wage (Model 3)	ln wage (Model 4)
female	-0.224**	-0.212**	-0.151**	-0.141**
	(0.012)	(0.012)	(0.012)	(0.012)
Age and education		YES	YES	YES
Family circumstances			YES	YES
Demographic background				YES
Job characteristics				YES
Union member				YES
Age in polynomial				YES
Hours in polynomial				YES
Observations	9,816	9,816	9,816	9,816
R-squared	0.036	0.043	0.182	0.195

Regression table detour

Regression table with many $x$ vars is hard to present
In presentation, suppress unimportant coefficients
In paper, you may present more, but mostly if you want to discuss them or for sanity check
Sanity check: do control variable coefficient make sense by and large?
Check $N$ of observations: if the same sample, should be exactly the same.
$R^2$ is enough, no need for other stuff (unless other methods are used)

MR: Prediction and benchmarking

Second reason to estimate a multiple regression is to make a prediction
- find the best guess for the dependent variable $y_j$ for a particular observation $j$ $\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1 + \hat{\beta}_2 x_2 + \cdots$
A $\hat{y} \ vs \ y$ Scatter plot is a good way to visualize the fit of a prediction.
- As well as identifying over or underpredictions.
We want the regression to produce as good a fit as possible.
- A common danger: Overfitting the data: finding patterns in the data that are not true in the population
We will discuss more about prediction in the next chapters.

MR: Variable selection for causal questions

Causal question: $x$ impact on $y$ . Having $z$ variables to condition on, to get closer to causality.
Our aim is to focus on the coefficient on one variable. What matters are the estimated value of the coefficient and its confidence interval. Not prediction.
Keep $z$ – keep variables that help comparing units
Drop $z$ if they not matter, or if they are part of the causal mechanism. (affected by $x$ )
- Functional form for $z$ matters only for crucial confounders. (linear is fine)
Present the model you judge is best, and then report a few other solutions – robustness.

1 / 52

Multiple regression analysis When one

$x$ variable is not enough Fernando Rios-Avila Levy Economics Institute August 26, 2024

Multiple regression analysis
Motivation
Topics to cover
How multivariate OLS works
Multivariate Regression
Multivariate Regression
MR: Ommited Variable Bias
MR: Ommited Variable Bias
MR: Ommited Variable Bias
Simple example
MR: Some language
Stata: Multiple regression
Estimation
MR: Standard Errors
MR: Collinearity
MR: Collinearity and SE
MR: Collinearity
MR: Testing Single hypotheses
MR: Testing Joint hypotheses
MR: Testing Joint hypotheses
MR: Testing hypotheses in Stata
MR: Non-linear patterns
CS: Understanding the gender difference in earnings
CS: The data
CS: The model
CS: The Regression
CS: Adjustment and Robustness
Qualitative variables and interactions
MR: Qualitative variables
MR: Qualitative variables
MR: How to pick a reference category?
CS: Gender difference in earnings and education
MR: Interactions
MR: Interactions and parallel lines
MR: Interaction with two continuous variable
MR: Interaction with two continuous variable
CS: Interaction between gender and age
MR: Stata corner
Conditioning and causality
MR: Causal analysis
MR: Causal analysis
MR: Causal analysis Don’t overdo it
MR: Causal analysis
CS: Understanding the gender difference in earnings
Regression table detour
Prediction
MR: Prediction and benchmarking
MR: Variable selection
MR: Variable selection for causal questions
MR: Variable selection – process
MR: Variable selection for prediction
Summary take-away