Show the code
Simple Linear Regression Modeling
Robin Donatello
2024-10-28
Purpose & Estimation
Purpose of Regression Modeling
- Learn more about the relationship between several independent or predictor variables and a quantitative dependent (response) variable.
- Regression is widely used in research as it allows us to ask the general question “what is the best predictor of…”, and does “additional variable A” or “additional variable B confound the relationship between my explanatory and response variable?”
Both Regression and Correlation can be used to
- Descriptive: Draw inferences regarding the relationship
- Predictive: Predict the value of Y for given values of one or more X’s.
Examples in practice
- Educational researchers might want to learn about the best predictors of success in high-school.
- Sociologists may want to find out which of the multiple social indicators best predict whether or not a new immigrant group will adapt to their new country of residence.
- Biologists may want to find out which factors (i.e. temperature, barometric pressure, humidity, etc.) best predict caterpillar reproduction.
Example - Lung function
Lung function data were obtained from an epidemiological study of households living in four areas with different amounts and types of air pollution. The data set used in PMA6 is a subset of the total data. In this example we use only the data taken on the fathers, all of whom are nonsmokers.
One of the major early indicators of reduced respiratory function is FEV1 or forced expiratory volume in the first second (amount of air exhaled in 1 second). Since it is known that taller males tend to have higher FEV1, we wish to determine the relationship between height and FEV1. We can use regression analysis for both a descriptive and predictive purpose.
- Descriptive: Describing the relationship between FEV1 and height
- Predictive: Determine the expected or normal FEV1 for a given height
Visualize the relationship
There does appear to be a tendency for taller men to have higher FEV1. Since this relationship is reasonably linear (the blue loess line is similar to the red linear line) we can write the model the population average FEV μy as a linear function of height x:
μy=β0+β1x
The intercept parameter, β0, represents where the line crosses the y-axis when x=0. The slope parameter, β1, represents the change in μy per 1 unit x.
Unifying model framework
We know that there is always random noise in real data (DATA = MODEL FIT + RESIDUAL) so we introduce a random error term, ϵi and assume the model:
yi=β0+β1X+ϵiϵi∼N(0,σ2)
This model states that the random variable y to be made up of a predictable part (a linear function of x) and an unpredictable part (the random error, ϵi). The error (residual) term includes the effects of all other factors, known or unknown.
Least Squares Estimation
- Most common method of fitting a straight line to two variables.
- Also known as “Ordinary Least Squares (OLS)”
- Calculates sample statistics b0 and b1 to estimate the population parameter values β0 and β1
- The estimated mean function is ˆyi=b0+b1xi
- The fitted value, ˆyi, is the estimated value for point i, calculated by plugging in a value for xi and calculating the result.
- The residual is the difference between the observed and the fitted value: ϵi=yi−ˆyi
Least Squares Estimation
The estimates b0 and b1 are found such that they minimize the sum of the squared residuals (the unexplained residual error)
n∑i=1ϵi
For simple linear regression the regression coefficient estimates that minimize the sum of squared errors can be calculated as:
^β0=ˉy−^β1ˉx and ^β1=rsysx
where r is the correlation coefficient between x and y.
Partitioning the Variance
Revisiting the Sum of Squares
Go to: https://paternogbc.shinyapps.io/SS_regression/. Then turn and talk to your group about the following features of the Sum of Squares Graphs
Total
- What are the dots?
- What does the horizontal line represent?
- What do the blue lines represent?
Regression
- What does the horizontal line represent?
- What does the sloped line represent?
- What are the green lines?
Error
- What are the dots?
- What does the sloped line represent?
- What do the red lines represent?
Sum of Squares - Regression
- SS Total- how far are the points away from ˉy?
- SS Regression - how far away is the regression line from ˉy?
- SS Error - how far are the points away from the estimated regression line?
Looking at it this way, we are asking “If I know the value of x, how much better will I be at predicting y than if I were just to use ˉy?
Fitting the model
Least Squares Estimation - in R
A linear model using least squares estimation can be performed in R using the function lm(y ~ x)
Call:
lm(formula = FFEV1 ~ FHEIGHT, data = fev)
Coefficients:
(Intercept) FHEIGHT
-4.0867 0.1181 The regression equation for the model to explain FEV1 using height as a predictor is:
ˆy=−4.087+0.118x
Using this model
ˆy=−4.087+0.118x
- b0: For a father that is 0 cm tall, the predicted FEV is -4.087 (an impossible result)
- b1: For every additional cm taller a father is, his predicted FEV increases by 0.118L.
- ˆyx=70: A father who is 70cm tall has a predicted FEV1 of −4.087+0.118(70)=4.17L
Other facts about LS regression
- A change of one sd in x corresponds to a change of r sd in y since b1=rsysx.
- If the correlation is 0, the slope of the LS line is 0. A test of β1=0 is equivalent to a test of ρ=0.
- The LS line always passes through the point (ˉx,ˉy).
- The distinction between explanatory and response variables is essential in regression. Reversing x and y results in a different regression line.
- The Root Mean Squared Error (RMSE) is an estimate for σ.
Assumptions
Mathematical Model
The mathematical model that we use for regression has these features that translate into assumptions.
Y|X∼N(μY|X,σ2)μY|X=β0+β1Xσ2=Var(Y|X)
- Independence The observations are the result of a simple random sample and thus are independent from each other
- Linearity: The mean of Y values at any given X follows a straight line
- Normality: Y values are normally distributed at any given X
- Homoscedasticity The variance of Y values at any X is σ2 (same for all X)
- The last two assumptions are checked by examining the residuals, and so can only be checked after the model has been fit.
Assumption - Independence
- Knowledge of the method of data collection here is key!
- If the sampling method is not random, observations may not be independent.
- No real good way to “test” for independence. Need to know how the sample was obtained.
- Non simple random sample will not result in the same variance estimates
- Can use hierarchical/multi-level models for clustered samples
Assumption - Linearity
- Slight departures OK
- Can use transformations of strongly skewed data to achieve it
- The lowess trend line reasonably follows a linear pattern.
Assumption - Normality
- Slight departures OK
- Can use transformations to achieve it
These plots are generated from the performance package.
- The residuals follow a normal distribution well.
- Don’t let the pattern of the residuals on the qqplot fool you, the y-axis is very zoomed in.
Assumption - Homoscedasticity
- If the variance (std. residual) changes with the value of ˆy, that is a sign of non-constant variance.
- Violations result in reduced validity of inference and typically larger standard errors for the coefficients.
- Could be caused by outliers or model mis-specificiation (e.g. non-normal data)
- FEV Example - even though there is a slight increase in the trend of fitted values against the residuals, this is within the tolerance range.
Out of range predictions
Caution!
The linear model is only valid within the range of the data used to fit the model
To take an extreme example, suppose a father was 2 feet tall. Then the equation would predict an impossible negative value of FEV1 (−1.255).
A safe policy is to restrict the use of the equation to the range of the X observed in the sample.
Model-check
Inference
Distribution of parameter estimates
- The estimated coefficients are functions of both x and y, and they are not themselves independent of each other (e.g. Cov(b0,b1)≠0).
- The joint vector ˆβ(y,x)=(b0,b1) has a multivariate normal distribution, with variance that depends on the predictor variables x only.
ˆβ(y,x)∼N(β,xTx−1σ2)
- The normality of the vector ˆβ is quite robust to the model assumptions.
- Even if the residuals are not normally distributed, the CLT ensures that the ˆβ are close to normal
- When sample sizes are low in a category, or Var(X) is close to zero, xTx can’t be inverted - leading to errors of “non positive definite”
Calculating Confidence Intervals
bp±1.96∗SE(bp)
point estimate ± critical value * standard error of estimate
But calculating the variance of bp involves xTx−1σ2, which is outside the scope of this class. So, we use R functions
We can be 95% confident that the true slope parameter between a fathers height and his FEV1 is contained in the interval (0.085, 0.151).
Hypotheis Testing
Let β1 be the true slope parameter that describes the change in FEV1 as a function of height in cm.
- H0:β1=0 There is no linear relationship between FEV1 and Height
- HA:β1≠0 Alternate Hypothesis: There is a linear relationship between FEV1 and Height
| term | estimate | std.error | statistic | p.value |
|---|---|---|---|---|
| (Intercept) | -4.087 | 1.152 | -3.548 | 0.001 |
| FHEIGHT | 0.118 | 0.017 | 7.106 | 0.000 |
The p-value for b1 is <.0001, so there is sufficient evidence to believe that there is a linear relationship between FEV1 and Height of fathers.
Write a conclusion
Base R (ish)
Conclusion
Each 1cm increase in height of a father is associated with a significant increase of 0.118 (0.09, 0.15) L of FEV1 (p<.0001). Height explains 25.4% of the variation in FEV1.
🔗 https://math615.netlify.app / Linear Regression
Simple Linear Regression Modeling Robin Donatello 2024-10-28