A residual is the difference between an observed value of the response variable and the value predicted by the regression line.
Residual = observed y - predicted y = y - ŷ
The least-square regression line (LSRL) is the line that makes the sum of the squared residuals as small as possible.
To find the LSRL on a calculator: Calc LSRL -> STAT ->Edit -> L1/L2 -> STAT -> Calc -> 8 -> LinReg -> ZoomStat
Calculating LSRL manually In addition to using technology to find the equation of a LSRL, we are able to find the equation using only the means and standard deviations of the variables and their correlation.
From the data, calculate the means x̄ and ȳ and the standard deviations Sx and Sy of the two variables and their correlation r. The LSRL is the line Ŷ = a + bx with:
Slope: b = r (Sy/Sx) Y-Intercept: a = ȳ - b(x̄)
A residual plot is a scatter plot of the residuals (on the y-axis) against the explanatory variables (on the x-axis).
When interpreting a residual, compare the actual and predicted value using direction, magnitude, and units. For example, for a problem comparing miles driven and price of truck with a residual of -$5004:
The actual price of the truck is $5004 less than predicted based on the number of miles driven.
Important note: Residual plot needs to have a random scatter for a linear model to be necessary.
Residual statistics
If we use a LSRL to predict the values of the response variable y from an explanatory variable x, then the standard deviation of the residuals (s) is:
S = √( (sum of residuals)2 / (n-2) )
When interpreting the standard deviation of the residuals, use this format: “When using [x-value] (x) to predict [y-value] (y), we will typically be off by about [standard deviation of the residuals].”
The Coefficient of Determination (r2) is a numerical quantity that tells us how well the LSRL predicts values of the response variable.
r2 = 1- ( (sum of residuals)2 / (sum of yi - ŷ)2 )
When interpreting the coefficient of determination, use this format (where r2 is transformed into a percentage): “[r2] of the variation in [y-value] is accounted for by the linear relationship between [y-value] (y) and [x-value] (x).”
The correlation coefficient can be found by taking the root of r2 where the sign is determined by the slope. Another reason why it’s always important to graph your data!
Correlation and Regression Wisdom
- The distinction between explanatory & response variables is important in regression. If we reverse the roles of the two variables, we get a different LSRL. However, this isn't true for correlation! Switching x and y doesn’t affect the value of r.
- Correlation and regression lines describe only linear relationships. Always plot your data!
- Correlation and LSRL are not resistant to outliers.
- Association does not imply causation! A strong association between two variables is not enough to draw conclusions about cause and effect.
- When discussing two quantitative variables, we can use the word correlation, but when discussing any two variables, we can use the word association.
Outliers and influential observations in regression
An outlier is an observation that lies outside the overall pattern of the other observations. Points that are outliers in the y direction but not the x direction of a scatter plot have large residuals. Other outliers may not have large residuals.
An observation is influential for a statistical calculation if removing it would change the result of the calculation. Points that are outliers in the x direction of a scatterplot are often influential for the LSRL.