In statistics, regression analysis is a set of statistical processes used to estimate relationships between a dependent variable and one or more independent variables. It helps one understand how the typical value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed.
Regression analysis is widely used for prediction and forecasting, where its use has substantial overlap with the field of machine learning. Regression analysis is also used to understand which among the independent variables are related to the dependent variable, and to explore the forms of these relationships. In restricted circumstances, regression analysis can be used to infer causal relationships between the independent and dependent variables.
What is p-value?
The p-value is a number between 0 and 1 and interpreted in the following way:
- A small p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis, so you reject the null hypothesis.
- A large p-value (> 0.05) indicates weak evidence against the null hypothesis, so you fail to reject the null hypothesis.
- p-values very close to the cutoff (0.05) are considered marginal (could go either way).
The null hypothesis usually states that there is no relationship between the variables of interest, or no difference between groups. So a small p-value means there likely is a relationship or difference.
The p-value is the probability of obtaining results at least as extreme as the observed results, assuming the null hypothesis is true. So if the p-value is 0.01, there is a 1% chance of getting the observed results under the null. Since 1% is low, the results seem unlikely under the null, so we reject it in favor of the alternative.
P-value in Regression Analysis
In regression analysis, the p-value helps determine the significance of each independent variable (predictor). The null hypothesis is that the coefficient for that variable is 0, meaning it has no relationship with the dependent variable.
For each predictor, the regression analysis calculates a p-value that indicates the likelihood the coefficient would be as extreme as observed if the true coefficient were zero. A low p-value (≤ 0.05) means it is unlikely to observe such an extreme coefficient if the predictor was unrelated to the outcome, leading to rejection of the null hypothesis. The predictor is then considered statistically significant.
Some key points about p-values in regression:
- They depend on the observed data, the regression model, and test assumptions.
- Significance levels like 0.05 are arbitrary cutoffs and should not be overinterpreted.
- A large p-value does not mean the null hypothesis is true, only that there is not enough evidence against it in the data.
- P-values do not measure clinical or real-world significance.
Interpreting the P-value of a Regression Coefficient
Here is a step-by-step interpretation of a regression coefficient p-value:
- State the null and alternative hypotheses:
- Null: The coefficient for predictor X is equal to 0.
- Alternative: The coefficient for predictor X is not equal to 0.
- Find the p-value for the coefficient in the regression output.
- If the p-value is less than the significance level (e.g. 0.05):
- Reject the null hypothesis.
- Conclude there is a statistically significant relationship between X and the outcome.
- If the p-value is greater than the significance level:
- Fail to reject the null hypothesis.
- Conclude there is not enough evidence of a significant relationship between X and the outcome.
The smaller the p-value, the stronger the evidence against the null hypothesis that the coefficient equals zero. But statistical significance should not be confused with practical significance, which considers the real-world impact.
Examples of P-values in Regression
Here are some examples to illustrate interpretation of p-values from regression output:
Example 1
A regression analysis finds that the p-value for the coefficient of predictor X is 0.02. Since this is below the 0.05 significance level:
- Reject the null hypothesis that the coefficient equals 0.
- Conclude there appears to be a statistically significant relationship between X and the outcome variable Y.
Example 2
A regression on predicting job performance finds the p-value for years of education is 0.18. Since this exceeds 0.05:
- Fail to reject the null hypothesis that the coefficient equals 0.
- Conclude there is not sufficient evidence of a significant relationship between education level and job performance.
Example 3
A logistic regression model has a p-value of 0.049 for the age coefficient. This is marginal:
- The evidence against the null hypothesis that the coefficient equals 0 is weak.
- Age may have a relationship with the outcome, but it is not very statistically significant.
- The practical significance of age should be considered relative to other predictors.
Conclusion
In summary, the p-value for a regression coefficient tests the null hypothesis that there is no relationship between that predictor and the outcome. A small p-value (≤ 0.05) indicates the null hypothesis can be rejected – the predictor appears to have a statistically significant effect on the outcome variable. But p-values should not be over-interpreted; statistical significance does not necessarily imply practical significance.