"In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables)."
A statistical method for modeling the relationship between multiple variables and a target outcome variable.
Multiple Linear Regression Model: The multiple linear regression model is a statistical model used to establish a relationship between multiple independent variables and one target variable.
Ordinary Least Squares (OLS) Estimation: OLS estimation is a statistical method used to estimate the parameters of a linear regression model. It finds the values that minimize the sum of squared residuals.
Assumptions of Multiple Linear Regression: Assumptions are made to ensure the reliability of the regression results. These include linearity, independence of errors, homoscedasticity, and normality of error terms.
Multicollinearity: Multicollinearity arises when independent variables are highly correlated with each other, making it difficult to estimate their individual effects on the dependent variable.
Dummy Variables: Dummy variables are used to represent non-numeric categorical variables in a regression model. They are binary variables that take the value of either 0 or 1.
Interaction Effects: Interaction effects occur when the effect of one independent variable on the dependent variable depends on the value of another independent variable.
Goodness of Fit Measures: Goodness of fit measures are used to determine the quality of the regression model. The most common measures are R-squared, adjusted R-squared, and root mean square error (RMSE).
Model Selection Criteria: Model selection criteria help to choose the best regression model among many alternative models. The most widely used criteria include Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and Schwarz's Bayesian Criterion (SBC).
Outliers and Influential Observations: Outliers are observations that are significantly different from the rest of the data, while influential observations have a significant impact on the regression results.
Heteroscedasticity: Heteroscedasticity refers to the situation where the variance of the error terms is not constant across observations.
Autocorrelation: Autocorrelation arises when the error terms of the regression model are correlated with each other, violating the independence assumption.
Ridge Regression: Ridge regression is a method used to address multicollinearity in a regression model. It adds a penalty term to the OLS estimation to shrink the coefficients towards zero.
Principal Component Regression (PCR): PCR is a technique used to deal with multicollinearity by creating new variables, known as principal components, that are uncorrelated with each other to replace the original variables in the regression model.
Partial Least Squares (PLS) Regression: PLS regression is another method used to handle multicollinearity. It finds a set of latent variables that are uncorrelated with each other to replace the original variables in the regression model.
Ordinary Least Squares (OLS) Regression: This is the most commonly used type of linear regression in economics. It assumes that the error term is normally distributed, and the coefficients are estimated using a method called least squares.
Weighted Least Squares Regression (WLS): WLS is used when the variance of the dependent variable is not constant across different values of the independent variables. The weights are derived from the variance of the dependent variable.
Generalized Least Squares Regression (GLS): GLS is a more flexible version of WLS. It allows for the correlation between the error terms and is used when OLS assumptions are not met.
Two-Stage Least Squares (2SLS) Regression: SLS is used when there is endogeneity, i.e., the independent variables are correlated with the error term. It is a commonly used technique in econometric analysis of causal relationships.
Three-Stage Least Squares (3SLS) Regression: SLS is an extension of 2SLS that is used when multiple endogenous variables are present.
Seemingly Unrelated Regressions (SUR): SUR is used when there are multiple equations, and the error terms between the equations are correlated. It is used to estimate a system of equations simultaneously.
Vector Autoregressive (VAR) Regression: VAR is used to model the relationships between several variables at the same time. It is useful in analyzing the interactions between multiple economic variables.
Panel Data Regression: Panel data regression is used when the data contains observations of the same units over time or space. It is useful in analyzing changes over time or differences between groups.
Bayesian Linear Regression: Bayesian linear regression is a method that allows for the inclusion of prior knowledge about the coefficients in the model. It is useful when prior information is available.
Ridge Regression: Ridge regression is used when there is multicollinearity in the model. It introduces a penalty term to reduce the effect of multicollinearity on the estimated coefficients.
"The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression."
"This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable."
"The relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data."
"Most commonly, the conditional mean of the response given the values of the explanatory variables (or predictors) is assumed to be an affine function of those values."
"Like all forms of regression analysis, linear regression focuses on the conditional probability distribution of the response given the values of the predictors."
"This is because models which depend linearly on their unknown parameters are easier to fit than models which are non-linearly related to their parameters and because the statistical properties of the resulting estimators are easier to determine."
"If the goal is error reduction in prediction or forecasting, linear regression can be used to fit a predictive model to an observed data set of values of the response and explanatory variables." "If the goal is to explain variation in the response variable that can be attributed to variation in the explanatory variables, linear regression analysis can be applied to quantify the strength of the relationship between the response and the explanatory variables."
"After developing such a model, if additional values of the explanatory variables are collected without an accompanying response value, the fitted model can be used to make a prediction of the response."
"To determine whether some explanatory variables may have no linear relationship with the response at all, or to identify which subsets of explanatory variables may contain redundant information about the response."
"Linear regression models are often fitted using the least squares approach."
"But they may also be fitted in other ways, such as by minimizing the 'lack of fit' in some other norm (as with least absolute deviations regression)." "...or by minimizing a penalized version of the least squares cost function as in ridge regression (L2-norm penalty) and lasso (L1-norm penalty)."
"Conversely, the least squares approach can be used to fit models that are not linear models."
"Thus, although the terms 'least squares' and 'linear model' are closely linked, they are not synonymous." Quote: "In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables)." Quote: "The relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data." Quote: "Most commonly, the conditional mean of the response given the values of the explanatory variables (or predictors) is assumed to be an affine function of those values." Quote: "Like all forms of regression analysis, linear regression focuses on the conditional probability distribution of the response given the values of the predictors." Quote: "This is because models which depend linearly on their unknown parameters are easier to fit than models which are non-linearly related to their parameters and because the statistical properties of the resulting estimators are easier to determine." Quote: "If the goal is error reduction in prediction or forecasting, linear regression can be used to fit a predictive model to an observed data set of values of the response and explanatory variables." Quote: "After developing such a model, if additional values of the explanatory variables are collected without an accompanying response value, the fitted model can be used to make a prediction of the response." Quote: "If the goal is to explain variation in the response variable that can be attributed to variation in the explanatory variables, linear regression analysis can be applied to quantify the strength of the relationship between the response and the explanatory variables." Quote: "Linear regression models are often fitted using the least squares approach." Quote: "But they may also be fitted in other ways, such as by minimizing the 'lack of fit' in some other norm (as with least absolute deviations regression)." Quote: "By minimizing a penalized version of the least squares cost function as in ridge regression (L2-norm penalty) and lasso (L1-norm penalty)." Quote: "Conversely, the least squares approach can be used to fit models that are not linear models." Quote: "Thus, although the terms 'least squares' and 'linear model' are closely linked, they are not synonymous."