"In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable... and one or more independent variables..."
A process of studying the relationship between a dependent variable and one or more independent variables.
Probability theory: A branch of mathematics that deals with the study of random events or processes and their properties.
Random variables: A variable whose possible values are based on a random process.
Probability distributions: A function that describes the likelihood of an event in the sample space.
Normal distribution: A continuous probability distribution that is often used to represent real-world phenomena.
Central Limit Theorem: A statistical theory that states that the distribution of the sum or average of independent random variables will tend towards a normal distribution.
Hypothesis testing: A statistical inference technique that helps in making conclusions about the population based on a sample.
Linear regression: A statistical technique used to determine the relationship between two variables in a linear function.
Multiple regression: A statistical technique used when there is more than one independent variable to predict the value of the dependent variable.
Nonlinear regression: A statistical technique used when the relationship between the independent and dependent variables is not linear.
Correlation: A statistical method that measures the strength and direction of the relationship between two variables.
Coefficient of determination: A statistical measure that indicates the proportion of the variance in the dependent variable that is explained by the independent variable.
Confidence interval: A range of values that is computed from a sample of data and provides a range of values where the true population value is likely to be found.
Categorical variables: A type of variable where the observations are divided into discrete categories.
Binary variables: A type of categorical variable where the observations can only be two categories.
Dummy variables: A mathematical tool used to include categorical variables in regression analyses.
Interaction terms: A way to introduce nonlinearity and interdependence between variables in regression models.
Model selection: The process of selecting the most appropriate regression model for a given dataset.
Variable selection: The process of selecting the most important independent variables that have the strongest relationship with the dependent variable.
Multicollinearity: A problem that occurs when independent variables in a regression model are highly correlated with each other.
Outliers: Observations that are significantly different from other observations in the sample that may affect the regression model's accuracy.
Simple Linear Regression: A statistical method that is used to study the relationship between two continuous variables. It assumes that there is a linear relationship between the predictor variable and the response variable.
Multiple Linear Regression: A statistical method that is used to study the relationship between two or more independent variables and a dependent variable. It assumes that there is a linear relationship between the predictor variables and the response variable.
Polynomial Regression: A statistical method that is used to study the relationship between an independent variable and a dependent variable where the relationship is not linear. It assumes that there is a polynomial relationship between the predictor variable and the response variable.
Logistic Regression: A statistical method that is used to study the relationship between a binary dependent variable and one or more independent variables. It assumes that there is a logistic relationship between the predictor variables and the binary response variable.
Poisson Regression: A statistical method that is used to study the relationship between a count dependent variable and one or more independent variables. It assumes that there is a Poisson distribution of the dependent variable.
Ridge Regression: A statistical method that is used to study the relationship between two or more independent variables and a dependent variable where the predictor variables are highly correlated. It is used to prevent over-fitting in the model.
Lasso Regression: A statistical method that is used to study the relationship between two or more independent variables and a dependent variable where the predictor variables are highly correlated. It is used to reduce the number of predictor variables in the model.
Elastic Net Regression: A statistical method that is used to study the relationship between two or more independent variables and a dependent variable where the predictor variables are highly correlated. It is a combination of Ridge Regression and Lasso Regression.
Quantile Regression: A statistical method that is used to study the relationship between a dependent variable and one or more independent variables at specific quantiles of the dependent variable. It is used when the relationship between the predictor variables and the response variable is non-linear.
Bayesian Regression: A statistical method that is used to study the relationship between two or more independent variables and a dependent variable using Bayes theorem. It is used to estimate the posterior probability distribution of the model parameters.
"Regression analysis is primarily used for two conceptually distinct purposes. First, regression analysis is widely used for prediction and forecasting... Second, in some situations regression analysis can be used to infer causal relationships between the independent and dependent variables."
"The most common form of regression analysis is linear regression..."
"For example, the method of ordinary least squares computes the unique line (or hyperplane) that minimizes the sum of squared differences between the true data and that line (or hyperplane)."
"... this allows the researcher to estimate the conditional expectation (or population average value) of the dependent variable when the independent variables take on a given set of values."
"Less common forms of regression use slightly different procedures to estimate alternative location parameters (e.g., quantile regression or Necessary Condition Analysis) or estimate the conditional expectation across a broader collection of non-linear models (e.g., nonparametric regression)."
"...where its use has substantial overlap with the field of machine learning."
"Regressions by themselves only reveal relationships between a dependent variable and a collection of independent variables in a fixed dataset."
"To use regressions for prediction... a researcher must carefully justify why existing relationships have predictive power for a new context."
"The latter is especially important when researchers hope to estimate causal relationships using observational data."
"...often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance."
"...often called 'predictors', 'covariates', 'explanatory variables' or 'features'."
"...computes the unique line (or hyperplane) that minimizes the sum of squared differences between the true data and that line (or hyperplane)."
"Less common forms of regression use slightly different procedures to estimate alternative location parameters (e.g., quantile regression or Necessary Condition Analysis)..."
"Yes, nonparametric regression can be used to estimate the conditional expectation across a broader collection of non-linear models."
"A researcher must carefully justify why existing relationships have predictive power for a new context."
"The latter is especially important when researchers hope to estimate causal relationships using observational data."
"...this allows the researcher to estimate the conditional expectation (or population average value) of the dependent variable..."
"... regression analysis is widely used for prediction and forecasting..."
"For example, the method of ordinary least squares computes the unique line (or hyperplane) that minimizes the sum of squared differences between the true data and that line (or hyperplane)."