Fichas
Machine Learning
Fichas del conjunto (163)
- 1Supervised LearningUsing algorithms to build models that predict outcomes from observations on study units. Predicting numeric and categorical outcomes are known as regression and classification.
Examples: ElasticNet, Random Forest - 2Unsupervised LearningUsing algorithms to detect patterns in the observed data without a specified response variable to guide (supervise) the outcome.
Example: Clustering - 3VariablesTerms used for the dependent and independent variables in supervised learning.
Inputs denoted X = (X_1,...,X_p)
Numerical output denoted Y
Categorical output with k levels denoted G = (G_1,...,G_k) - 4Inputs/FeaturesPredictor variables
- 5OutputsResponse or outcome variable
- 6IndicesLet i = 1,...,N index study units and j = 1,...,p index input variables
- 7Prediction Functionf(x) denotes a numeric function that produces a predicted value from a set of inputs, x
ˆf (x ) denotes a prediction function that has been estimated from the data - 8What is the outcome from a regression prediction function for a continuous outcome?When predicting continuous outcomes we can take the result directly from the prediction function as with ^Y(x) = ^f(x)
- 9What is the outcome for the regression prediction function of an integer outcome?For an integer outcome you would have to round the output and would have something like ^Y(x) = round(^f(x))
- 10What is the outcome for prediction function in a classification setting (categorical outcome) ?In the case of classification, a decision rule must be pre-determined to decide on rules for how to map the numeric outcomes into categorical output levels (classes).
- 11How is a predictive model defined?A predictive model requires a prediction function and a decision rule to map the output of the prediction function to the output space
- 12Predictive PerformanceHow well a model does at predicting outcomes that were not included in the training data
- 13What are the goals of supervised learning?1. Identify an appropriate class of models to predict the outcome of interest
2. Estimate predictive performance
3. Develop a model to optimize predictive performance
4. Perform model diagnostics and assess the quality of predicted outcomes
5. Report numerical and graphical summaries of model performance
6. Interpret model predictor variables - 14Expected Prediction ErrorDefined as the expected value of a specified loss function measuring the difference between true outcomes and the predicted outcomes.
- In the deterministic case, this is the same as MSE - 15What do models maximize?Models maximize predictive performance which is a function of bias and variance (expected prediction error can be decomposed into bias and variance)
- 16How do bias and variance compare in simple vs complex models? What is the implication?- Simple models tend to have higher bias but less variance
- more complex models tend to have less bias but more variance
~ more complex models are not necessarily better and do not necessarily have better predictive performance - 17MSEMean Squared Error -- squared differences between the observed and predicted values
- 18RMSERoot Mean Squared Error -- square root of the mean squared error (MSE)
- computed on the same scale as predicted values
- smaller values indicate better performance - 19MAEMean Absolute Error -- robust version of RMSE (mean of the absolute value of the difference between observed and predicted values)
- 20Coefficient of DeterminationR^2: proportion of variability about the mean that is explained by the predicted values (1 - SSR/SSE)
-- For linear regression with an intercept, R^2 is between 0 and 1
-- In other cases it is possible to produce negative values - 21When should you use a logistic regression model?Logistic regression can be used to model probabilities for a categorical response variable.
- 22Describe Classification and what a classifier isClassification: uses a vector of prediction functions to classify outcomes into set categories. Ideally uses probability functions if available.
Classifier: also called categorical predictor takes the classification outcome with the largest probability assigns that category - 23Bayes Classifier- Predicts the probability for each category
- Assigns the category with the highest probability as the predicted outcome
- minimizes expected error with a 0-1 loss function (0 if true and predicted match, 1 if they don't) - 240-1 Loss Function- Assigns a value of 0 if the observed and predicted outcomes agree
- Assigns a value of 1 if the observed and predicted outcomes do not agree - 25How can the probability function be used in the case of binary response?- can be used for classification
- can be used for predicted probabilities - 26AccuracyPrecent agreement between observed and predicted outcomes
- taken alone is not an informative measure of predictive performance
- should compare to accuracy relative to the majority class - 27Cohen's KappaRelative agreement of observed and predicted values beyond what would be expected by chance alone
- ranges from -1 to 1
- <= 0 indicates no agreement
- kappa of 1 is perfect agreement - 28ROC AUC- also called 'concordance index'
- probability that a randomly selected subject with an event will have a greater predicted value than a subject without the event
- 0.5 is no concordance, 1 is perfect concordance
- only depends on ordering of predicted responses
- invariant to 1:1 transformations of the responses - 29Sensitivity- proportion of correctly predicted events among observed events (TP / (TP + FN))
- Also called 'true positive rate' or 'recall'
- 'false negative rate' (FNR) is 1 - sensitivity - 30SpecificityCorrectly predicted non-events among all observed non-events (TN / (TN + FP))
- also called 'true negative rate'
- 'false positive rate' (FPR) is 1 - specificity - 31Positive Predicted ValuePPV: Proportion of observed events among predicted events
- TP / (TP + FP) - 32Negative Predicted ValueNPV: Proportion of observed non-events among non-events
- FP / (FP + TN) - 33Confusion Matrixsummarizes cross classification of observed and predicted responses
- 34Name two proper score functions and state why they are 'proper'1. Brier Score
2. Cross Entropy
- proper because represent expected loss with respect to a probability distribution - 35Brier Score- MSE applied to predicted probabilities
- may be inadequate to distinguish between rare categories
- proper score function
- conventional to only include one of the two dichotomous levels in the calculation - 36Name two improper score functions and state why they are improper1. Accuracy
2. Cohen's Kappa
- improper because they are calculated from predicted class membership rather than class probabilities - 37Cross Entropy- proportional to negative log likelihood for multinomial distribution
- minimizing cross entropy is equivalent to maximizing multinomial likelihood
- proper score function - 38ROC IndexSpecifies a tradeoff between sensitivity and specificity
- ROC index function allows specification of a relationship between sensitivity and specificity that reflects their relative importance (relative. imp. depends on specific application) - 39Specify the Axes of an ROC curveY-axis: sensitivity (TPR)
X-axis: 1-specificity (FPR) - 40True Positive RateTPR: synonymous with sensitivity
- 41False Positive RateFPR: 1 - specificity
- 42Specify the axes of a Precision Recall (PR) Curve -When are PR curves used?Y-axis: Precision (PPV)
X-axis: Recall (sensitivity)
- typically used for predicting rare outcomes - 43Survival FunctionS(t) = Pr[T>t] is the probability of surviving beyond time t
- since S(t) is a probability, it takes values in the interval [0, 1] - 44CensoringCases when an event is not observed are censored
- withdrew from study
- lost to follow-up
- no longer at risk for the event
- study terminated before their event was observed - 45Why are survival methods used?If cases are followed over time in order to observe occurrence of an event
- 46What is Cox regression and what is the form of the Cox model?Cox Regression is for time to event survival data using the cox proportional hazards model
λ(T ,X ) = λ0(T ) exp{β1X1 + ... + βp Xp }
-> λ(T ,X ) is the hazard rate as a function of time and predictors, X
--> λ0(T ) is the baseline hazard rate common to all cases
- hazard rate is instantaneous rate at a point in time - 47How can we interpret coefficients in a Cox model?- naturally interpretable as hazard ratios
- the exponential of the coefficient for predictor j is interpreted as relative increase in the event rate for one unit increase in xj
- hazard ratios are constant over time hence proportional hazards model - 48Name one advantage and one disadvantage of the Cox modelAdvantage: survival estimates do not rely on distributional assumptions for the survival times
Disadvantage: estimation is separate for the coefficients and for the survival function itself - 49What is the Concordance Index?- also called c-index
- number of case pairs with concordant observed and predicted survival times divided by the number of concordant and discordant pairs
- 0 is perfect discordance
- 1 is perfect concordance
- 0.5 is chance concordance
--> (ˆti > ˆtj ) = I (x >i ˆβ< x >j ˆβ) for cox models so just use linear predictor instead of survival times - 50What is a loss function and what are examples for continuous and binary outcome loss functions?- A loss function finds the average distance between some predictors and some prediction function
Binary: 0-1 loss (0 if truth and predictive match, 1 otherwise)
Continuous: squared error loss
- can also use performance metrics as loss function - 51Bias-Variance TradeoffExpected squared error loss at a new point, x0 is:
Err(x0) = irreducible Error + Bias^2 + Variance.
- Irreducible error is the variance of the response variable about its true mean
- Bias^2 is the squared difference between the mean of the estimator and the true mean
- variance is the variance of the estimator
--> balance var and bias by minimizing prediction error
- expected prediction error proportional to
prediction bias2 (accuracy) + variance (precision) for squared-error
loss - 52In-sample prediction errorAverage expected loss of predicting new response values at the predictor variable values observed in the training set
- ex: training error
- overly optimistic since the prediction function performs best on the data used to train the model - 53What is Optimism and what practical considerations are there?Optimism: difference between in sample error and training error
- can actually only estimate in sample optimism
-- predictor variables are considered fixed, expectation taken over observed y used in deriving the prediction function - 54What is the relationship between training error and in sample error?Expected in sample error is equal to expected training error plus expected optimism:
E[err_in] = E[err_train] + E[opt] - 55Out of Sample Prediction ErrorExpected prediction error on independent test samples given the specific training set used to develop the prediction function
-- also called 'generalization error' or 'test error' - 56Expected Prediction ErrorJust like out of sample prediction error but instead of conditioning on a single training set, averages over many training sets
- 57K-Fold Cross Validation1. partition data into K equal sized parts (roughly equal)
2. cases in the kth partition (test set) are predicted from a prediction function fit to the other k-1 partitions (training set)
3. repeat so each of the k partitions serve as the test set
4. compute the sample avg of the values in loss function evaluated at the observed and predicted values to estimate expected prediction error
--gives predictions for each of the cases in the dataset
--can use all of the available data - 58What do you consider when choosing 'k' for k-fold cross validation- leave one out k-fold yields approx. unbiased estimates of EPE
- computational expense and variance of estimated error increase with k (more partitions = higher expense and higher variance)
- bias toward overestimating prediction error increases as k decreases
- common choices for k are between 5 and 10 - 59What is repeated cross-validation and why is it used?- averages over the k-fold cross validation algorithm applied to multiple different partitions of the data
- reduces the variance of EPE - 60What is bootstrap resampling used for?bootstrap resampling is used to estimate the sampling distribution of a statistic computed on a dataset
- 61Describe Bootstrap Resampling1. draw B datasets with replacement from the original dataset
2. compute statistic of interest on each of the - 62How is bootstrapping for error estimation?Tends to severely underestimate error because some of the cases being predicted are held within the bootstrap sample
-- Correction for this exists from Efron and Gong to correct for the optimism - 63What is the 'split sample method' and why would you use it?- Split the data into training, testing, and validation sets
-- Training set: fits candidate models
-- Validation set: estimation of candidate models predictive performance for the purpose of selecting the final one (only needed if there are multiple candidate models)
-- Testing set: estimation of the final models predictive performance
- Useful for selecting models and estimating predictive performance - 64What are some tradeoffs of 'split sample' method versus resampling methods?Split sample:
- inefficient use of data
- good accuracy
- low computational burden
- bad precision
Resampling:
- eliminates need for one or both of training and validation sets
- more computationally expensive
- more bias in performance estimates
- accuracy and precision increase with increased sample size but computational expense goes up - 65What is variable importance?Variable importance is a relative measure of the contributions of model predictors
- 66What is the difference between model specific and model agnostic variable importance?Model specific: based on the computational routine used to fit a particular model (thus, varies depending on the model)
-- commonly computed from log transformed pvalues
-- not available for all models
Model agnostic: general algorithm that can be applied to any model used to estimate variable importance
-- common choice is permutation variable importance - 67What is the general algorithm for permutation based variable importance?1. fit a model to a training dataset
2. compute training performance
2. sample a subset of the training set without replacement (optional)
3. randomly permute the variable values
4. compute performance on the permutation set
5. compute importance as the difference or ratio between training and permutation set
6. reset variable to its original values
7. return summary stat (ie mean) to display variable importance for each variable in the model - 68What are some advantages and disadvantages of permutation variable importance?Advantages:
- available for all models
- can be computed for any performance metric defined for the given response variable type
- can be computed for any predictor variable in the original dataset
Disadvantages:
- computationally expensive
- computation of performance of a ratio can be problematic for variables that can take a value of 0 - 69Why does permutation work for variable importance?It reduces or eliminates the association with other variables thereby mimicking removal from the dataset
- 70What are some things to consider before choosing model specific variable importance?- cannot be computed for every model
- limited to metrics computed in their source packages
- computed on derived rather than original predictor variables - 71What does negative variable importance mean and when can this occur?Negative variable importance can occur when using permutation based methods and it indicates that the variables are detrimental to model performance.
- 72What does a partial dependence plot show?It shows the marginal effects of predictors on a response variable
- 73Broadly, how is dependence typically computed?Typically computed for each predictor over a range of values and averaged over the dataset on which the prediction function was fit.
- 74Can we get partial dependence for multiple variables? What would this tell us?Yes, we can extend partial dependence to multiple predictors to get a sense of interaction effects and we could also use summary statistics other than the mean.
- 75Are partial dependence plots robust to the specific dataset/population?No. This is because partial dependence is marginal since it is averaged over the remaining variables whose distribution is dependent on the population represented in the dataset
- 76What is a calibration curve?A calibration curve is a visual representation of the agreement between model-predicted and observed values
- 77What do calibration curves tell us that other performance metrics can't?Calibration curves act as a performance metric supplement by providing information as to model fit for different regions of predicted values.
-- Additionally, they serve as a more direct assessment of model fit that metrics like ROC AUC that do not account for scale and location differences - 78What are the general steps to constructing binned calibration curves? (continuous response)1. partition cases into equal width intervals based on their resampled predicted responses
2. calculate mean observed response in each bin
3. plot mean observed response on y-axis and bin midpoint on the x-axis - 79What are the general steps to constructing binned calibration curves? (binary response)1. partition modeled predicted responses into a number of bins
2. compute observed event proportions in each bin
3. plot resulting proportions against bin midpoints
-- could use smoothing instead of discrete bins (gives more precise curves)
-- well calibrated models have a curve close to the 45 degree line - 80Decision Boundary- can be linear (ie linear regression) or nonlinear (ie k-nearest neighbors)
- set of predictors corresponding to designated cutoff values from a classification problem - 81Briefly describe the logistic regression classifierLogistic regression classifier returns the index to the class with the highest predicted probability
- used most often when it is important to understand the role of predictor variables in influencing the outcome (interpretability)
- like linear regression classifier, can accommodate numeric and categorical response variables - 82Name two classification methods with no tuning parameters1. linear regression
2. logistic regression - 83What are the assumptions of linear discriminant analysis (LDA)?1. multivariate normal distribution for predictors
2. homogeneity of normal covariance matrix across all levels of the response - 84For what can Linear discriminant analysis be used?- classification problems only
- can still perform well if multivariate normal and homogeneity of var-cox matrix is violated (fairly robust) - 85Define shrinkage methods and give a reason for using them- shrinkage methods estimate regression coefficients by subjecting them to constraints
- reduce prediction variance for a small increase in bias
- results in better predictive performance - 86How does ridge regression work?- minimizes the penalizes residual sums of squares
- penalty term is the L2 norm of the predictor coefficient vector
- beta coefficients depend on the scale of each predictor
-- predictors should be standardized first
- shrinks coefficients of highly correlated coefficients toward zero ultimately averages the correlated terms together - 87How does LASSO work?Least Absolute Shrinkage and Selection Operator:
- minimizes penalized residual sum of squares (as with ridge)
- penalty is L1 norm of the predictor coefficient vector
- should standardize predictors first
- coefficients can be shrunk exactly to zero
-- performs shrinkage and variable selection
- allows for parameter estimation when p > N - 88How does Elastic Net work?- Averages the ridge and LASSO penalties
- when alpha = 0, ENet = ridge
- when alpha = 1, ENet = LASSO
- generally results in more selected predictors than LASSO but with more shrinkage of the parameters - 89Can shrinkage methods be optimized? If so, how? If not, why not?Yes, resampling methods can be used to determine optimal values for alpha and lambda
- optimal in the sense the predictive performance is maximized
- alpha and lambda would be considered hyper parameters (aka tuning parameters) because we are tuning them but not directly interested in them - 90What does CART mean?CART: Classification And Regression Trees
- a tree-based regression method - 91How do tree based methods work broadly speaking?- partition the predictor space into a set of rectangles
- fit a simple model (such as a constant) within each rectangle - 92What type of response variables can a regression tree be used for?Regression trees are relevant for NUMERIC responses
- 93What is a big challenge for tree methods?The challenge is in the partitioning of the predictor space into a useful set of rectangles
- 94Define 'Impurity Measure'.The objective function is termed 'impurity measure (so the function that is being optimized)
- 95Cost Complexity CriterionThis is the function that helps us determine how complex of a tree to make -- it is used when pruning a tree to get a meaningful subtree
- 96What type of decision rules come from CART?CART yields binary decision rules that are easy to interpret visually and appreciated by clinicians
- 97What are some drawbacks of CART?1. categorical predictors with many levels tend to be favored so at risk of severe overfitting in the case of many levels
2. Branching into more than two nodes is possible but not advisable because it can split the data too fast
3. unlikely to capture additive effects
4. High variability is a big issue because errors propagate down to all nodes below the node with the error - 98What is meant by the term 'neural network'?- nonlinear hierarchical regression models
- single hidden layer
- m hidden unit (not directly observable)
- activation function (basis expansion of linear combination of the predictors)
- final transformation linking hidden units to the responses
- weights which are unknow model parameters (often way too many of these leading to overfitting) - 99What does model fitting for neural networks do?- finds network weights that minimize the specified error function (often squared error or deviance)
- back-propagation to estimate weights
- neural nets often overparameterized yielding unstable estimates
- parameters should be scaled to ensure equal treatment in the penalization (similar to ridge) - 100When are neural networks most effective?They are most effective in high signal to noise scenarios and when prediction without parameter interpretation is the goal
- limited relevance for medical fields where model interpretation is crucial - 101How can we think about Support Vector Machines?SVMs are a generalization of linear decision boundaries to non-linear decision boundaries
- construct optimal hyperplanes (linear) boundaries on transformed versions of the predictor space - 102What does estimation in SVM depend on?For SVM, estimation depends only on the kernel function and requires no knowledge of the transformation
- 103Name three popular kernel functions for SVM1. Linear
2. Polynomial
3. Radial Basis - 104How does the cost parameter, C, effect the hyperplane boundary in SVM?- larger C leads to overfit boundary that is very wiggly in the original predictor space
- smaller C results in a smoother boundary - 105What are the tuning parameters for SVM?1. Cost parameter
2. Kernel function - 106What is the purpose of ensemble methods?To build a prediction model by combining the strengths of a collection of base learning models
- 107What are two main tasks of ensemble learning?1, Develop a population of base learners from the training data
2. Combine the base learners to form a single composite prediction model - 108How does composite prediction typically work in 1. the numeric prediction setting 2. the categorical prediction setting1. Averages over the base learners
2. employs majority voting or averages over class probabilities if those are available - 109How does AdaBoost work?- Sequentially apply a classification algorithm to repeatedly modified versions of the data and then take a majority vote of the resulting base learners to produce a composite classifier
- weights begin equal for all observations
- subsequent steps give more weight to misclassified observations so observations that are more difficult to classify get increasingly more weight - 110What are the tuning parameters for AdaBoost?- same as for decision trees
- mfinal: number of boosting iterations (M) so number of total trees fit
- maxdepth: maximum depth of any node in each one of the trees - 111What type of model can we view boosting as?Boosting can be viewed as an additive model
- basis expansion of the coefficients where classifiers are the basis functions - 112How are additive models typically fit?Additive models are typically fit by minimizing a loss function over training data
- 113What is gradient boosting?Gradient boosting is an approximation to the minimization of a loss function
- exact minimization can be very computationally burdensome so gradietn boosting reduces computational time
- gradient boosting involves a combination of forward stagewise additive modeling and gradient descent numerical optimization - 114How is the tree size affected by gradient boosting?Gradient boosting requires all trees to have the same size
- tree size can be treated as a tuning parameter - 115How can you interpret a tree split that leads to a terminal node?Tree splits leading to a terminal node represent interaction effects between the variables involved
- thus, can tune tree size either by specifying maximum interaction order or tree size
-- max interaction order is equal to three depth minus 1 - 116How does interaction order in a gradient boosted tree affect variance?Higher interaction order tends to lead to increased variance and test set error
- 117How can training loss be decreased for boosted models? How small can training loss get?Training loss can be made arbitrarily small by increasing the number of boosting iterations
- 118What is a drawback of fitting training data too well (decreasing training loss)?Fitting the training data too well can lead t overfitting which degrades the predictive performance.
- 119Is gradient boosting used for classification or regression?Gradient boosting can be used for both classification and regression
- 120What are the learners fit to in gradient boosting?In gradient boosting a series of learners are fit to pseudo residuals
- 121How are pseudo residuals calculated?Pseudo residuals are fit in terms of a likelihood function specified for the response variable
- likelihood function could be
1. Gaussian or Laplace (numeric)
2. Bernoulli/Multinomial (categorical) - 122What are the tuning parameters for gradient boosted trees?1. tree depth
2. boosting iterations
2. shrinkage (learning rate)
3. proportion of cases to sample for each tree - 123A modification of bagging resulted in which method?Random Forest
- 124How do random forests work? What is the goal?Random forests work by building and averaging a large collection of de-correlated trees
- goal: reduce the prediction variance while accounting for the correlation between trees - 125What is the key step for Random Forests to work?Key for random forests is the randomly select m<=p candidate predictors at each split of the tree
- decreasing m reduces the correlation but may increase bias - 126Under what circumstances can we expect random forests to perform poorly?If the number of predictors is large but the fraction of relevant predictors is low, the random forests will perform badly if m is small
- 127What are the tuning parameters for Random Forests?1. number of trees
2. number of predictors to sample for each branch (m)
3. minimal terminal node size - 128What is grid search optimization?- seeks hyperparameter values that optimize performance over a parameter space of discrete points
-- may be over all points or a random selection of points in the grid
-- can contain hyperparameters of any type (numeric and non-numeric) - 129What is sequential optimization?- optimizes performance over a space of hyperparameters
- typically a closed space like a hypercube
-iterative algorithms are employed to generate new hyperparameter values based on current ones - 130What is Bayesian optimization?Bayesian optimization is one of the most popular forms of sequential optimization
- 131Describe the algorithm of Bayesian optimization1. fit the Gaussian process model to an initial set of hyperparameters
2. For every set of hyperparameter points, select a new set that maximizes the acquisition function and then evaluate your chosen performance metric
3. Return the set of hyperparameters associated with the best performance - 132What is model selection?The process of identifying an optimal model from a candidate set of potential models
- for predictive models, optimality is defined by predictive performance - 133Compare model selection with split sample vs. nested resampling.1. split sample method could always be used whereas nested resampling assumes the selection process can be embedded in the resampling algorithms
2. split sample splits the data so it cannot all be used for training whereas nested resampling is a more efficient use of data
3. - 134What is feature selection?Feature selection is the process of identifying an optimal set of predictor variables to use in model construction
- 135Name three considerations in feature selection.1. testing a specific hypothesis
2. understanding relationships between predictor and response variables
3. maximizing predictive performance - 136What is one practical and one statistical consideration of using fewer variables in a model?practical: makes the model easier to interpret and less cost for making future predictions
statistical: eliminating non-informative predictors reduces uncertainty in predicted values and increases overall model effectiveness - 137What are the three main groups of feature selection methods?1. Filter
2. Wrapper
3. Embedded - 138What is 'Filter Method'?Filter is a type of feature selection in which
1. relevance of all predictor is evaluated apart from the model
2. build a model only with those that pass the initial threshold (ie univariable regression screening) - 139What are pros and cons of the filter method?Pros: reduce overfitting and compute time
Cons: neglects joint effects of predictors not directly related to the effectiveness of the full model - 140What are 'wrapper methods'?Wrapper methods evaluate model performance over different variable subsets and select an optimal one
(think: stepwise selection) - 141What are pros and cons of wrapper methods?Pros: consider joint effects of the predictors
Cons: tend to increase risk of overfitting and costly compute time - 142What is 'Embedded selection'?Embedded selection is when variable selection is performed as a part of model fitting algorithms
(think: tree-based methods and LASSO) - 143What are pros and cons of embedded selection?Pro: combines the advantages of filters and wrappers
Con: May end up exploring a limited set of models - 144What is univariate filtering?Univariate filtering is a filter methods in which univariable stat methods select variables for future modeling steps
- 145What is recursive feature elimination?Recursive Feature Elimination (RFE) is backward variable selection that begins with a full model of all predictors the eliminates those with relatively low importance
- 146What are the tuning paramters for recursive feature elimination:1. subset size
2. recalculation of variable importance - 147Name one method the is insensitive to predictor encoding and one method that is sensitive.Insensitive: tree methods
sensitive: linear regression - 148What is a synonym of 'predictor encoding'?Feature engineering
- 149What does optimal feature engineering depend on?1. Model type
2. true relationship between the predictor and outcome - 150What can you do if you have categorical variables but need quantitative codings for model fitting?Use Dummy variables where each category is represented by a 0/1 indicator
- also called 'one-hot encoding' - 151Why might you remove variables before model fitting?1. fewer predictors results in less computational time and more interpretable models
2. degeneracy might exist (near zero variance)
3. multicollinearity might exist - 152How can pre-processing help with multicollinearity?If you have multicollinearity you can use a high correlation filter
- removes redundant predictors
- remove the minimum number of predictors necessary so pairwise correlations are below a specified threshold - 153What can you do if you have missing data?1. remove subjects or predictors with missingness
2. use models like tree-based methods that account for missing data within their model fitting algorithms
3. impute missing values - 154Mean/Mode imputationMean/mode imputation replaces the missing value with with the mean or mode of the observed value
- preserves the observed sample mean/mode
- easy to understand and apply
- computationally speedy
- assumes MCAR (big assumption)
- attenuates associations
- not recommended for multivariate analyses - 155Bagged tree imputationReplaces missing values in a numeric or nominal variable with predictions from a bagged tree fit with the remaining variables as predictors
- aggregation over decision trees
- reduce prediction variance
- good in multivariate settings
-assumes MAR (big assumption)
- computationally slow
- requires more tuning parameters - 156K-nearest neighbors imputationReplaces missing values in a numeric or nominal variable with the mean or mode respectively of the K closest neighbors
- closeness is relative to the remaining variables and can be computed on mixtures of numeric and nominal vars
- good in multivariate settings
- robust to choice of K
- computationally fast
- assumes MAR (big assumption)
- requires storage and application of training data - 157What are some methods of transforming individual predictors?1. centering and scaling (needed for any technique with L1 or L2 penalty)
2. skweness
3. basis expansion
4. univariate transformation - 158Why might you choose to center and scale your predictors?Improves numerical stability of some calculations and performance of some models
- 159Why might you choose to transform on skewness?May need to reduce skewness to deemphasize the effects of observations in the tails of their distributions
- 160Why might you transform with basis expansions?May allow for more flexible modeling of predictor effects
- 161Why might you perform univariate transformations?May be applied based on subject-matter knowledge or modeling strategies
- 162How might you go about reducing dimensionality?You could use a method like principal components analysis (PCA) to generate a smaller set of predictors that still capture much of the information
- 163What is 'binning predictors'?Turns continuous predictors into categorical predictors
- aids in interpretability
- lose information
- degrades model performance
- lose precision in predictions
- gives high false positive rate
Información
- Descripción
- Fichas sobre Machine Learning, creado por Annika Helverson el 07/05/2022.
- Creado por
- Annika Helverson
- Visitas
- 2
- Valoración