Flashcards
Data Science/Engineering Review
Cards in this deck (17)
- 1What is Feature Engineering?Creating/Adding, Dropping, or Changing a Feature/Column
- 2What is Imputation?Filling in missing data with values
- 3How do you handle outliers?Outlier detection with standard deviation, or dropping outliers, or capping outliers
- 4What is Dimensionality Reduction?Reducing the number of features/columns
Advantages: Model trains faster, improves accuracy, less storage space - 5What's the difference between the Training, Validation, and the Test Set?Training set = data where the model SEES and LEARNS from the data
VALIDATION set = Provides an unbiased evaluation of a model
TEST set = the data set used for PERFORMANCE evaluation - 6What is Normalization?Changing the value of the numeric columns to a COMMON scale WITHOUT distorting differences in the range of values
- 7What is the difference between relational databases and Hadoop Distributed File System (HDFS)?Relational Database RELY on a Schema; great for reading single-line records
HDFS can have structured or unstructured data; great for writing huge blocks of data - 8Difference between structured and unstructured data?The format and organization; structured data has a format
- 9What factors do you consider when you partition your data?How the data is READ
How often the data is WRITTEN - 10Can you explain what partitioning is?Partitioning is how your data is distributed so it can be easily access or maintained
- 11What's the difference between an ACTION and TRANSFORMATION in spark?ACTION = involves LOADING data into memory and performing operations
TRANSFORMATIONS = only creates REFERENCES; does not load data into memory - 12What does "Laziness" refer to in spark?Variables are not initialized until they are used.
ex. a transformation is considered "lazy" because data is not loaded into memory - 13What are some differences between a PARQUET and a CSV file?Parquet is efficiently compressed
Parquets use columnar storage vs. rowbased
- optimized for fast retrieval of columns (drastrically reduces the amount of data you need to load from disk - 14What is the normal distribution?The BELL CURVE
Data is distributed around a central value without any bias - 15What is the BIAS, VARIANCE trade-off?Bias is the ERROR due to over simplification in the model which can lead to underfitting
VARIANCE is the ERROR due to complexity of the model which can lead to Overfitting
the TRADE-OFF is an inverse relationship
- Increasing Bias, Decreases Variance
- this changes based on parameters - 16What is selection bias?Selection bias results from the method of collecting samples where RANDOMIZATION is not achieved properly
- 17What is REGULARIZATION?The process of adding a tuning parameter to a model to prevent overfitting
the model prediction should minimize the loss function
Information
- Description
- Flashcards on Data Science/Engineering Review, created by Reksmey Mouk on 11/06/2019.
- Created by
- Reksmey Mouk
- Views
- 0
- Rating