Flashcards
Data Engineering Lecture 1: Introduction to Data Engineering
Cards in this deck (23)
- 1Data Engineering“Data engineers are the designers, builders and managers of the information or "big data" infrastructure. They develop the architecture that helps analyze and process data in the way the organization needs it. And they make sure those systems are performing smoothly”
- 2DE concernsData recording, cleaning, integration, representation, visualization, and interpretation
- 3DE skillsProgramming, distributed systems, data pipelines
- 4DS skillsMath/statistics, ML/AI, advanced analytics
- 5Overlapping skillsProgramming, analysis, big data
- 6Big data“It is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.”
- 7The three V’s- Volume
- Velocity
- Variety - 8VolumeEnterprises are awash with ever-growing data of all types, easily amassing terabytes - even petabytes - of information
- 9VelocitySometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise to maximize its value
- 10VarietyBig data is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more
- 11Data PipelinesA data pipeline aggregates, organizes, and moves data to a destination for storage, insights, and analysis. Modern data pipeline systems automate the ETL (extract, transform, load) process and include data ingestion, processing, filtering, transformation, and movement across any cloud architecture and add additional layers of resiliency against failure
- 12Stages in a big data pipeline
- 13Stages in a big data pipeline: open source technologies
- 14Continuous delivery for machine learning: MLOps
- 15Principles of MLOps:- Software engineering approach
- Cross-functional team
- Producing software based on code, data, and machine learning models
- Small and safe increments
- Reproducible and reliable software release
- Software release at any time
- Short adaptation cycles - 16Software engineering approachIt enables teams to efficiently produce high quality software.
- 17Cross-functional teamExperts with different skill sets and workflows across data engineering, data science, machine learning engineering, development, operations
- 18Producing software based on code, data, and machine learning modelsAll artifacts of the ML software production process require different tools and workflows that must be versioned and managed accordingly.
- 19Small and safe incrementsThe release of the software artifacts is divided into small increments, which allows visibility and control around the levels of variance of its outcomes, adding safety into the process.
- 20Reproducible and reliable software releaseWhile the model outputs can be non-deterministic and hard to reproduce, the process of releasing ML software into production is reliable and reproducible, leveraging automation as much as possible.
- 21Software release at any timeIt is important that the ML software could be delivered into production at any time, making the decision about when to release it a business decision rather than a technical one.
- 22Short adaptation cyclesShort cycles means development cycles are in the order of days or even hours, not weeks, months or even years.
- 23MLOps =Machine Learning (experiment, data acquisition, business understanding, initial modeling) + Develop (modeling, testing, continuous integration & deployment) + Operate (Continuous delivery data feedback loop, system & model monitoring)
Information
- Description
- Engineering Flashcards on Data Engineering Lecture 1: Introduction to Data Engineering, created by Tom van Eijk on 20/12/2021.
- Created by
- Tom van Eijk
- Views
- 4
- Rating