Data engineering

Data engineering

1. Types of data

   • Structured
   • Semi-structured
   • Unstructured

2. Data operations

   • Data integration
   • Data transformation
   • Data consolidation

3. Some concepts

   • Data pipelines
     • Data pipelines are used to orchestrate activities that transfer and transform data. Pipelines are the primary way in which data engineers implement repeatable extract, transform, and load (ETL) solutions that can be triggered based on a schedule or in response to events.
   • Data lakes
     • The idea with a data lake is to store everything in its original, untransformed state. A data lake is a storage repository that holds large amounts of data in native, raw formats. Data lake stores are optimized for scaling to massive volumes of data. The data typically comes from multiple heterogeneous sources, and may be structured, semi-structured, or unstructured.
   • Data warehouses
     • A data warehouse is a centralized repository of integrated data from one or more disparate sources.  Data engineers are responsible for designing and implementing relational data warehouses, and managing regular data loads into tables.
   • Apache Spark
     • Apache Spark is a parallel processing framework that takes advantage of in-memory processing and a distributed file storage. It's a common open-source software (OSS) tool for big data scenarios. Data engineers need to be proficient with Spark, using notebooks and other code artifacts to process data in a data lake and prepare it for modeling and analysis.

4. Azure Data Lake Storage Gen2

   • Provides a scalable, secure, cloud-based solution for data lake storage. A data lake is a repository of data that is stored in its natural format, usually as blobs or files. Azure Data Lake Storage is a comprehensive, massively scalable, secure, and cost-effective data lake solution for high performance analytics built into Azure.
   • stages
     • Ingest
       • The ingestion phase identifies the technology and processes that are used to acquire the source data. This data can come from files, logs, and other types of unstructured data that must be put into the data lake. The technology that is used will vary depending on the frequency that the data is transferred. For example, for batch movement of data, pipelines in Azure Synapse Analytics or Azure Data Factory may be the most appropriate technology to use. For real-time ingestion of data, Apache Kafka for HDInsight or Stream Analytics may be an appropriate choice.
     • Store
     • Prep and train
       • The prep and train phase identifies the technologies that are used to perform data preparation and model training and scoring for machine learning solutions. Common technologies that are used in this phase are Azure Synapse Analytics, Azure Databricks, Azure HDInsight, and Azure Machine Learning.
     • Model and serve
       • Finally, the model and serve phase involves the technologies that will present the data to users. These technologies can include visualization tools such as Microsoft Power BI
   • Big data processing and analytics
     • Big data scenarios usually refer to analytical workloads that involve massive volumes of data in a variety of formats that needs to be processed at a fast velocity - the so-called "three v's". Azure Data Lake Storage Gen 2 provides a scalable and secure distributed data store on which big data services such as Azure Synapse Analytics, Azure Databricks, and Azure HDInsight can apply data processing frameworks such as Apache Spark, Hive, and Hadoop

Bilder und Dateien der Mindmap