Data Processing

Data Processing

1. Data processing is the conversion of raw data to meaningful information.

2. Batch processing: multiple data records are collected and stored before being processed together in a single operation.

   • HOW
     • Newly arriving data are collected and store to process the whole group as a batch. Each batch can be processed based on a schedule time interval, a trigger, or as the result of an event.
       • Generated data is captured in a stream source: a folder in a cloud data store, or a table in a database.
         • SOURCES
           • Azure Event Hub, Azure IoT Hub, Azure Data Lake Store Gen2, Apache Kafka.
       • Event data is processed by a perpetual query. Results are written to an output/sink for further processing.
         • SINKS
           • Azure Event Hub, Azure Data Lake Store Gen2, Azure blob storage, Azure SQL Database, Azure Synapse Analytics, Azure Databricks, Power BI.
   • ADVANTAGES
     • Processing large volumes of data at a time. Support for specific schedules/needs.
   • DISADVANTAGES
     • Time delay between ingestion and results visualization. Input data must be carefully checked before batch processing.
   • CHARACTERISTICS
     • DATA SCOPE: batch processing can process all the data in the dataset.
     • DATA SIZE: suitable for large dataset.
     • PERFORMANCE: latency is expected as a few hours.
     • ANALYSIS: to perform complex analytics.

3. Stream processing: a source of data where new data events occur is constantly monitored and processed in real-time.

   • HOW
     • Each new piece of data is processed as individual units in real-time.
   • ADVANTAGES
     • No need to wait for the next processing interval. Ideal for time-critical operations that require an inmediate response.
   • CHARACTERISTICS
     • DATA SCOPE: only access to the most recent data received or within a rolling time window.
     • DATA SIZE: intended for individual records or micro batches (groups of few records).
     • PERFORMANCE: occurs immediately or within seconds/milliseconds.
     • ANALYSIS: for simple response functions, aggregates, or calculations.

4. Combining Batch & Stream Processing

   • 1. Data from a stream source is captured.
     • 2. Data from other sources is ingested/stored for batch processing (often, a data lake).
       • 3. If real-time analytics isn't required, stream data is written into the data store for batch processing. If real-time analytics is required, a stream processing technology is used for analysis/visualization.
         • 5. Batch data is periodically processed, results are stored in an analytical data store (often, a data warehouse).
           • 6. Results of stream processing might also be stored in the analytical data store.
             • 7. Analytical/visualization tools are used for real-time and historical data.

Bilder und Dateien der Mindmap