Flashcards
Data Engineering Lecture 3: Containerization for Big Data
Cards in this deck (17)
- 1ContainersFuture of Big Data
- 2Infrastructure▪ Agility and elasticity
▪ Standardized environments (dev, test, prod)
▪ Portability (on-premises and cloud)
▪ Higher resource utilization - 3Applications▪ Fool-proof packaging (configs, libraries, driver versions, etc.)
▪ Repeatable builds and orchestration
▪ Faster app dev cycles - 4Kubernetes (K8s)- Open source “platform” for container orchestration
- Platform building blocks vs. turnkey platform (=pay per minute/second for a ready platform)
- Stateless = “unless you
ask about the data, you will not know what data I have”. If Kafka was shared across sources (array of cellphones), internal stateful. If I’m a TensorFlow operation outside, Kafka is externally stateless.
- Top use case is stateless / microservices deployments
- Evolving for stateful applications
- Kubernetes will orchestrate for example a swarm of containers + TensorFlow object
- It will maintain the state of your application over time if you have Kubernetes running - 5Kubernetes (K8s) framework
- 6Spark on Kubernetes- Spark creates a Spark driver running within a Kubernetes pod. A pod is a sequence of Kubernetes containers. It is a clustering of similar containers, part of the same middleware.
- Spark = middleware, Spark driver = listener, eats your program and makes it happen in parallel, Spark job = what you submit to Spark for a form of data intensive computing.
- The driver creates executors pods dynamically and executes application code
- When the application completes, executor pods are terminated (ala serverless type behavior)
- Driver pod remains in “completed” state in the Kubernetes API (doesn’t use any compute/memory) – waits for next job - 7Spark on Kubernetes framework
- 8Kuberneters applicationApplication specific operations (written for k8s). Allow you to have an application specific operator; Kubernetes requires an operator for whatever application. These operators are basically an interpreter. Middle layer that allows them to understand each other. YAML files tell Kubernetes how to bring up an infrastructure
- 9K8s cluster
- 10KubeDirector (bluek8s/kubedirector)- KubeDirector is a custom controller. It is an optional component. Allows you to plug resources into your Kubernetes cluster (e.g., cell phone data) without having to build specific operators for these.
- Eliminates the need to build app-specific operator
- Watches for custom resources to appear / change
- Creates / modifies standard Kubernetes resources (StatefulSets, etc.) in response to custom resources
- Reconciliation - 11KubeDirector model
- 12Desired statenext generation hybrid / multi-cloud. Build containers for all your applications (e.g., python, pandas, R, etc.) and connect that to cloud Kubernetes storage
- 13Desired state model
- 14Kubernetes Takeaways- Large-scale distributed data analytics and machine-learning need specialized data-intensive architectures.
- Requires specific data-intensive design principles (e.g., isolated environments, elasticity, data management).
- Containers and Kubernetes are essential for a new cloud-ready, agile architecture.
- Solution must deploy ‘existing’ (non-cloud native) apps and ‘newly’ minted K8s apps, - 15Security considerations- Security is essential since containers and host share one kernel
▪ Non-privileged containers
- Achieved through layered set of capabilities
- Different capabilities provide different levels of isolation and protection
- Add “capabilities” to a container based on what operations are permitted - 16Docker takeawaysAll DIAs must be “Dockerized
- Traditional bare-metal approach to Big Data is rigid and inflexible.
- Containers (e.g., Docker) provide a more flexible & agile model.
- Faster app dev cycles for Big Data app developers, data scientists, & engineers.
- Performance degradation is “gracious” – i.e., it still allows for data-intensive computing at rapid paces. - 17There are unique Big Data pitfalls & challenges with Containers (Docker)- For enterprise deployments, you will need to overcome these and more:
▪ Docker base images include Big Data libraries and jar files.
▪ Container orchestration, including networking and storage.
▪ Resource-aware runtime environment (e.g., including focus on CPU, RAM, resident mem., …).
- More pitfalls:
▪ Access to Container secured with ssh keypair or PAM module (LDAP/AD);
▪ Fast access to external storage.
▪ Management agents in Docker images.
▪ Runtime injection of resource and configuration information.
Information
- Description
- Engineering Flashcards on Data Engineering Lecture 3: Containerization for Big Data, created by Tom van Eijk on 20/12/2021.
- Created by
- Tom van Eijk · Dec 2021
- Updated
- Dec 20, 2021
- Views
- 2
- Rating