Flashcards
Data Engineering Lecture 2: Virtualization and Cloud computing
Cards in this deck (59)
- 1Cloud computingIs a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
- 2Utility ComputingPay-as-you-go manner to the public. The service being sold is Utility Computing
- 3CloudA Cloud is a type of distributed system consisting of interconnected and virtualized computers dynamically provisioned and presented as one (or more) unified computing resource(s) based on service-level agreements established through negotiation between service providers and consumers.
- 4Central ideas Cloud Computing- Utility Computing over data
- SOA – Service-Oriented (Data) Architectures
- SLA - Service Level Agreements - 5Properties and characteristics- High scalability and elasticity
- High availability and reliability
- High manageability and interoperability
- High accessibility and portability
- High performance and optimization - 6High scalability and elasticityDynamic provision, multi-tenant design. Elasticity means that the scalability of infrastructure goes up & down based on requirement.
- 7High availability and reliabilityFault tolerance, system resilience & security
- 8High manageability and interoperabilityControl automation, bulling system, system monitoring
- 9High accessibility and portabilityUniform access, thin client
- 10High performance and optimizationParallel processing, load balancing, job scheduling
- 11Enabling techniques- Hardware virtualization
- Parallelized and distributed computing
- Web-services - 12Service iswhat you connect using Web Services. Service is the endpoint of a connection
- 13Functionalities of service- A service should be well-defined
- A service should be self-contained
- A service should not depend on the context or state of other services. - 14Web serviceSelf-describing and stateless modules that perform discrete units of work and are available over the network Web service providers offer APIs that enable developers to exploit functionality over the Internet, rather than delivering full-blown applications
- 15Web Services Description Language (WSDL):- Expressed in XML which include both data type and messages
- Four types of operations:
1. One-way - Messages sent without a reply required
2. Request & response - Sending and replying to messages
3. Solicit response - A request for a response
4. Notification - Messages sent to multiple receivers - 16Service oriented architecture▪ Service Oriented Architecture (SOA) is essentially a collection of services which communicate with each other
▪ Contain a flexible set of design principles used during the phases of systems development and integration
▪ Provide a loosely integrated suite of services that can be used within multiple business domains - 17SOA approachUsually implemented by Web Service model
- 18Quality of Service (QoS)Set of technologies for managing network traffic in a cost-effective manner to enhance user experiences for home and enterprise environments
- 19Used in the following areas- Customer care evaluations
- Technological evaluations - 20Customer care evaluations▪ QoS is usually measured in terms of issues that have a direct impact on the experience of the customer
▪ Only issues that produce a negative effect on the goods and services received by the customer come under scrutiny - 21Technological evaluations▪ QoS has to do with the efficient operation of various systems
▪ This can lead to adjusting procedures or adapting software programs and code to achieve the desired effect while making a more efficient use of available resources - 22Service-level agreement (SLA)Contract between a network service provider and a customer that specifies, usually in measurable terms (QoS), what services the network service provider will furnish
- 23Performance guarantee metrics▪ Up-time and down-time ratio
▪ System throughput
▪ Response time - 24ScalabilityA desirable property of a system, a network, or a process, which indicates its ability to either handle growing amounts of work in a graceful manner or to be readily enlarged
- 25ElasticityThe ability to apply a quantifiable methodology that allows for the basis of an adaptive introspection within a real time infrastructure
- 26But how to achieve these scalability & elasticity- Dynamic provisioning
- Multi-tenant design - 27Dynamic Provisioningis a simplified way to explain a complex networked server computing environment where server computing instances are provisioned or deployed from an administrative console or client application by the server administrator, network administrator, or any other enabled user.
- 28Cloud provisioning
- 29Over-provisioning – out of cloud
- 30Cloud provisioning gains improved performance and reduced costs, but there can be two problems:1. Overestimating system utilization results in low utilization.
- 31Second problem2. Underestimating system utilization results in loss of revenue and loss of users
- 32Solve these two problems by dynamic resource provisioning- Meet seasonal demand variations
- Meet demand variations between different industries
- Meet burst demand for some extraordinary events
- 33Basic Architecture & Deployment Principle: Virtualization- Hardware resources (CPU, RAM, etc.) are partitioned and shared among multiple virtual machines (VMs)
- The virtual machine monitor (VMM or hypervisor) governs the access to the physical resources among running VMs
- Performance isolation and security - 34Virtual Machine
- 35Service models (IaaS, PaaS & SaaS)
- 36IaaSDeployment platform that abstract the infrastructure. The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer can deploy and run arbitrary software, which can include operating systems and applications.
The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components. - 37IaaS enabling technique▪ Virtualization (an abstraction of logical resources away from underlying physical resources)
▪ Server Virtualization
▪ Storage Virtualization
▪ Network Virtualization - 38IaaS provided services▪ Resource Management Interface
▪ System Monitoring Interface - 39PaaSDevelopment platform that abstract the infrastructure, OS, and middleware to drive developer productivity. The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider.
The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations. - 40PaaS enabling techniqueRuntime Environment
- 41PaaS provide services▪ Programming IDE
Programming APIs
Development tools
▪ System Control Interface
Policy based approach
Workflow based approach - 42SaaSfinished applications that you rent and customize. The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email)
- 43SaaS consumer restrictionsThe consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.
- 44SaaS enabling techniqueWeb Service
- 45SaaS provide services▪ Web-based Applications
General applications
Business applications
Scientific applications
▪ Government applications
Web Portal - 46Primary cloud deployment models.- Public Cloud
- Private Cloud
- Community Cloud
- Hybrid Cloud - 47Public Cloud- The cloud infrastructure is made available to the public or a large industry group and is owned by an organization selling cloud services.
- Also known as external cloud or multi-tenant cloud, this model essentially represents a cloud environment that is openly accessible.
- Basic characteristics:
▪ Homogeneous infrastructure
▪ Common policies
▪ Shared resources and multi-tenant
▪ Leased or rented infrastructure
▪ Economies of scale - 48Private Cloud- The cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on premise or off premise.
- Also referred to as internal cloud or on-premises cloud, a private cloud intentionally limits access to its resources to service consumers that belong to the same organization that owns the cloud.
- Basic characteristics:
▪ Heterogeneous infrastructure
▪ Customized and tailored policies
▪ Dedicated resources
9
▪ In-house infrastructure
▪ End-to-end control - 49Community CloudThe cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations).
- 50Hybrid CloudThe cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).
- 51General cloud characteristics- Scaling up and scaling down of resource usage on an as-needed basis
- Economies of scale: The cloud provider can procure real estate, power, cooling, bandwidth, and hardware at the best possible prices
- Pay-as-you-go: Resource allocation decisions have an immediate effect on resource consumption and the level of overall costs - 52Stateless components - Context▪ A componentized application subsumes components that can fail.
▪ No data is lost if an instance fails
▪ Multiple components have a common internal state
▪ The scalability of the central data store becomes the major challenge
▪ Distributed data stores
- 53Stateless components - Challenges▪ Cloud resources can display low availability.
▪ Component instances can be added and removed regularly when the demand changes. - 54Stateless components - SolutionImplement the components in a way that they do not contain any internal state, but completely rely on external persistent storage
- 55Elastic components - Context▪ A componentized application uses multiple compute nodes provided by an elastic infrastructure.
▪ If the utilization of compute nodes exceeds a specified threshold, additional hosting components are provisioned that contain the same application component.
▪ If the components are implemented as stateless
components, the operations for adding and removal of components are significantly simplified.
- 56Elastic components - Challenges▪ To benefit from the elastic infrastructure, the management process to scale-out a componentized application must be automated.
▪ Manual resource scaling would not support pay-per-use nor quickly changing workload - 57Elastic components - SolutionMonitor the utilization of compute nodes that host application components and automatically adjust their numbers using the provisioning functionality provided by the elastic infrastructure.
- 58Microservices architecture principles- Architecture Principles:
- Statelessness
- Available in multiple copies
- No data sharing among peers
- Each microservice its own DB
- Data-Intensive Message Brokering
▪ Message queues - 59Microservices
Information
- Description
- Engineering Flashcards on Data Engineering Lecture 2: Virtualization and Cloud computing, created by Tom van Eijk on 20/12/2021.
- Created by
- Tom van Eijk
- Views
- 0
- Rating