Flashcards
Data Engineering Lecture 8: DevOps and CI/CD
Cards in this deck (31)
- 1Formal definition DevOps“DevOps is a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality"
- 2Developers (Devs) and operators (Ops) do not always pursue the same goals- Developers develop models
- Operators they take care of the operation of the live system (hosting & customers) - 3Problems that DevOps is trying to solve:- Poor communication between Dev and Ops
- Opposing goals
▪ Devs want to push new features
▪ Ops want to keep the system available (uptime, downtime, performance, rollout, etc.)
▪ This leads to slow-release schedule
- Limited capacity of operations staff (team dev > team ops)
- Limited insight into operations
- Why companies care?
▪ IBM has managed to schedule new features with success by applying DevOps - 4DevOps motivation- Organizations want to reduce time to market for new features, without sacrificing quality
- Requires business-IT alignment (align the technical and business ideas)
- DevOps practices will influence…
▪ the way you organize teams
▪ the way you build systems
▪ even the structure of the systems that you build
- Unlikely that you’ll be able to “throw your final version over the fence” and let operations worry about running it!
- DevOps is about...
▪ bringing “agile” methods to operations
▪ encouraging collaboration between development and operations staff, get them talking
▪ Formally: shared goals and teams of Devs and Ops - 5What does it mean:- Quality of the code must be high
- Quality of the build & delivery mechanism must be high
- Time is split
- Goal-oriented definition - 6Quality of the code must be highTesting
- 7Quality of the build & delivery mechanism must be highAutomation & more testing
- 8Time is split▪ From commit to deployment to production
▪ From deployment to acceptance into normal production - 9Goal-oriented definition▪ May use best practices like agile methods, continuous deployment (CD), etc.
▪ Likely to use tools - 10DevOps practices
- 11Treat Ops as first-class citizens throughout the lifecycle –e.g., in requirements elicitation▪ Many decisions can make operating a system harder or easier
▪ Logging and monitoring to suit Ops - 12Make Dev more responsible for relevant incident handlingShorten the time between finding and repairing errors
- 13Use continuous integration & continuous deployment to automate everythingCommits trigger automatic build, testing, deployment
- 14Enforce that the deployment process is used by all stakeholders Release management▪ No ad-hoc deployments
▪ Ensures changes are traceable - 15Develop infrastructure code with the same set of practices as application code▪ “Infrastructure as Code”: using IaaS APIs, etc., to automate the creation of environments
▪ Misconfiguration can derail your application
▪ Ops scripts are traditionally more ad-hoc - 16ExampleRequirement that when a change was made in the SAP system, a real-time integration was required to
the Quintiq system - 17Continuous Integration (CI)The software development
practice of automatically testing and compiling changes made
by developers as they happen. This is usually accomplished by
a series of tools, as different requirements by the developers and
by the clients drive different outcomes. - 18CI cycle
- 19Three pillars of improvement:1. Higher quality models
2. Faster development
3. Streamlined testing - 20Old way vs New way- Only the most common problems are tested in the old way.
- A lot of things can go wrong in the old way. Moreover, it slows release.
- In the new way, a reviewer can approve multiple changes in one go -> more speed.
- 21Dozen stepsWe will require a lot of manual testing
- 22Streamlined testing
- 23Streamlined testing advantageEvery method / function we implement in our programming language, the developer must write a UTF test script for that. So, if you go into the model, there is not only code, but also the test script.
- 24Functional test frameworkScript that does some actions needed to test a new feature, for instance, a couple of buttons need to be pressed, some input needs to be given, mouse movement, etc.
▪ Do I get the expected output? Fail / pass. - 25Integration test frameworkIntegration of several systems (we have inbound and outbound interfaces with external system).
▪ It is done by creating some sample messages (xml, Jason) and to see if the systems behave as expected & nothing breaks. - 26PFTBuilt on the FTF and is a simulation of user action and measures performance (time, durations, etc.)
▪ Performance is still tested manually! Work in progress.
- 27Case DELMIA Quintiq continuous integration1. Developer pushes a commit
2. Bitbucket notifies Jenkins of a new commit
3. Jenkins-file launches the Quintiq library for the configured
build process
4. DQLib retrieves data for build from file server (datasets, data)
5. DQLib generates a set of instructions for docker to initialize a MSSQL
server. Only if database functionality is enabled.
6. Docker generates a container with MSSQL Server to connect to
Quintiq application.
7. DQLib attempts to load model to check for errors/warnings and run
tests (if tests available).
8. After build, DQLib publishes adjusted data to file server where it is configured to.
9. DQLib sends results of the build to Jenkins for display
10. Jenkins sends a webhook to bitbucket to update the status of the build - 28Continuous deploymentCI was about every bill & every release, and it was just about testing (i.e., you can automatically test a release). But what about rolling out the release to some staging area?
- 29Delivery and deployment differ in only a little, namely, deployment contains only automatic steps while delivery has manual deployment to production. DevOps, entails the whole process, including CI & CD!!
- 30Continuous Delivery vs Continuous Deployment
- 31Release managementRM is important when you implement a big roll-out program for different size factories.
Information
- Description
- Engineering Flashcards on Data Engineering Lecture 8: DevOps and CI/CD, created by Tom van Eijk on 21/12/2021.
- Created by
- Tom van Eijk
- Views
- 0
- Rating