Flashcards
M6 Exam
Cards in this deck (42)
- 1What are the main types of databases and what are the main differences between them?-Relational databases
-NoSQL databases
-Object-oriented databases
- Graph databases
The main differences:
Data structures
The way they store and organize data
The way they handle queries and data access
Main advantages:
Relational databases are well suited for structured data and support complex queries
NoSQL databases are well suited for unstructured data and are more scalable
Object-oriented databases are designed for object-oriented programming languages
Graph databases are designed to handle complex relationships between data entities - 2How can we use a database in an ML project, and can you give an example?Databases can be used to store, manage, and preprocess large amounts of data, which can then be used for model training, testing, and validation.
Use a database, such as MySQL or PostgreSQL, to store data in a structured format. Then use SQL queries to extract and transform the data into a format that can be used for training and testing a machine learning model - 3What is SQLSQL database:
SQL stands for Structured Query Language
SQL is a domain-specific language for managing relational databases (tables with relationships)
CRUD operations: Create, Read, Update, and Delete data
Commonly used in data analytics, data science, and data engineering - 4What is a relational databaseRelational Databases
Relational database managament systems (RDBMS) store data in tables with rows and columns
Each row represents a record or an entity
Each column represents an attribute of the record
Tables are connected through relationships (primary and foreign keys) - 5What is a foreign key in a relational database?A foreign key is a column or a combination of columns that references the primary key of another table.
It is used to establish a relationship between two tables by creating a link between them.
Foreign keys ensure referential integrity by preventing actions that would violate relationships between tables, such as deleting a record that is referenced by a foreign key in another table.
They also allow for efficient retrieval of related data from multiple tables. - 6Can you give me some examples of SQL syntax (statements) and their applications?SELECT: Read data from the database
INSERT: Add new records to the database
UPDATE: Modify existing records in the database
DELETE: Remove records from the database
CREATE, ALTER, DROP: Manage database structure
WHERE: Filter rows based on a condition
ORDER BY: Arrange rows based on a column
JOIN: Combine tables based on common columns
GROUP BY: operations on group level - 7Give some examples of SQL applications.Business Analytics:
- Data warehousing for storing historical data
- Structured data analysis for business intelligence and reporting
- Support for complex queries and aggregations for decision-making
Machine Learning:
- Data preprocessing for machine learning algorithms
- Feature engineering to create relevant features from raw data
- Joining multiple data sources for a comprehensive dataset
- Storing and managing machine learning model metadata and results - 8What are the different types of SQL and what are their key features and use cases?- MySQL
- PostgreSQL
-SQLite
-Microsoft SQL Server
-MariaDB
-Oracle - 9What are the main advantages of SQL?-ACID transactions
- Standardized language
- High Performance
- Scalability
- Flexibility
- Security
- Data Integrity
- Compatibility - 10What is NoSQL- NoSQL stands for "Not only SQL"
- Non-relational databases (document, key-value, column-family, graph)
- Designed to handle unstructured data and scalability challenges
- Flexible schema allows for evolving data structures
- Horizontal scalability for handling large volumes of data
- Generally, weaker consistency models compared to SQL databases - 11What are the different types of NoSQL and what are their key features and use cases?Document Databases:
- Store data in documents
-Documents can have nested structures
Key-Value Stores:
- Store data as key-value pairs
- Keys are unique identifiers for the data
Column-Family Stores:
- Store data in column families
- Designed for high write and read performance on large-scale data
Graph Databases:
- Store data as nodes (entities) and edges (relationships)
- Designed for graph traversal and querying connected data - 12What are the main advantages of NoSQL?Flexibility:
Schema flexibility for changing data requirements to handle unstructured data
Scalability/performance:
High scalability for large data volumes and high read/write rates
Availability:
High availability and fault tolerance through data replication even if a node cluster fails
Schemaless:
Optimized for specific use cases, depending on the type of NoSQL database
Cost:
NoSQL databases can be less expensive than traditional SQL databases - 13What is Apache Spark used for?Apache Spark is an open-source distributed computing system that provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. It is designed to handle large datasets by distributing the data and computation across a cluster of computers
Common use cases:
- Data processing
- Machine learning
- Real-time analytics
- Graph processing
- Data integration - 14Can you provide an example of a use case where Apache Spark might be preferred?e-commerce website that generates a massive amount of data, including customer interactions, clickstream data, and purchase history
-Real-time analytics (Spark streaming API): Analyze customer behavior to improve conversion and experience
- Machine learning (built in libraries): Build ML models for bettering experience (recommender system)
- Data integration: Intergrate data from multiple sources
- Data processing: Process and clean up large volumes of data - 15What is Polars used for?What is Polars? The best way to understand Polars is that it is a better dataframe library than Pandas. Here are some advantages of Polars over Pandas:
Polars does not use an index for the dataframe. Eliminating the index makes it much easier to manipulate the dataframe (the index is mostly redundant in Pandas dataframe anyway).
Polars represents data internally using Apache Arrow arrays while Pandas stores data internally using NumPy arrays. Apache Arrow arrays is much more efficient in areas like load time, memory usage, and computation.
Polars supports more parallel operations than Pandas. As Polars is written in Rust, it can run many operations in parallel.
Polars supports lazy evaluation. Based on your query, Polars will examine your queries, optimize them, and look for ways to accelerate the query or reduce memory usage. Pandas, on the other hand, support only eager evaluation, which immediately evaluates an expression as soon as it encounters one. - 16Example of a use case where Polars might be preferred over pandas?One use case where Polars might be preferred over pandas is when dealing with large datasets that do not fit in memory. Unlike pandas, which loads the entire dataset into memory, Polars uses a memory-mapped file format called Arrow that allows it to read and write data in chunks. This makes it possible to work with very large datasets without running into memory issues.
Overall, if you are working with large datasets, need to perform computationally intensive operations, or are working with time-series data, Polars may be a better choice than pandas. However, if you are working with smaller datasets that fit in memory and do not require complex operations, pandas may be simpler and more efficient to use. - 17What are the key differences between Apache Spark, Polars, and Pandas, and their use cases?Overall, the key differences between these tools come down to their scalability and performance capabilities. Apache Spark is designed for processing and analyzing very large datasets that require distributed processing, while Polars is designed for fast, memory-efficient processing of large structured datasets. Pandas is designed for working with smaller datasets that can fit into memory on a single machine. The choice of tool depends on the size and complexity of the dataset, the performance requirements, and the specific use case.
- 1814. What is MLOps, and why is it important in the context of machine learning projects?MLOps (Machine Learning Operations) is a set of practices and tools used to manage the lifecycle of ML models in production environments. It combines principles and techniques from ML, DevOps, and data engineering to create a consistent and efficient process for managing ML projects.
Why is MLOps important:
- Collaboration:
- Efficiency:
- Reproducibility:
- Scalability:
MLOps is important because it helps to streamline the development, deployment, and maintenance of models, making it easier to collaborate across teams, improve efficiency, ensure reproducibility, and scale up projects as needed. - 19How does MLOps help in streamlining the machine learning lifecycle from development to deployment?Data preparation:
MLOps provides tools and techniques for managing data, including data cleaning, transformation, and feature engineering.
Model development:
MLOps provides tools and techniques for managing the model development process, including version control, testing, and documentation.
Model training:
MLOps provides tools and techniques for managing the model training process, including scalability and reproducibility.
Model deployment:
MLOps provides tools and techniques for managing the model deployment process, including automation and scalability.
Model monitoring:
MLOps provides tools and techniques for managing the model monitoring process, including tracking model performance and detecting anomalies. - 20How do Continuous Integration (CI) and Continuous Deployment (CD) principles apply to MLOps?CI/CD can help streamline the ML development process, by automating tasks involved in building, testing, and deploying models. For example:
Continuous Integration:
- Data scientists can use CI tools such as GitHub Actions, CircleCI, or Jenkins to automatically build and test their models whenever changes are made to the code or data.
- For example, when a new feature is added to the model or when new data becomes available, the CI system can automatically rebuild and test the model to ensure that it is still working as expected.
Continuous Deployment:
- Once a model has been developed and tested, CD tools such as Kubernetes, Argo, or Jenkins can be used to automatically deploy the model to production.
- For example, when a new version of the model is ready, the CD system can automatically deploy it to a testing environment, where it can be further evaluated and tested. Once the model has passed all tests, it can be automatically deployed to production. - 21What are some popular tools and platforms used in MLOps for managing and deploying machine learning models?Experiment Tracking and Model Metadata Management Tools:
- MLFlow
- Comet ML
- Weights & Biases
Orchestration and Workflow Pipelines MLOps Tools:
- Prefect
- Metaflow
- Kedro
Data and Pipeline Versioning Tools:
- Pachyderm
- Data Version Control (DVC)
Model Deployment and Serving Tools:
- TensorFlow Extended (TFX) Serving
- BentoML
- Cortex
Model Monitoring in Production ML Ops Tools:
- Evidently
- Fiddler
- Censius AI
End-to-End MLOps Platforms
- AWS SageMaker
- DagsHub
- Kubeflow - 22Can you give a best practice for implementing MLOps in an organization?Define your ML goals and use cases:
- Identify business problems that you want to solve using machine learning.
Assemble a team:
- Build a team with the necessary skills to work on the project.
Define data pipelines:
- Create data pipelines to collect, preprocess, and store data for machine learning.
Develop and train models:
- Develop models using best practices in data science and machine learning.
Test and evaluate models:
- Test models using validation and testing techniques to ensure that they are accurate and reliable.
Deploy models:
- Deploy models using containerization tools like Docker or Kubernetes.
Monitor and optimize:
- Establish a process to monitor the performance of models in production and to optimize them when necessary. - 23What is MLflow, and how can it help in terms of MLOps?MLFlow is a platform for managing the end-to-end machine learning lifecycle
Experiment Tracking:
Helps track ML experiments by recording and visualizing
Model Packaging:
Simple format for packaging code in a reusable and reproducible way
Model Registry:
Centralized repository for managing and sharing ML models - 24What types of information can be stored in MLflow?MLflow can store information related to ML experiments, including:
Metrics: Numeric values that track the performance of a model during training or testing, such as accuracy, precision, recall, and F1-score.
Parameters: Settings and configurations used during model training, such as learning rate, batch size, and number of epochs.
Artifacts: Any output generated during model training or evaluation, such as trained models, visualizations, or data preprocessing scripts.
Source code: The code used to train and evaluate the model, along with any dependencies and environment settings. Experiment metadata: Information about the experiment, such as the name, date, user, and any tags or annotations. - 25What is the purpose of SSH?SSH = Secure Shell
Provides a secure connection between two entities (computer/computer or computer/remote server) over an unsecured network such as the internet - 26What is Docker, and why is it useful?Docker is an open-source containerization platform that allows software developers and system administrators to package, distribute, and run applications in isolated and portable containers.
Docker is useful for several reasons:
- Portability: Docker allows developers to package an application with all its dependencies and configurations into a container
- Consistency: By using Docker, developers can ensure that the application runs consistently across different environments,
- Scalability: Docker enables developers to quickly and easily spin up multiple instances of an application in a matter of seconds
- Isolation: Docker containers provide a secure and isolated runtime environment
- Efficiency: Docker containers are lightweight and have minimal overhead - 27What are containers, and how do they relate to Docker?- Containers are a technology that allows developers to package and run applications with all their dependencies and configurations in an isolated environment.
- Containers are lightweight, portable, and provide a consistent runtime environment for applications, regardless of the underlying infrastructure.
A Docker container is a running instance of a Docker image.
When a Docker image is run, it creates a container that runs in isolation from the host operating system, providing a secure, reproducible and consistent runtime environment for the application. - 28What is the difference between a virtual machine and a Docker container?The key difference between a virtual machine (VM) and a Docker container is the way they use and interact with the host operating system.
- Virtual machines are like a computer within a computer. They run a complete operating system and require a significant amount of resources to run (CPU, memory, disk space). Each VM is isolated from the host operating system and other VMs running on the same hardware.
- Docker containers are like a lightweight wrapper around an application and its dependencies. They share the same kernel as the host operating system and are isolated from other containers running on the same hardware. - 29What is a Docker image, and how is it created?Docker image:
A Docker image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings.
ow is it created:
- Docker images are created using a Dockerfile, which is a text file that contains instructions on how to build the image.
- The Dockerfile specifies the base image, sets environment variables, copies files, installs dependencies, and runs commands to configure the software. - 30What is Docker Hub, and what is its purpose?Docker Hub:
- Public registry that allows developers to store, share, and distribute Docker images.
- It serves as a central repository for Docker images (like GitHub does for code).
Purpose of Docker Hub:
- To store Docker images and make them available to others.
- Discover and download images that others have created and shared.
- Collaborate with others by sharing images and collaborating on Dockerfiles.
- Automate the build and testing of images using Continuous Integration (CI) tools. - 31What is a Dockerfile, and what are its main components?Base image:
This is the starting point for the image. The base image provides a set of pre-configured operating system files and packages that the new image can build on.
Instructions:
These are the commands that tell Docker how to build the image. For example, an instruction might install a specific package or copy files from the local file system into the image
Arguments:
These are variables that can be passed to the Dockerfile at build time.
Environment variables:
These are variables that can be set within the image and used by the container at runtime.
Labels:
These are key-value pairs that provide metadata about the image, such as the version, author, and build date.
Working directory:
This is the directory where the Dockerfile instructions are executed. It can be set to any location within the image. - 32How do you build a Docker image using a Dockerfile?Create a Dockerfile:
Create a text file called "Dockerfile" in a directory where you want to build the image
Write the Dockerfile instructions:
In the Dockerfile, write the instructions for building the image.
These instructions typically start with a base image and include commands for installing packages, copying files, and configuring the environment.
Build the Docker image:
Run the "docker build" command in the directory where the Dockerfile is located. This command will read the instructions in the Dockerfile and build a Docker image based on those instructions. - 33How can you share data between Docker containers?Docker volumes (recommended method):
- A volume is a persistent data storage mechanism that can be shared between multiple containers.
Docker networks:
- Docker networks enable containers to communicate with each other, and data can be shared between containers through network connections.
Shared file systems:
- If multiple containers are running on the same host, they can share data using a shared file system.
Docker Compose:
- Docker Compose is a tool for defining and running multi-container Docker applications. - 34How would you define Code Refactoring (CF)?Definition:
CF is the process of restructuring existing code to improve its internal structure, without changing its external behaviour
Goal:
To make the code easier to read,understand, and maintain as well as to improve it's performance, scalabiltiy and reliability
Steps:
- Transformation of 'working' code into deployed code
- Separating parts of code into multiple files
- Organizing repository to be deployment-ready - 35How does CF influence the external behavior of the code?Code refactoring should not change the external behavior of the code. The purpose of refactoring is to improve the internal structure and design of the code, without modifying its functionality or behavior.
- 36What are the concrete goals one wants to achieve with CF?Common goals of CF include:
- Improving code readability: By improving the structure and organization of the code, it becomes easier for developers to understand and work with it.
- Increasing code maintainability: CF can help simplify complex code, reduce dependencies, and eliminate redundant code, making it easier to maintain it.
- Enhancing code reusability: CF can help extract common functionality into reusable components, reducing the amount of code that needs to be written and improving overall code efficiency.
- Optimizing code performance: CF can help identify and eliminate bottlenecks and improve the overall performance.
- Reducing technical debt: By addressing technical debt, or the accumulation of bad code practices over time, CF can help improve the overall quality and sustainability of the codebase.
- Making the code more testable: CF can help reduce coupling and increase cohesion, making the code more modular and easier to test. - 37What do we call “code smell,” and what are the most common types?"Code smell" refers to potential design problems or poor programming practices in code.
Common types of code smell includes:
Long functions
Duplicate code
Dead code
Data clumps
Improper names
Other types includes:
Large class
Feature envy
Primitive obsession
Shotgun surgery
God object - 38How can we improve understanding in reusability of our code when publishing in a specific platform or in specific company?Standardization:
Developing a set of coding guidelines and standards can help ensure that the code is written in a consistent and reusable manner.
Documentation:
Documenting the code, including comments, API documentation, and user guides
Code reuse analysis:
Analyzing the code to identify common patterns or functions that can be reused
Education and training:
Help devs understand the importance of code reusability and best practices for achieving it.
Collaboration:
Encouraging collaboration among devs can help improve understanding of code reusability. - 39What is an API?
- 40What types of API do we know based on their availability?Public API
is open and available for use by any outside developer or business. These are also called open or external APIs
Partner API
is only available to specifically selected and authorized outside developers or API consumers. It facilitates business-to-business activities
Private API
is intended only for use within the enterprise to connect systems and data within the business
Composite API
is a sequence of tasks that run synchronously as a result of the execution and not at the request of a task. - 41How are the diffrent types of API used? Public API: Partner API: Private API: Composite APIs:Public APIs are useful for promoting services or products
Partner APIs are useful for integrating with specific services or platforms
Private APIs are useful for facilitating communication within an organization
Composite APIs are useful for simplifying complex processes - 42What are the types based on their protocol? (API)REST (Representational State Transfer)
is a web services API and crucial for modern web applications
SOAP (Simple object access protocol)
is a well-established protocol but comes with strict rules, rigid standards
RPC (Remote Procedure Call protocol)
is the oldest and simplest type of API with a goal for the client to execute code on a server
Event-driven or asynchronous APIs
transmit information in quasi-real-time. The advantage is that it allows the source to send a response only when the information is new or has changed, useful for stock exchanges
Information
- Description
- Computer Science Flashcards on M6 Exam, created by Nikolaj Vestergaard on 09/05/2023.
- Created by
- Nikolaj Vestergaard
- Views
- 12
- Rating