ETL, Data Integration and Data Interchange

What the resources cover

Much of this unit follows Informatica-style ETL work. Mind maps break down individual transformations: Lookup trans, Lookup query, HTTP Trans, SQL Override, Stored Procedure Trans, Filter Trans and AGGREGATOR. Several have matching notes made from the map, so you can read the same material as running text.

A second group covers data interchange: XPath, XML & XML Schema, JSON Overview, Data Exchange Methods/Systems and a few maps on JSON and API data. There are also maps on SAP and BODS data transfers.

Where to start

If you are new to ETL, begin with ETL PHASE and ETL Functions to get the overall picture. Then move to the transformation maps. Lookup is covered in the most depth, so it makes a good first transformation: read Lookup trans, then Lookup condition and Lookup query.

If your interest is data exchange, open JSON Overview and XML & XML Schema first, then XPath.

Using quizzes and maps together

The quizzes are scenario-based and tool-specific. There are quizzes on Oracle Data Integrator, on IBM DataStage design questions, and on Informatica and mapplets. A sensible routine:

1. Read a map on one transformation.
2. Take a quiz and note the questions you missed.
3. Go back to the map or note that covers the missed point.
4. Use the flashcard deck LOOKUP Transformation for quick recall.

The flashcards on distributed and heterogeneous databases, data integration and data quality add a more theoretical angle for lecture-based courses.

Where this unit sits

This unit belongs to the Data Management and Engineering strand of Databases and data science. Most resources suit university students and people preparing for professional roles, and teachers can pull maps and quizzes into their own lessons.

Next step

Choose one transformation or one data format, work through its map, and then see how the matching quiz goes. Making your own maps and flashcards is a good way to fill whatever gaps remain.