All terms
Glossary
ETL
Also: Data pipeline
Data is extracted from source systems, transformed and loaded into a target, usually on a schedule.
Classically this runs overnight as a batch. Modern variants load first and transform in the target, because compute there has become cheap.
The hard part is never copying, it is dealing with deviations: missing fields, changed formats, duplicate records.
How you notice it
- Data from several systems needs to be combined.
- Reports run against the production database.
- Numbers contradict each other depending on the source.
Frequently asked
ETL or ELT?
ELT when the target is a modern analytics database: load first, transform there, because the compute is available anyway and raw data is preserved. ETL stays right when data must be anonymized or filtered before loading.