Data Lineage
Data lineage tracks your data’s journey from its source to its eventual end purpose. It logs every step along the way, including explanations of how and why the data was moved from one location to the next. You can track data lineage via your data catalog to keep up with day-to-day data usage or to aid in error resolution.
For example:
If you weren’t sure why a dataset is missing a particular column or data point, you could use data lineage to trace the data back to its’ origin and find the cause of the missing information.
To perform data lineage effectively, you’ll need to undergo a substantial amount of sorting through your data system. You’ll need to identify the critical data that requires data lineage, usually by asking your business users. Then track that data back to its origin, one by one, creating a spreadsheet to reference sources and link elements together. Finally, you’ll need to create a map that can provide a complete picture of your data’s journey.
As volumes of data grow, data lineage is essential as it will provide business intelligence with all the necessary information about the data lifecycle. Understanding and improving your data’s movement through ETL, databases, reports, etc., can benefit product development and reduce the time it takes for larger projects like data migration. It also helps with locating data for regulatory requirements like the GDPR.
