[Feature] DolphinScheduler Lineage

[zhongjiajie]

This discussion is about DolphinScheduler Lineage. The ref ISSUE is #249, but it more like vote tickle instead of issue, So I create a discussion to continue the work.

I init thinks we should draw following conclusions after discuss:

• The purpose of data lineage(from [Feature] DolphinScheduler Lineage #6596 (comment))
  • Discover potential dependencies
  • Build workflow through data lineage automation
  • Easy to debug
• The way we implement lineage, we have to add a documentation before we start write the code(Of cause we could add some POC code for it)
• To be adding if someone have other idea.

[wen-hemin]

It is recommended to start with collecting requirements, What is the purpose of data lineage ?

• Discover potential dependencies
• Build workflow through data lineage automation
• Easy to debug

[ChenpiDog]

@zhongjiajie

[ChenpiDog]

The data blood relationship information is not suitable for storage in traditional relational databases. Whether to consider introducing a graph database to store this kind of information.

[zhongjiajie]

@ChenpiDog looking good, do you interested in this feature? I think currently this issue still in help wantted

[ChenpiDog]

@zhongjiajie I'm very interested in this, but I'm not familiar with Java and can't contribute to the code. I can provide some ideas.

[ChenpiDog]

There are two main aspects to consider in visualization:

1. Automatically generate Lineage data kinship from online code (such as SQL);
2. It is necessary to consider the views and materialized views of the database, which are generally stored in the database meta information table.

Note: The global Lineage data kinship needs to combine the above two points. For example, the source table used in my SQL is a view.