Skip an operator if this is a duplicate op instead of raise error

[chjuncn]

Rationals

I checked how to generate basic nodes from Dataset and I think this is the best change to achieve what we want for the interface. But we might have better solutions if we change how we generate nodes from Dataset. If this change violates some principles of the design in PZ, I'm happy to discuss other solutions.

Final Effects

1. Dataset() init only has one responsibility: wrap a datasource to a Dataset. I think this is a better interface.
2. No extra convert() will be added to the plan.
3. When users add the same op multiple times dataset.convert(File).convert(File), the system will just dedup the same op instead of raise error.

Issue

Currently Dataset(src, schema) initiation has 2 responsibilities:

1. read source
2. convert source to schema.

When we want to save schema param for users and use default schema for Dataset init(source, schema=DefaultSchema) for users, the code works like:

1. Read source to schema that DataSource provides. This schema is derived by the system, so the users don't know (don't need to know).
2. Convert Source schema to DefaultSchema.

The system will make one more convert call to convert SourceSchema to DefaultSchema every time, which is definitely wrong.

Solution

1. We use schema from Datasource if exists, which is reasonable.
2. If we do 1, then we'll get a dataset node that no actual op as its input_schema ==output_schema, so I updated a line in optimizer to just skip the node if it doesn't do anything instead raiseerror.

Real Examples

Before

Generated plan:
0. MarshalAndScanDataOp -> PDFFile

1. PDFFile -> LLMConvertBonded -> DefaultSchema (contents, filename, text_conte) -> (value) Model: Model.GPT_4o Prompt Strategy: PromptStrategy.COT_QA

2. DefaultSchema -> MixtureOfAgentsConvert -> ScientificPaper (value) -> (contents, filename, paper_auth) Prompt Strategy: None Proposer Models: [GPT_4o] Temperatures: [0.0] Aggregator Model: Model.GPT_4o Proposer Prompt Strategy: chain-of-thought-mixture-of-agents-proposer Aggregator Prompt Strategy: chain-of-thought-mixture-of-agents-aggregation

3. ScientificPaper -> LLMFilter -> ScientificPaper (contents, filename, paper_auth) -> (contents, filename, paper_auth) Model: Model.GPT_4o Filter: The paper mentions phosphorylation of Exo1

4. ScientificPaper -> MixtureOfAgentsConvert -> Reference (contents, filename, paper_auth) -> (reference_first_author, refere) Prompt Strategy: None Proposer Models: [GPT_4o] Temperatures: [0.8] Aggregator Model: Model.GPT_4o Proposer Prompt Strategy: chain-of-thought-mixture-of-agents-proposer Aggregator Prompt Strategy: chain-of-thought-mixture-of-agents-aggregation

After the change

Generated plan:
0. MarshalAndScanDataOp -> PDFFile

1. PDFFile -> LLMConvertBonded -> ScientificPaper (contents, filename, text_conte) -> (contents, filename, paper_auth) Model: Model.GPT_4o Prompt Strategy: PromptStrategy.COT_QA

2. ScientificPaper -> LLMFilter -> ScientificPaper (contents, filename, paper_auth) -> (contents, filename, paper_auth) Model: Model.GPT_4o Filter: The paper mentions phosphorylation of Exo1

3. ScientificPaper -> MixtureOfAgentsConvert -> Reference (contents, filename, paper_auth) -> (reference_first_author, refere) Prompt Strategy: None Proposer Models: [GPT_4o] Temperatures: [0.8] Aggregator Model: Model.GPT_4o Proposer Prompt Strategy: chain-of-thought-mixture-of-agents-proposer Aggregator Prompt Strategy: chain-of-thought-mixture-of-agents-aggregation