Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement PipelinePair primitive and related utility functions #4

Merged
merged 8 commits into from
May 22, 2024

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented May 16, 2024

This is the first of a series of PRs that revolves around implementing an evaluation harness (c.f deepset-ai/haystack#7526).

PipelinePair is a utility class that allows for two pipelines to be used in tandem. The motivation behind this abstraction is to reduce the amount of glue code it takes to connect a, say, RAG pipeline with an evaluation pipeline. It aims to be generic enough work with arbitrary pipelines while making special accommodations to ensure that pipelines of different "shapes" can be connected without much extra work from the user. "Shape" here refers to the modality of pipeline inputs and outputs, e.g: a typical RAG pipeline takes a query and returns an answer whereas a typical RAG evaluation pipeline will take batches of queries and answers. Connecting two such pipelines is handled by automatically aggregating and deaggregating inputs and outputs at the interface between the pipelines.

Blocked by #2.

@davidsbatista
Copy link
Contributor

This was a rather a big one, I tried to cover as much detail as possible, the logic looks OK to me, but I might have missed some edge cases - maybe checking what the test coverage shows to be sure we are testing every relevant piece of code.

Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this idea will simplify running a pipeline with the outputs of another pipeline as inputs. It looks quite flexible. 👍 Let's try it out with some examples and get feedback from users.

@shadeMe shadeMe requested a review from julian-risch May 21, 2024 12:09
Copy link
Member

@julian-risch julian-risch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to seeing this in action in RAGEvaluationHarness!

@shadeMe shadeMe marked this pull request as ready for review May 22, 2024 09:28
@shadeMe shadeMe requested a review from a team as a code owner May 22, 2024 09:28
@shadeMe shadeMe requested review from masci and removed request for a team May 22, 2024 09:28
@coveralls
Copy link

coveralls commented May 22, 2024

Pull Request Test Coverage Report for Build 9189096904

Details

  • 251 of 251 (100.0%) changed or added relevant lines in 5 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+10.7%) to 99.615%

Totals Coverage Status
Change from base Build 9186835496: 10.7%
Covered Lines: 259
Relevant Lines: 260

💛 - Coveralls

@shadeMe shadeMe merged commit d9454fc into deepset-ai:main May 22, 2024
5 checks passed
@shadeMe shadeMe deleted the feat/eval-harness-pipeline-pair branch May 22, 2024 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants