Skip to content
This repository has been archived by the owner on Mar 1, 2024. It is now read-only.

Wikipedia tools should return text, not documents (Tools aren't DataLoaders!) #828

Merged
merged 3 commits into from
Jan 4, 2024

Conversation

tslmy
Copy link
Contributor

@tslmy tslmy commented Dec 30, 2023

Description

IMO, a tool should work differently from a Data Loader. A tool should return the content, and just the content, if possible.
It should not expose to the LLM the fact that it is a Python program. The original implementation gives off the lower-level details to the LLM by emitting characters like these in the Observation:

[Document(id_='ea46abda-4165-4ba1-8cc6-36b0e681c0c5', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='735a0b01227b2ff21ece7f97445fbb9816668aee7922123966d16e057b0a1268', text='

This PR thrives to fix that. Here's a demo:

Before:

    {
      "role": "user",
      "content": "Look up cheesecakes on Wikipedia."
    },
    {
      "role": "assistant",
      "content": "Thought: I need to use a tool to help me find information about cheesecakes.\nAction: load_data\nAction Input: {'pages': ['Cheesecake']}"
    },
    {
      "role": "user",
      "content": "Observation: [Document(id_='ea46abda-4165-4ba1-8cc6-36b0e681c0c5', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='735a0b01227b2ff21ece7f97445fbb9816668aee7922123966d16e057b0a1268', text='Cheesecake is a dessert made with a soft fresh cheese

After:

    {
      "role": "user",
      "content": "Summarize Wikipedia page on Wellerism."
    },
    {
      "role": "assistant",
      "content": "Thought: I need to use a tool to help me answer the question.\nAction: load_data\nAction Input: {'page': 'Wellerism', 'lang': 'en'}"
    },
    {
      "role": "user",
      "content": "Observation: Wellerisms, named after sayings of Sam Weller in Charles Dickens's novel

I also don't think it's wise to load multiple pages with this tool, because each Wikipedia page alone is prone to exceed the context window of the LLM. Plus, the document separators would be too tiny for the LLM to notice. Most importantly, the multi-page support just doesn't match the README!

Type of Change

Please delete options that are not relevant.

  • New Loader/Tool
  • Bug fix / Smaller change
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new unit/integration tests
  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

Suggested Checklist:

  • I have added a library.json file if a new loader/tool was added
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

@nerdai
Copy link
Contributor

nerdai commented Jan 3, 2024

Fair arguments @tslmy. I can get behind this.

Can you fix so that it passes lint checks?

@tslmy
Copy link
Contributor Author

tslmy commented Jan 3, 2024

Done. Thanks for the review, @nerdai :D

@nerdai
Copy link
Contributor

nerdai commented Jan 3, 2024

looks like format check not passing still. Did you run make format and make lint?

@tslmy
Copy link
Contributor Author

tslmy commented Jan 4, 2024

@nerdai , not yet, sorry. I was at work and no access to my personal dev machine. Doing it now.

@nerdai nerdai merged commit b0d553b into run-llama:main Jan 4, 2024
3 checks passed
@tslmy tslmy deleted the patch-2 branch January 4, 2024 06:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants