Add an agent that targets Omen markets that are still open but have a known outcome #33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalkthroughThe recent update focuses on enhancing the prediction market agent's capabilities and organizational structure. It reorganizes import paths for web scraping and search tools across several agent files, introduces a new agent for markets with known outcomes along with deployment capabilities, and adds new tools for web scraping to Markdown and for performing web searches via the Tavily API. This update streamlines the agent's operation and expands its functionality, particularly in handling known outcome predictions and improving data extraction methods. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 5
Configuration used: CodeRabbit UI
Files ignored due to path filters (2)
poetry.lockis excluded by:!**/*.lockpyproject.tomlis excluded by:!**/*.toml
Files selected for processing (8)
- prediction_market_agent/agents/autogen_agent.py (1 hunks)
- prediction_market_agent/agents/custom_agent.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py (1 hunks)
- prediction_market_agent/agents/llamaindex_agent.py (1 hunks)
- prediction_market_agent/tools/web_scrape/markdown.py (1 hunks)
- prediction_market_agent/tools/web_search/tavily.py (1 hunks)
Additional comments: 8
prediction_market_agent/tools/web_search/tavily.py (2)
- 13-14: The retry mechanism is correctly implemented using the
tenacitylibrary, ensuring resilience against transient failures.- 16-16: Using
@persistent_inmemory_cacheis a good practice for reducing redundant web search calls, enhancing performance.prediction_market_agent/agents/llamaindex_agent.py (1)
- 8-9: Reorganizing imports for
web_scrapeandgoogle_searchmodules improves the clarity and maintainability of the code.prediction_market_agent/tools/web_scrape/markdown.py (2)
- 9-11: The retry mechanism using
tenacityis correctly applied to enhance the reliability of thefetch_htmlfunction.- 12-13: Applying
@persistent_inmemory_cachetofetch_htmlis a good practice for improving performance by avoiding redundant network calls.prediction_market_agent/agents/known_outcome_agent/deploy.py (1)
- 17-18: The
market_is_saturatedfunction is a simple and effective way to filter out markets with a high probability of a known outcome.prediction_market_agent/agents/autogen_agent.py (1)
- 10-11: Reorganizing imports for
WebScrapingToolandGoogleSearchToolimproves the clarity and maintainability of the code.prediction_market_agent/agents/custom_agent.py (1)
- 20-23: Reorganizing imports for
web_scrape_structured_and_summarizedandgoogle_searchmodules improves the clarity and maintainability of the code.
Comment on lines
17
to
38
| def web_search(query: str, max_results: int) -> list[WebSearchResult]: | ||
| """ | ||
| Web search using Tavily API. | ||
| """ | ||
| tavily_api_key = secret_str_from_env("TAVILY_API_KEY") | ||
| tavily = TavilyClient(api_key=tavily_api_key.get_secret_value()) | ||
| response = tavily.search( | ||
| query=query, | ||
| search_depth="advanced", | ||
| max_results=max_results, | ||
| include_raw_content=True, | ||
| ) | ||
|
|
||
| results = [ | ||
| WebSearchResult( | ||
| url=result["url"], | ||
| query=query, | ||
| ) | ||
| for result in response["results"] | ||
| ] | ||
|
|
||
| return results |
There was a problem hiding this comment.
The web_search function correctly utilizes the Tavily API for performing web searches. However, consider handling potential exceptions from the TavilyClient's search method to ensure robust error handling.
try:
response = tavily.search(
query=query,
search_depth="advanced",
max_results=max_results,
include_raw_content=True,
)
except Exception as e:
# Log the error and handle it appropriately
print(f"Error during Tavily search: {e}")
return []Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
Suggested change
| def web_search(query: str, max_results: int) -> list[WebSearchResult]: | |
| """ | |
| Web search using Tavily API. | |
| """ | |
| tavily_api_key = secret_str_from_env("TAVILY_API_KEY") | |
| tavily = TavilyClient(api_key=tavily_api_key.get_secret_value()) | |
| response = tavily.search( | |
| query=query, | |
| search_depth="advanced", | |
| max_results=max_results, | |
| include_raw_content=True, | |
| ) | |
| results = [ | |
| WebSearchResult( | |
| url=result["url"], | |
| query=query, | |
| ) | |
| for result in response["results"] | |
| ] | |
| return results | |
| def web_search(query: str, max_results: int) -> list[WebSearchResult]: | |
| """ | |
| Web search using Tavily API. | |
| """ | |
| tavily_api_key = secret_str_from_env("TAVILY_API_KEY") | |
| tavily = TavilyClient(api_key=tavily_api_key.get_secret_value()) | |
| try: | |
| response = tavily.search( | |
| query=query, | |
| search_depth="advanced", | |
| max_results=max_results, | |
| include_raw_content=True, | |
| ) | |
| except Exception as e: | |
| # Log the error and handle it appropriately | |
| print(f"Error during Tavily search: {e}") | |
| return [] | |
| results = [ | |
| WebSearchResult( | |
| url=result["url"], | |
| query=query, | |
| ) | |
| for result in response["results"] | |
| ] | |
| return results |
Comment on lines
+21
to
+53
| def web_scrape(url: str, timeout: int = 10) -> str: | ||
| """ | ||
| Taken from polywrap/predictionprophet | ||
|
|
||
| https://github.com/polywrap/predictionprophet/blob/97aeea8f87e9b42da242d00d93ed5754bd64f21e/prediction_prophet/functions/web_scrape.py | ||
| """ | ||
| try: | ||
| response = fetch_html(url=url, timeout=timeout) | ||
|
|
||
| if "text/html" in response.headers.get("Content-Type", ""): | ||
| soup = BeautifulSoup(response.content, "html.parser") | ||
|
|
||
| [x.extract() for x in soup.findAll("script")] | ||
| [x.extract() for x in soup.findAll("style")] | ||
| [x.extract() for x in soup.findAll("noscript")] | ||
| [x.extract() for x in soup.findAll("link")] | ||
| [x.extract() for x in soup.findAll("head")] | ||
| [x.extract() for x in soup.findAll("image")] | ||
| [x.extract() for x in soup.findAll("img")] | ||
|
|
||
| text: str = soup.get_text() | ||
| text = markdownify(text) | ||
| text = " ".join([x.strip() for x in text.split("\n")]) | ||
| text = " ".join([x.strip() for x in text.split(" ")]) | ||
|
|
||
| return text | ||
| else: | ||
| print("Non-HTML content received") | ||
| return "" | ||
|
|
||
| except requests.RequestException as e: | ||
| print(f"HTTP request failed: {e}") | ||
| return "" |
There was a problem hiding this comment.
The web_scrape function correctly fetches and processes HTML content. However, consider adding error handling for the requests.get call and validating the response status code to ensure robustness.
try:
response = requests.get(url, headers=headers, timeout=timeout)
+ response.raise_for_status() # Ensure the request was successful
except requests.RequestException as e:
print(f"HTTP request failed: {e}")
return ""Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
Suggested change
| def web_scrape(url: str, timeout: int = 10) -> str: | |
| """ | |
| Taken from polywrap/predictionprophet | |
| https://github.com/polywrap/predictionprophet/blob/97aeea8f87e9b42da242d00d93ed5754bd64f21e/prediction_prophet/functions/web_scrape.py | |
| """ | |
| try: | |
| response = fetch_html(url=url, timeout=timeout) | |
| if "text/html" in response.headers.get("Content-Type", ""): | |
| soup = BeautifulSoup(response.content, "html.parser") | |
| [x.extract() for x in soup.findAll("script")] | |
| [x.extract() for x in soup.findAll("style")] | |
| [x.extract() for x in soup.findAll("noscript")] | |
| [x.extract() for x in soup.findAll("link")] | |
| [x.extract() for x in soup.findAll("head")] | |
| [x.extract() for x in soup.findAll("image")] | |
| [x.extract() for x in soup.findAll("img")] | |
| text: str = soup.get_text() | |
| text = markdownify(text) | |
| text = " ".join([x.strip() for x in text.split("\n")]) | |
| text = " ".join([x.strip() for x in text.split(" ")]) | |
| return text | |
| else: | |
| print("Non-HTML content received") | |
| return "" | |
| except requests.RequestException as e: | |
| print(f"HTTP request failed: {e}") | |
| return "" | |
| def web_scrape(url: str, timeout: int = 10) -> str: | |
| """ | |
| Taken from polywrap/predictionprophet | |
| https://github.com/polywrap/predictionprophet/blob/97aeea8f87e9b42da242d00d93ed5754bd64f21e/prediction_prophet/functions/web_scrape.py | |
| """ | |
| try: | |
| response = fetch_html(url=url, timeout=timeout) | |
| response.raise_for_status() # Ensure the request was successful | |
| if "text/html" in response.headers.get("Content-Type", ""): | |
| soup = BeautifulSoup(response.content, "html.parser") | |
| [x.extract() for x in soup.findAll("script")] | |
| [x.extract() for x in soup.findAll("style")] | |
| [x.extract() for x in soup.findAll("noscript")] | |
| [x.extract() for x in soup.findAll("link")] | |
| [x.extract() for x in soup.findAll("head")] | |
| [x.extract() for x in soup.findAll("image")] | |
| [x.extract() for x in soup.findAll("img")] | |
| text: str = soup.get_text() | |
| text = markdownify(text) | |
| text = " ".join([x.strip() for x in text.split("\n")]) | |
| text = " ".join([x.strip() for x in text.split(" ")]) | |
| return text | |
| else: | |
| print("Non-HTML content received") | |
| return "" | |
| except requests.RequestException as e: | |
| print(f"HTTP request failed: {e}") | |
| return "" |
Comment on lines
21
to
78
| class DeployableKnownOutcomeAgent(DeployableAgent): | ||
| model = "gpt-3.5-turbo-0125" | ||
|
|
||
| def load(self) -> None: | ||
| self.markets_with_known_outcomes: dict[str, Result] = {} | ||
|
|
||
| def pick_markets(self, markets: list[AgentMarket]) -> list[AgentMarket]: | ||
| picked_markets: list[AgentMarket] = [] | ||
| for market in markets: | ||
| # Assume very high probability markets are already known, and have | ||
| # been correctly bet on, and therefore the value of betting on them | ||
| # is low. | ||
| if not market_is_saturated(market=markets[0]): | ||
| answer = get_known_outcome( | ||
| model=self.model, | ||
| question=market.question, | ||
| max_tries=3, | ||
| ) | ||
| if answer.has_known_outcome(): | ||
| picked_markets.append(market) | ||
| self.markets_with_known_outcomes[market.id] = answer.result | ||
|
|
||
| return picked_markets | ||
|
|
||
| def answer_binary_market(self, market: AgentMarket) -> bool: | ||
| # The answer has already been determined in `pick_markets` so we just | ||
| # return it here. | ||
| return self.markets_with_known_outcomes[market.id].to_boolean() | ||
|
|
||
| def calculate_bet_amount(self, answer: bool, market: AgentMarket) -> BetAmount: | ||
| if market.currency == Currency.xDai: | ||
| return BetAmount(amount=0.1, currency=Currency.xDai) | ||
| else: | ||
| raise NotImplementedError("This agent only supports xDai markets") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| agent = DeployableKnownOutcomeAgent() | ||
| github_repo_url = "https://github.com/gnosis/prediction-market-agent" | ||
| agent.deploy_gcp( | ||
| repository=f"git+{github_repo_url}.git@{get_current_git_commit_sha()}", | ||
| market_type=MarketType.OMEN, | ||
| labels={OWNER_KEY: getpass.getuser()}, | ||
| secrets={ | ||
| "OPENAI_API_KEY": "EVAN_OPENAI_API_KEY:latest", | ||
| "TAVILY_API_KEY": "GNOSIS_AI_TAVILY_API_KEY:latest", | ||
| "BET_FROM_PRIVATE_KEY": "0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307:latest", | ||
| }, | ||
| memory=1024, | ||
| api_keys=APIKeys( | ||
| BET_FROM_ADDRESS="0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307", | ||
| BET_FROM_PRIVATE_KEY=None, | ||
| OPENAI_API_KEY=None, | ||
| MANIFOLD_API_KEY=None, | ||
| ), | ||
| cron_schedule="0 */4 * * *", | ||
| timeout=540, | ||
| ) |
There was a problem hiding this comment.
The DeployableKnownOutcomeAgent class is well-structured and includes essential methods for deploying the agent. Ensure that the deployment script correctly handles any potential exceptions or errors during deployment to avoid partial or failed deployments.
try:
agent.deploy_gcp(
...
)
except Exception as e:
print(f"Deployment failed: {e}")Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
Suggested change
| class DeployableKnownOutcomeAgent(DeployableAgent): | |
| model = "gpt-3.5-turbo-0125" | |
| def load(self) -> None: | |
| self.markets_with_known_outcomes: dict[str, Result] = {} | |
| def pick_markets(self, markets: list[AgentMarket]) -> list[AgentMarket]: | |
| picked_markets: list[AgentMarket] = [] | |
| for market in markets: | |
| # Assume very high probability markets are already known, and have | |
| # been correctly bet on, and therefore the value of betting on them | |
| # is low. | |
| if not market_is_saturated(market=markets[0]): | |
| answer = get_known_outcome( | |
| model=self.model, | |
| question=market.question, | |
| max_tries=3, | |
| ) | |
| if answer.has_known_outcome(): | |
| picked_markets.append(market) | |
| self.markets_with_known_outcomes[market.id] = answer.result | |
| return picked_markets | |
| def answer_binary_market(self, market: AgentMarket) -> bool: | |
| # The answer has already been determined in `pick_markets` so we just | |
| # return it here. | |
| return self.markets_with_known_outcomes[market.id].to_boolean() | |
| def calculate_bet_amount(self, answer: bool, market: AgentMarket) -> BetAmount: | |
| if market.currency == Currency.xDai: | |
| return BetAmount(amount=0.1, currency=Currency.xDai) | |
| else: | |
| raise NotImplementedError("This agent only supports xDai markets") | |
| if __name__ == "__main__": | |
| agent = DeployableKnownOutcomeAgent() | |
| github_repo_url = "https://github.com/gnosis/prediction-market-agent" | |
| agent.deploy_gcp( | |
| repository=f"git+{github_repo_url}.git@{get_current_git_commit_sha()}", | |
| market_type=MarketType.OMEN, | |
| labels={OWNER_KEY: getpass.getuser()}, | |
| secrets={ | |
| "OPENAI_API_KEY": "EVAN_OPENAI_API_KEY:latest", | |
| "TAVILY_API_KEY": "GNOSIS_AI_TAVILY_API_KEY:latest", | |
| "BET_FROM_PRIVATE_KEY": "0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307:latest", | |
| }, | |
| memory=1024, | |
| api_keys=APIKeys( | |
| BET_FROM_ADDRESS="0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307", | |
| BET_FROM_PRIVATE_KEY=None, | |
| OPENAI_API_KEY=None, | |
| MANIFOLD_API_KEY=None, | |
| ), | |
| cron_schedule="0 */4 * * *", | |
| timeout=540, | |
| ) | |
| if __name__ == "__main__": | |
| agent = DeployableKnownOutcomeAgent() | |
| github_repo_url = "https://github.com/gnosis/prediction-market-agent" | |
| try: | |
| agent.deploy_gcp( | |
| repository=f"git+{github_repo_url}.git@{get_current_git_commit_sha()}", | |
| market_type=MarketType.OMEN, | |
| labels={OWNER_KEY: getpass.getuser()}, | |
| secrets={ | |
| "OPENAI_API_KEY": "EVAN_OPENAI_API_KEY:latest", | |
| "TAVILY_API_KEY": "GNOSIS_AI_TAVILY_API_KEY:latest", | |
| "BET_FROM_PRIVATE_KEY": "0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307:latest", | |
| }, | |
| memory=1024, | |
| api_keys=APIKeys( | |
| BET_FROM_ADDRESS="0x3666DA333dAdD05083FEf9FF6dDEe588d26E4307", | |
| BET_FROM_PRIVATE_KEY=None, | |
| OPENAI_API_KEY=None, | |
| MANIFOLD_API_KEY=None, | |
| ), | |
| cron_schedule="0 */4 * * *", | |
| timeout=540, | |
| ) | |
| except Exception as e: | |
| print(f"Deployment failed: {e}") |
Comment on lines
121
to
162
| def get_known_outcome(model: str, question: str, max_tries: int) -> Answer: | ||
| """ | ||
| In a loop, perform web search and scrape to find if the answer to the | ||
| question is known. Break if the answer is found, or after a certain number | ||
| of tries, and no definite answer is found, return an 'unknown' answer. | ||
| """ | ||
| tries = 0 | ||
| date_str = datetime.now().strftime("%d %B %Y") | ||
| previous_urls = [] | ||
| llm = ChatOpenAI(model="gpt-4-1106-preview", temperature=0.4) | ||
| while tries < max_tries: | ||
| search_prompt = ChatPromptTemplate.from_template( | ||
| template=GENERATE_SEARCH_QUERY_PROMPT | ||
| ).format_messages(date_str=date_str, question=question) | ||
| search_query = llm.invoke(search_prompt).content.strip('"') | ||
| search_results = web_search(query=search_query, max_results=5) | ||
| if not search_results: | ||
| raise ValueError("No search results found.") | ||
|
|
||
| for result in search_results: | ||
| if result.url in previous_urls: | ||
| continue | ||
| previous_urls.append(result.url) | ||
|
|
||
| scraped_content = web_scrape(url=result.url) | ||
|
|
||
| prompt = ChatPromptTemplate.from_template( | ||
| template=ANSWER_FROM_WEBSCRAPE_PROMPT | ||
| ).format_messages( | ||
| date_str=date_str, | ||
| question=question, | ||
| scraped_content=scraped_content, | ||
| ) | ||
| answer = llm.invoke(prompt).content | ||
| parsed_answer = Answer.model_validate(completion_str_to_json(answer)) | ||
|
|
||
| if parsed_answer.result is not Result.UNKNOWN: | ||
| return parsed_answer | ||
|
|
||
| tries += 1 | ||
|
|
||
| return Answer(result=Result.UNKNOWN, reasoning="Max tries exceeded.") |
There was a problem hiding this comment.
The get_known_outcome function correctly implements the logic for determining the known outcome of a market. Consider adding more detailed logging or error handling for better observability and robustness, especially in the web scraping and search parts.
+ import logging
...
try:
...
except Exception as e:
+ logging.error(f"Error while determining known outcome: {e}")
return Answer(result=Result.UNKNOWN, reasoning="Error occurred.")Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
Suggested change
| def get_known_outcome(model: str, question: str, max_tries: int) -> Answer: | |
| """ | |
| In a loop, perform web search and scrape to find if the answer to the | |
| question is known. Break if the answer is found, or after a certain number | |
| of tries, and no definite answer is found, return an 'unknown' answer. | |
| """ | |
| tries = 0 | |
| date_str = datetime.now().strftime("%d %B %Y") | |
| previous_urls = [] | |
| llm = ChatOpenAI(model="gpt-4-1106-preview", temperature=0.4) | |
| while tries < max_tries: | |
| search_prompt = ChatPromptTemplate.from_template( | |
| template=GENERATE_SEARCH_QUERY_PROMPT | |
| ).format_messages(date_str=date_str, question=question) | |
| search_query = llm.invoke(search_prompt).content.strip('"') | |
| search_results = web_search(query=search_query, max_results=5) | |
| if not search_results: | |
| raise ValueError("No search results found.") | |
| for result in search_results: | |
| if result.url in previous_urls: | |
| continue | |
| previous_urls.append(result.url) | |
| scraped_content = web_scrape(url=result.url) | |
| prompt = ChatPromptTemplate.from_template( | |
| template=ANSWER_FROM_WEBSCRAPE_PROMPT | |
| ).format_messages( | |
| date_str=date_str, | |
| question=question, | |
| scraped_content=scraped_content, | |
| ) | |
| answer = llm.invoke(prompt).content | |
| parsed_answer = Answer.model_validate(completion_str_to_json(answer)) | |
| if parsed_answer.result is not Result.UNKNOWN: | |
| return parsed_answer | |
| tries += 1 | |
| return Answer(result=Result.UNKNOWN, reasoning="Max tries exceeded.") | |
| import logging | |
| def get_known_outcome(model: str, question: str, max_tries: int) -> Answer: | |
| """ | |
| In a loop, perform web search and scrape to find if the answer to the | |
| question is known. Break if the answer is found, or after a certain number | |
| of tries, and no definite answer is found, return an 'unknown' answer. | |
| """ | |
| tries = 0 | |
| date_str = datetime.now().strftime("%d %B %Y") | |
| previous_urls = [] | |
| llm = ChatOpenAI(model="gpt-4-1106-preview", temperature=0.4) | |
| while tries < max_tries: | |
| search_prompt = ChatPromptTemplate.from_template( | |
| template=GENERATE_SEARCH_QUERY_PROMPT | |
| ).format_messages(date_str=date_str, question=question) | |
| search_query = llm.invoke(search_prompt).content.strip('"') | |
| search_results = web_search(query=search_query, max_results=5) | |
| if not search_results: | |
| raise ValueError("No search results found.") | |
| for result in search_results: | |
| if result.url in previous_urls: | |
| continue | |
| previous_urls.append(result.url) | |
| scraped_content = web_scrape(url=result.url) | |
| prompt = ChatPromptTemplate.from_template( | |
| template=ANSWER_FROM_WEBSCRAPE_PROMPT | |
| ).format_messages( | |
| date_str=date_str, | |
| question=question, | |
| scraped_content=scraped_content, | |
| ) | |
| answer = llm.invoke(prompt).content | |
| parsed_answer = Answer.model_validate(completion_str_to_json(answer)) | |
| if parsed_answer.result is not Result.UNKNOWN: | |
| return parsed_answer | |
| tries += 1 | |
| return Answer(result=Result.UNKNOWN, reasoning="Max tries exceeded.") |
Comment on lines
75
to
163
| if __name__ == "__main__": | ||
| load_dotenv() | ||
| tomorrow_str = (datetime.now(tz=pytz.UTC) + timedelta(days=1)).strftime("%d %B %Y") | ||
|
|
||
| # Fetch questions from existing markets, or make some up, where the | ||
| # outcome is known. | ||
| qs_with_known_outcome: list[QuestionWithKnownOutcome] = [ | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will 'Barbie' win an Academy Award for best original song by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xceb2a4ecc217cab440acf60737a9fcfd6d3fbf4b", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will the 2024 Oscars winner for Best Picture be announced by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xb88e4507709148e096bcdfb861b17db7b4d54e6b", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Liverpool win against Atalanta in the Europa League quarter-finals by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x1d5a462c801360b4bebbda2b9656e52801a27a3b", | ||
| result=Result.NO, | ||
| notes="The match is scheduled for 11 April 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Donald Trump officially become the GOP nominee for the 2024 presidential elections by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x859a6b465ee1e4a73aab0f2da4428c6255da466c", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will SpaceX successfully test a Starship reentry without losing contact by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xcc9123af8db309e0c60c63f9e2b8b82fc86f458b", | ||
| result=Result.NO, | ||
| notes="The only scheduled test flight occured, and contact was lost during the test.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Arsenal reach the Champions League semi-finals on {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x606efd175b245cd60282a98cef402d4f5e950f92", | ||
| result=Result.NO, | ||
| notes="They are scheduled to play the first leg of the quarter-finals on 9 April 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will the jury deliver a verdict on James Crumbley's 'bad parenting' case on {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xe55171beda0d60fd45092ff8bf93d5cb566a2510", | ||
| result=Result.NO, | ||
| notes="The verdict was announced on 15th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will Lewis Hamilton win the 2024/2025 F1 drivers champtionship?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will the cost of grain in the Spain increase by 20% by 19 July 2024?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will over 360 pople have died while climbing Mount Everest by 1st Jan 2028?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| ] | ||
|
|
||
| benchmarker = Benchmarker( | ||
| markets=[q.to_market() for q in qs_with_known_outcome], | ||
| agents=[ | ||
| KnownOutcomeAgent( | ||
| agent_name="known_outcome", | ||
| model="gpt-3.5-turbo-0125", | ||
| max_tries=3, | ||
| max_workers=1, | ||
| ), | ||
| ], | ||
| cache_path="./benchmark_cache.json", | ||
| ) | ||
| benchmarker.run_agents() | ||
| md = benchmarker.generate_markdown_report() | ||
|
|
||
| output = f"./known_outcome_agent_benchmark_report.{int(time.time())}.md" | ||
| with open(output, "w") as f: | ||
| print(f"Writing benchmark report to: {output}") | ||
| f.write(md) | ||
|
|
||
| # Check all predictions are correct, i.e. mean-squared-error == 0 | ||
| metrics = benchmarker.compute_metrics() | ||
| assert metrics["MSE for `p_yes`"][0] == 0.0 |
There was a problem hiding this comment.
The benchmarking script is well-structured and includes a comprehensive list of test cases. Ensure that the benchmarking process includes error handling for cases where the agent might fail to predict an outcome, to provide a complete picture of the agent's performance.
try:
benchmarker.run_agents()
metrics = benchmarker.compute_metrics()
assert metrics["MSE for `p_yes`"][0] == 0.0
except Exception as e:
print(f"Benchmarking failed: {e}")Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
Suggested change
| if __name__ == "__main__": | |
| load_dotenv() | |
| tomorrow_str = (datetime.now(tz=pytz.UTC) + timedelta(days=1)).strftime("%d %B %Y") | |
| # Fetch questions from existing markets, or make some up, where the | |
| # outcome is known. | |
| qs_with_known_outcome: list[QuestionWithKnownOutcome] = [ | |
| QuestionWithKnownOutcome( | |
| question=f"Will 'Barbie' win an Academy Award for best original song by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xceb2a4ecc217cab440acf60737a9fcfd6d3fbf4b", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will the 2024 Oscars winner for Best Picture be announced by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xb88e4507709148e096bcdfb861b17db7b4d54e6b", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Liverpool win against Atalanta in the Europa League quarter-finals by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x1d5a462c801360b4bebbda2b9656e52801a27a3b", | |
| result=Result.NO, | |
| notes="The match is scheduled for 11 April 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Donald Trump officially become the GOP nominee for the 2024 presidential elections by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x859a6b465ee1e4a73aab0f2da4428c6255da466c", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will SpaceX successfully test a Starship reentry without losing contact by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xcc9123af8db309e0c60c63f9e2b8b82fc86f458b", | |
| result=Result.NO, | |
| notes="The only scheduled test flight occured, and contact was lost during the test.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Arsenal reach the Champions League semi-finals on {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x606efd175b245cd60282a98cef402d4f5e950f92", | |
| result=Result.NO, | |
| notes="They are scheduled to play the first leg of the quarter-finals on 9 April 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will the jury deliver a verdict on James Crumbley's 'bad parenting' case on {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xe55171beda0d60fd45092ff8bf93d5cb566a2510", | |
| result=Result.NO, | |
| notes="The verdict was announced on 15th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will Lewis Hamilton win the 2024/2025 F1 drivers champtionship?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will the cost of grain in the Spain increase by 20% by 19 July 2024?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will over 360 pople have died while climbing Mount Everest by 1st Jan 2028?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| ] | |
| benchmarker = Benchmarker( | |
| markets=[q.to_market() for q in qs_with_known_outcome], | |
| agents=[ | |
| KnownOutcomeAgent( | |
| agent_name="known_outcome", | |
| model="gpt-3.5-turbo-0125", | |
| max_tries=3, | |
| max_workers=1, | |
| ), | |
| ], | |
| cache_path="./benchmark_cache.json", | |
| ) | |
| benchmarker.run_agents() | |
| md = benchmarker.generate_markdown_report() | |
| output = f"./known_outcome_agent_benchmark_report.{int(time.time())}.md" | |
| with open(output, "w") as f: | |
| print(f"Writing benchmark report to: {output}") | |
| f.write(md) | |
| # Check all predictions are correct, i.e. mean-squared-error == 0 | |
| metrics = benchmarker.compute_metrics() | |
| assert metrics["MSE for `p_yes`"][0] == 0.0 | |
| if __name__ == "__main__": | |
| load_dotenv() | |
| tomorrow_str = (datetime.now(tz=pytz.UTC) + timedelta(days=1)).strftime("%d %B %Y") | |
| # Fetch questions from existing markets, or make some up, where the | |
| # outcome is known. | |
| qs_with_known_outcome: list[QuestionWithKnownOutcome] = [ | |
| QuestionWithKnownOutcome( | |
| question=f"Will 'Barbie' win an Academy Award for best original song by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xceb2a4ecc217cab440acf60737a9fcfd6d3fbf4b", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will the 2024 Oscars winner for Best Picture be announced by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xb88e4507709148e096bcdfb861b17db7b4d54e6b", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Liverpool win against Atalanta in the Europa League quarter-finals by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x1d5a462c801360b4bebbda2b9656e52801a27a3b", | |
| result=Result.NO, | |
| notes="The match is scheduled for 11 April 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Donald Trump officially become the GOP nominee for the 2024 presidential elections by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x859a6b465ee1e4a73aab0f2da4428c6255da466c", | |
| result=Result.YES, | |
| notes="Happened on 10th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will SpaceX successfully test a Starship reentry without losing contact by {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xcc9123af8db309e0c60c63f9e2b8b82fc86f458b", | |
| result=Result.NO, | |
| notes="The only scheduled test flight occured, and contact was lost during the test.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will Arsenal reach the Champions League semi-finals on {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0x606efd175b245cd60282a98cef402d4f5e950f92", | |
| result=Result.NO, | |
| notes="They are scheduled to play the first leg of the quarter-finals on 9 April 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question=f"Will the jury deliver a verdict on James Crumbley's 'bad parenting' case on {tomorrow_str}?", | |
| url="https://aiomen.eth.limo/#/0xe55171beda0d60fd45092ff8bf93d5cb566a2510", | |
| result=Result.NO, | |
| notes="The verdict was announced on 15th March 2024.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will Lewis Hamilton win the 2024/2025 F1 drivers champtionship?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will the cost of grain in the Spain increase by 20% by 19 July 2024?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| QuestionWithKnownOutcome( | |
| question="Will over 360 pople have died while climbing Mount Everest by 1st Jan 2028?", | |
| result=Result.UNKNOWN, | |
| notes="Outcome is uncertain.", | |
| ), | |
| ] | |
| benchmarker = Benchmarker( | |
| markets=[q.to_market() for q in qs_with_known_outcome], | |
| agents=[ | |
| KnownOutcomeAgent( | |
| agent_name="known_outcome", | |
| model="gpt-3.5-turbo-0125", | |
| max_tries=3, | |
| max_workers=1, | |
| ), | |
| ], | |
| cache_path="./benchmark_cache.json", | |
| ) | |
| try: | |
| benchmarker.run_agents() | |
| metrics = benchmarker.compute_metrics() | |
| assert metrics["MSE for `p_yes`"][0] == 0.0 | |
| except Exception as e: | |
| print(f"Benchmarking failed: {e}") | |
| md = benchmarker.generate_markdown_report() | |
| output = f"./known_outcome_agent_benchmark_report.{int(time.time())}.md" | |
| with open(output, "w") as f: | |
| print(f"Writing benchmark report to: {output}") | |
| f.write(md) |
There was a problem hiding this comment.
That's a bold assertion! Cool!
There was a problem hiding this comment.
@kongzii, thank you for the enthusiasm! It's great to see positive feedback. If you have any more suggestions or need further assistance, feel free to ask. 😊
evangriffiths
commented
Mar 20, 2024
| from prediction_market_agent_tooling.benchmark.agents import AbstractBenchmarkedAgent | ||
| from prediction_market_agent_tooling.benchmark.benchmark import Benchmarker | ||
| from prediction_market_agent_tooling.benchmark.utils import ( | ||
| Market, |
There was a problem hiding this comment.
Will release a new version of prediction-market-agent-tooling, and bump it in this repo and update this code. Happy to do as part of this PR or as a follow-up PR.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (5)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py (1 hunks)
- prediction_market_agent/tools/web_scrape/structured_summary.py (1 hunks)
- prediction_market_agent/tools/web_search/tavily.py (1 hunks)
Files skipped from review due to trivial changes (1)
- prediction_market_agent/tools/web_scrape/structured_summary.py
Files skipped from review as they are similar to previous changes (4)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py
- prediction_market_agent/agents/known_outcome_agent/deploy.py
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
- prediction_market_agent/tools/web_search/tavily.py
|
Benchmarking output: |
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (1)
- prediction_market_agent/agents/known_outcome_agent/deploy.py (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- prediction_market_agent/agents/known_outcome_agent/deploy.py
gabrielfior
reviewed
Mar 20, 2024
|
|
||
| def calculate_bet_amount(self, answer: bool, market: AgentMarket) -> BetAmount: | ||
| if market.currency == Currency.xDai: | ||
| return BetAmount(amount=Decimal(0.1), currency=Currency.xDai) |
There was a problem hiding this comment.
Should we make this configurable (e.g. ENV from Google Cloud) and thus editable after deployment?
There was a problem hiding this comment.
We currently have a way of changing agent source code after deployment via gcp, so I don't think that is needed. ATM we can update the commit hash that the agent's container installs (see pic), and click 'save and redeploy'
gabrielfior
reviewed
Mar 20, 2024
| scraped_content=scraped_content, | ||
| ) | ||
| answer = str(llm.invoke(prompt).content) | ||
| parsed_answer = Answer.model_validate(completion_str_to_json(answer)) |
There was a problem hiding this comment.
Isn't it a bit risky to assume the LLM will answer using the correct format?
I would expect some try-catch block because of this, but I might be wrong.
There was a problem hiding this comment.
I'd say it depends on what comes after this.
Is this needed (if so let's error out and do it more robust in another PR) or can we skip it and continue ?
There was a problem hiding this comment.
gpt4 is very reliable at producing json output!
kongzii
reviewed
Mar 20, 2024
prediction_market_agent/agents/known_outcome_agent/benchmark.py
Outdated
Show resolved
Hide resolved
prediction_market_agent/agents/known_outcome_agent/benchmark.py
Outdated
Show resolved
Hide resolved
prediction_market_agent/agents/known_outcome_agent/benchmark.py
Outdated
Show resolved
Hide resolved
Comment on lines
75
to
163
| if __name__ == "__main__": | ||
| load_dotenv() | ||
| tomorrow_str = (datetime.now(tz=pytz.UTC) + timedelta(days=1)).strftime("%d %B %Y") | ||
|
|
||
| # Fetch questions from existing markets, or make some up, where the | ||
| # outcome is known. | ||
| qs_with_known_outcome: list[QuestionWithKnownOutcome] = [ | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will 'Barbie' win an Academy Award for best original song by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xceb2a4ecc217cab440acf60737a9fcfd6d3fbf4b", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will the 2024 Oscars winner for Best Picture be announced by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xb88e4507709148e096bcdfb861b17db7b4d54e6b", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Liverpool win against Atalanta in the Europa League quarter-finals by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x1d5a462c801360b4bebbda2b9656e52801a27a3b", | ||
| result=Result.NO, | ||
| notes="The match is scheduled for 11 April 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Donald Trump officially become the GOP nominee for the 2024 presidential elections by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x859a6b465ee1e4a73aab0f2da4428c6255da466c", | ||
| result=Result.YES, | ||
| notes="Happened on 10th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will SpaceX successfully test a Starship reentry without losing contact by {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xcc9123af8db309e0c60c63f9e2b8b82fc86f458b", | ||
| result=Result.NO, | ||
| notes="The only scheduled test flight occured, and contact was lost during the test.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will Arsenal reach the Champions League semi-finals on {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0x606efd175b245cd60282a98cef402d4f5e950f92", | ||
| result=Result.NO, | ||
| notes="They are scheduled to play the first leg of the quarter-finals on 9 April 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question=f"Will the jury deliver a verdict on James Crumbley's 'bad parenting' case on {tomorrow_str}?", | ||
| url="https://aiomen.eth.limo/#/0xe55171beda0d60fd45092ff8bf93d5cb566a2510", | ||
| result=Result.NO, | ||
| notes="The verdict was announced on 15th March 2024.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will Lewis Hamilton win the 2024/2025 F1 drivers champtionship?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will the cost of grain in the Spain increase by 20% by 19 July 2024?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| QuestionWithKnownOutcome( | ||
| question="Will over 360 pople have died while climbing Mount Everest by 1st Jan 2028?", | ||
| result=Result.UNKNOWN, | ||
| notes="Outcome is uncertain.", | ||
| ), | ||
| ] | ||
|
|
||
| benchmarker = Benchmarker( | ||
| markets=[q.to_market() for q in qs_with_known_outcome], | ||
| agents=[ | ||
| KnownOutcomeAgent( | ||
| agent_name="known_outcome", | ||
| model="gpt-3.5-turbo-0125", | ||
| max_tries=3, | ||
| max_workers=1, | ||
| ), | ||
| ], | ||
| cache_path="./benchmark_cache.json", | ||
| ) | ||
| benchmarker.run_agents() | ||
| md = benchmarker.generate_markdown_report() | ||
|
|
||
| output = f"./known_outcome_agent_benchmark_report.{int(time.time())}.md" | ||
| with open(output, "w") as f: | ||
| print(f"Writing benchmark report to: {output}") | ||
| f.write(md) | ||
|
|
||
| # Check all predictions are correct, i.e. mean-squared-error == 0 | ||
| metrics = benchmarker.compute_metrics() | ||
| assert metrics["MSE for `p_yes`"][0] == 0.0 |
There was a problem hiding this comment.
That's a bold assertion! Cool!
| BET_FROM_ADDRESS=verify_address( | ||
| "0xb611A9f02B318339049264c7a66ac3401281cc3c" | ||
| ), | ||
| BET_FROM_PRIVATE_KEY=None, |
There was a problem hiding this comment.
BET_FROM_PRIVATE_KEY should (can) be defined here instead of manually in secrets. Deployment will automatically treat it like a secret thanks to:
There was a problem hiding this comment.
Ah nice. It feels a bit counter intuitive that we have 2 places where private keys can be passed in. If we wanted to make it so all keys are specified in one place (assuming APIKeys can never contain all key-types an agent might want), then the way to do this would be to remove APIKeys altogether. Got any opinions on that?
There was a problem hiding this comment.
then the way to do this would be to remove APIKeys altogether. Got any opinions on that?
Yep, I believe we have GitHub issue for that 😄 Looking forward to someone picking it up.
My current reasoning for using APIKeys here is that we use them heavily across the tooling repo, so APIKeys needs to be enforced here to force the right variable names. And then add to secrets anything agent-specific (tavily for example)
prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
Outdated
Show resolved
Hide resolved
prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
Outdated
Show resolved
Hide resolved
prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (4)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py (1 hunks)
- prediction_market_agent/tools/web_scrape/basic_summary.py (1 hunks)
Files skipped from review as they are similar to previous changes (3)
- prediction_market_agent/agents/known_outcome_agent/benchmark.py
- prediction_market_agent/agents/known_outcome_agent/deploy.py
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
Additional comments: 2
prediction_market_agent/tools/web_scrape/basic_summary.py (2)
- 9-14: The addition of the
separatorsparameter with a default value enhances the flexibility of the_summaryfunction, allowing for customization of text splitting based on the content structure. This is a positive change for handling different types of content more effectively.- 12-12: Updating the
ChatOpenAImodel parameter to "gpt-3.5-turbo-0125" from a previous version indicates an upgrade to a potentially more efficient or accurate model. It's important to verify that this new model version is compatible with the existing codebase and that it meets the performance and accuracy requirements for the task at hand.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (2)
- prediction_market_agent/agents/known_outcome_agent/deploy.py (1 hunks)
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py (1 hunks)
Files skipped from review as they are similar to previous changes (2)
- prediction_market_agent/agents/known_outcome_agent/deploy.py
- prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
kongzii
approved these changes
Mar 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This agent is designed to target markets that are still open, but the outcome is easily knowable with a high degree of certainty.
e.g. https://aiomen.eth.limo/#/0x859a6b465ee1e4a73aab0f2da4428c6255da466c
At the time of writing (20/03/2024), this market was open, but the answer was already known to be 'yes'.
The agent works as follows:
Also added scripts for benchmarking (see results in comments) and deploying the agent.
Also did some re-org of the agent tools. I've put tools with the same functionality into sub-folders (web search, web scraping).
Summary by CodeRabbit