Add number of markets and resolution stats to the benchmark output #27
Conversation
|
Warning Rate Limit Exceeded@kongzii has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 17 minutes and 31 seconds before requesting another review. How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. WalkthroughThe recent update introduces a new feature to the prediction market agent tooling by adding a method for retrieving comprehensive information about the markets. This includes data on the total number of markets, as well as the proportions of markets that have been resolved, and those that have ended in YES or NO outcomes. Additionally, the report generation function has been enhanced to incorporate this overall market information, providing a more detailed overview in the generated reports. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Status
Actionable comments generated: 1
Configuration used: CodeRabbit UI
Files selected for processing (1)
- prediction_market_agent_tooling/benchmark/benchmark.py (2 hunks)
Additional comments: 1
prediction_market_agent_tooling/benchmark/benchmark.py (1)
- 525-527: The update to
generate_markdown_reportmethod to include overall market information usingpd.DataFrame(self.get_markets_overall_info()).to_markdown(index=False)is correctly implemented. This change effectively integrates the new market information into the benchmark report, enhancing its utility and informativeness. No issues or improvements are identified in this specific change.
| def get_markets_overall_info(self) -> dict[str, list[str | float]]: | ||
| return { | ||
| "Number of markets": [len(self.markets)], | ||
| "Proportion resolved": [ | ||
| sum(1 for m in self.markets if m.is_resolved) / len(self.markets) | ||
| ], | ||
| "Proportion YES": [ | ||
| sum( | ||
| 1 | ||
| for m in self.markets | ||
| if m.probable_resolution == MarketResolution.YES | ||
| ) | ||
| / len(self.markets) | ||
| ], | ||
| "Proportion NO": [ | ||
| sum( | ||
| 1 | ||
| for m in self.markets | ||
| if m.probable_resolution == MarketResolution.NO | ||
| ) | ||
| / len(self.markets) | ||
| ], | ||
| } |
There was a problem hiding this comment.
The implementation of get_markets_overall_info method correctly calculates the overall market information, including the number of markets and their resolution proportions. However, consider using a more efficient approach for calculating proportions to improve readability and potentially performance. For example, you could calculate the total number of markets once and reuse it for each proportion calculation instead of calling len(self.markets) multiple times.
- sum(1 for m in self.markets if m.is_resolved) / len(self.markets)
+ resolved_markets = sum(1 for m in self.markets if m.is_resolved)
+ total_markets = len(self.markets)
+ "Proportion resolved": [resolved_markets / total_markets],Apply similar changes for "Proportion YES" and "Proportion NO".
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
| def get_markets_overall_info(self) -> dict[str, list[str | float]]: | |
| return { | |
| "Number of markets": [len(self.markets)], | |
| "Proportion resolved": [ | |
| sum(1 for m in self.markets if m.is_resolved) / len(self.markets) | |
| ], | |
| "Proportion YES": [ | |
| sum( | |
| 1 | |
| for m in self.markets | |
| if m.probable_resolution == MarketResolution.YES | |
| ) | |
| / len(self.markets) | |
| ], | |
| "Proportion NO": [ | |
| sum( | |
| 1 | |
| for m in self.markets | |
| if m.probable_resolution == MarketResolution.NO | |
| ) | |
| / len(self.markets) | |
| ], | |
| } | |
| def get_markets_overall_info(self) -> dict[str, list[str | float]]: | |
| resolved_markets = sum(1 for m in self.markets if m.is_resolved) | |
| total_markets = len(self.markets) | |
| return { | |
| "Number of markets": [total_markets], | |
| "Proportion resolved": [resolved_markets / total_markets], | |
| "Proportion YES": [ | |
| sum( | |
| 1 | |
| for m in self.markets | |
| if m.probable_resolution == MarketResolution.YES | |
| ) | |
| / total_markets | |
| ], | |
| "Proportion NO": [ | |
| sum( | |
| 1 | |
| for m in self.markets | |
| if m.probable_resolution == MarketResolution.NO | |
| ) | |
| / total_markets | |
| ], | |
| } |
| @@ -498,6 +522,9 @@ def compute_expected_returns_summary( | |||
|
|
|||
| def generate_markdown_report(self) -> str: | |||
| md = "# Comparison Report\n\n" | |||
| md += "## Overall Info\n\n" | |||
There was a problem hiding this comment.
Nit, but could rename Overall Info to Market Results, and adjust the heading structure below so it's
## Market Results
...
## Agent Results
### Summary Statistics
### Markets
### Expected value
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (1)
- prediction_market_agent_tooling/benchmark/benchmark.py (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- prediction_market_agent_tooling/benchmark/benchmark.py
Summary by CodeRabbit