Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump PMAT version to 0.9.3, simplify calculate_bet_amount for KnownOutcomeAgent #51

Merged
merged 4 commits into from
Apr 8, 2024
Merged

Bump PMAT version to 0.9.3, simplify calculate_bet_amount for KnownOutcomeAgent #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3006a31
Bump PMAT version to 0.9.3, simplify
evangriffiths Apr 5, 2024
bc864ec
Update poetry.lock
evangriffiths Apr 5, 2024
38bbeba
Skip markets with small liquidity
evangriffiths Apr 8, 2024
513de02
Oops
evangriffiths Apr 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 12 additions & 14 deletions prediction_market_agent/agents/known_outcome_agent/deploy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,13 @@
from prediction_market_agent_tooling.deploy.constants import OWNER_KEY
from prediction_market_agent_tooling.gtypes import SecretStr, private_key_type
from prediction_market_agent_tooling.markets.agent_market import AgentMarket
from prediction_market_agent_tooling.markets.data_models import BetAmount, Currency
from prediction_market_agent_tooling.markets.data_models import BetAmount
from prediction_market_agent_tooling.markets.markets import MarketType
from prediction_market_agent_tooling.markets.omen.omen import OmenAgentMarket
from prediction_market_agent_tooling.tools.utils import (
check_not_none,
get_current_git_commit_sha,
get_current_git_url,
should_not_happen,
)

from prediction_market_agent.agents.known_outcome_agent.known_outcome_agent import (
Expand All @@ -28,14 +27,19 @@ def market_is_saturated(market: AgentMarket) -> bool:


class DeployableKnownOutcomeAgent(DeployableAgent):
model = "gpt-4-1106-preview"
model = "gpt-4-turbo-preview"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed this as well, this can point to a different model in future without notice, is that desirable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot by Dropbox Capture

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm good question. Maybe in general not, but I reckon for this case it's good - my reasoning being that otherwise over time the agent will get less and less 'up to date with current events' which I would assume is bad for making future predictions.


def load(self) -> None:
self.markets_with_known_outcomes: dict[str, Result] = {}

def pick_markets(self, markets: list[AgentMarket]) -> list[AgentMarket]:
picked_markets: list[AgentMarket] = []
for market in markets:
if not isinstance(market, OmenAgentMarket):
raise NotImplementedError(
"This agent only supports predictions on Omen markets"
)

print(f"Looking at market {market.id=} {market.question=}")

# Assume very high probability markets are already known, and have
Expand All @@ -45,6 +49,10 @@ def pick_markets(self, markets: list[AgentMarket]) -> list[AgentMarket]:
print(
f"Skipping market {market.id=} {market.question=}, because it is already saturated."
)
elif market.get_liquidity_in_xdai() < 5:
print(
f"Skipping market {market.id=} {market.question=}, because it has insufficient liquidity."
)
else:
picked_markets.append(market)

Expand Down Expand Up @@ -83,17 +91,7 @@ def answer_binary_market(self, market: AgentMarket) -> bool | None:

def calculate_bet_amount(self, answer: bool, market: AgentMarket) -> BetAmount:
if isinstance(market, OmenAgentMarket):
if market.currency != Currency.xDai:
should_not_happen()
return BetAmount(
# On markets without liquidity, bet just a small amount for benchmarking.
amount=(
Decimal(1.0)
if market.get_liquidity_in_xdai() > 5
else market.get_tiny_bet_amount().amount
),
currency=Currency.xDai,
)
return BetAmount(amount=(Decimal(1.0)), currency=market.currency)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know we now filtering for markets with liquidity, but people can still create markets with for example, $0.01 in liquidity and then it doesn't make much sense to me to bet more than the minimal bet. wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. In that case, I think it makes sense to filter the market at the pick_markets stage, so as not to waste openai API credits. I've made that change above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you want to keep them for benchmark purposes?

If not, okay by me, just that

elif market.get_liquidity_in_xdai() > 5:
                print(
                    f"Skipping market {market.id=} {market.question=}, because it has insufficient liquidity."
                )

Should it be with the opposite sign, probably?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! 😆

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you want to keep them for benchmark purposes?

I'm not too concerned about that, more about trying to get the markets that should be p_yes/no==0.95 to that point

else:
raise NotImplementedError("This agent only supports xDai markets")

Expand Down
2 changes: 1 addition & 1 deletion prediction_market_agent/agents/known_outcome_agent/known_outcome_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ def get_known_outcome(model: str, question: str, max_tries: int) -> Answer:
).format_messages(date_str=date_str, question=question)
print(f"Invoking LLM for {search_prompt=}")
search_query = str(llm.invoke(search_prompt).content).strip('"')
print(f"Searchig for {search_query=}")
print(f"Searching for {search_query=}")
search_results = web_search(query=search_query, max_results=5)
if not search_results:
raise ValueError("No search results found.")
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ poetry = "^1.7.1"
poetry-plugin-export = "^1.6.0"
functions-framework = "^3.5.0"
cron-validator = "^1.0.8"
prediction-market-agent-tooling = "^0.9.1"
prediction-market-agent-tooling = "^0.9.3"
pydantic-settings = "^2.1.0"
autoflake = "^2.2.1"
isort = "^5.13.2"
Expand Down
Loading