Index all Twitter Agents in Subnet 42 & Iterate on Scoring

[Luka-Loncar]

Overview

Create a comprehensive dataset of all X Agents (automated accounts) in subnet 42, utilizing existing scraped data from Virtuals and Creator Bid applications. This will enable pre-scoring and forecast earnings directly in the UI.

Background

• Current system does not index all X.com Agents
• Existing scraped data available from @hide on bush includes Agent accounts labeled as automated on X (he scraped them from Virtuals and Creator Bid)
• Opportunity to create a unified mega dataset for better analysis and forecasting

Requirements

1. Integrate scraped data from Virtuals and Creator Bid apps
2. Index all accounts marked with "automated accounts" label on X.com
3. Implement pre-scoring system in the UI
4. Add forecast earnings functionality

Technical Considerations

• Need to ensure efficient indexing of large dataset
• Consider storage requirements for comprehensive agent data
• Implement proper data validation for scraped information
• Design scalable pre-scoring system

Expected Outcomes

• Complete database of all x.com Agents
• Pre-scored agents visible in UI
• Forecast earnings calculations available
• Improved data accessibility for subnet 42

[grantdfoster]

There are some pieces of this ticket that may no longer be relevant, i.e. "Implement pre-scoring system in the UI". The features described in this ticket will be ported to the TEE indexer in the new SN42 architecture, i.e. the component that will determine the queries / posts to scrape. The current list of agents is housed in our miner_profiles table in the protocol API, and the TEE indexer will likely pull from this list. To support scraping many more agents as described in this ticket, we can potentially create a new table to house data on said agents