Lock players from seeding when in a game. #311
Conversation
|
Thanks for taking this on, @Janzert. I'm not sure what the clause 'WHERE maxTime < DATE_SUB(NOW(), INTERVAL 10 MINUTE)' does exactly, but I'm guessing it locks players out from a new game for 10 minutes? I would recommend something a lot lower, like 1 minute, or perhaps 2 minutes at most. When users push new botcode, the feedback from the first n games is hotly anticipated. I think it's OK to have a little duplication, if that's what would result from setting a smaller window. From a player perspective, I think ppl would rather have the way it is now, than to be rate-limited to 6 games in first hour of submission (oh, I don't even like typing that, please don't do it! ...) On my most recent bot, I'm pretty sure I got to #1 after ~80 minutes (I forget how many games that was), which was incredible, and much better than before. It's just that the first 20 games against the same players seemed like a waste, bc TrueSkill learned nothing from the latter 19. I think if players can get to stabilized ranking after 60-90 minutes, that's a pretty happy place -- and it seems like we're there now. With real minor rate limiting like I'm suggesting, I think leaderboard climbing can happen as fast if not faster, and the servers can put those cycles to better use for other players. I also want to be sure we avoid any "parity bit" errors, where my bot never plays your bot because I'm locked out when you're looking for a match and vice versa. I suppose that limiting this to high sigma players would largely avoid that. I will also add, though, that I noticed that the "spurt of games" behavior also happens with me as a top bot. I think it's because I'm often in the pool of players that haven't had game in long time, so I get chosen as seed from that pool a few times, before the initial game results are back to remove me from that pool. |
|
We are very interested in adding something like this. Thanks for the PR. One benefit other than the rate limiting is the ability to track what players are running on each worker. If a few workers get downed, we can easily check to see whether there is a common, renegade player between them. |
|
From @erdman:
This is exactly right. |
Have you had trouble setting up the website/manager locally? |
|
The 10 minute timeout is just a failsafe in case the worker that takes the game doesn't report a result back for some reason. That should happen extremely rarely, the normal case is that the player will be eligible for seeding as soon as the current game finishes. It is quite hard to say what the effect on clock time to rank stabilization will be. The number of games until appropriate opponents are being played will certainly be much, much reduced. It will probably be 20 or less even when going all the way to the extreme ends of the rankings, depending on number of players in the games and the random opponents selected. Yes, this system is and should only be used to lock out seeding a player not for getting pulled into a game. And yes the current state of patch doesn't fix the least recent played seeding. The other two seeding queries can also be modified to fix that as well. I go back and forth on whether that will increase or decrease the number of games you see as a top player that has separated out from the nearest contenders. I currently think that it will be close to a wash but that you'll see better opponents on average. |
Yep, it had even crossed my mind to add something along those lines to the status page. |
I haven't actually tried it yet, was digging through earlier to figure how. The s3 bucket storage for replays is one trouble point I'm needing to look at closer. |
|
This seems to be working with local testing now. One important note when adding this to an existing setup, the database changes must be made prior to updating the website and worker. Otherwise seeding and game tasks will break. I also decided to only change the high sigma (i.e. new user) seeding for the moment since that is the largest problem point and the others can be changed fairly easily once this is up and working. |
| @@ -65,6 +65,13 @@ private function getTrueskillMatchQuality($rankingValues) { | |||
| return floatval($lines[0]); | |||
| } | |||
|
|
|||
| private function clearPairing() { | |||
There was a problem hiding this comment.
Why not pass $paringID as a parameter to clearParing instead of relying on its being a POST parameter?
There was a problem hiding this comment.
Yeah, that makes sense. It just got factored out into a separate function just because the game api function can exit (return) from two code paths, each of which need to clear the pairing.
| DROP TABLE IF EXISTS `Pairing`; | ||
| /*!40101 SET @saved_cs_client = @@character_set_client */; | ||
| /*!40101 SET character_set_client = utf8 */; | ||
| CREATE TABLE `Pairing` ( |
There was a problem hiding this comment.
Paring is a bit vague. Might want to change this to GameParing.
There was a problem hiding this comment.
A Pairing row doesn't really relate to a Game row in the database so that may be confusing. But it could be changed.
There was a problem hiding this comment.
Discord discussion settled on GameTask. Not sure on ID fields, going with gametaskID for the moment.
|
Will test and merge. |
|
My latest bot (v12) has been an interesting test case ... with the workers slowed, my games were dribbled out serially (as if I was rate-limited), and so my progress up leaderboard was much more dramatic (as measured at rate of per-game, not per-hour!). I reached top 10 after just 15 games. As result, I'm thinking that we can also solve the "two distributions" problem here as well. If rate-limiting allows new submissions to reach their natural ability level much more quickly, there is no need to have an extended period as a "seed" (which I think is currently set at ~400 games). I think we can lower that 400 game level quite a bit, to 50 games, or even 30 games. As a result, most bots will be spending most of their lives as non-seeds, and the differing distribution you face as a seed is much less of an issue. |
|
I would worry about your getting the same game rate as some RandomBot with 5000+ games. Seems inefficient. |
Perhaps there is a way to decrease the frequency of games for low mu players with a lot of games? |
This is to fix issue #310.
Currently it is untested (so probably broken) since I do not have a local setup. Also it is only currently locking out high sigma seeding, the other seeding can be locked similarly by updating their queries. The high sigma seeding is the biggest problem though.
I'm putting this up now to make sure that the organizer's are at least in principle interested in changing this and specifically using this approach.