Skip to content

Latest commit

 

History

History
128 lines (99 loc) · 5.49 KB

Models.md

File metadata and controls

128 lines (99 loc) · 5.49 KB

Models

Specific in-game data

Detects: 79 brawlers + 750 skins + each skin/brawler abilities + 5 types of balls + gems + power cubes + poison clouds + walls + bushes
Reliability: 5.5/10
Maintenance: Very high

model_showcase.mp4

How was it trained:
The assets were 107,395 images And the dataset was 100k images.

Why so much?:
Let's look at a brawler:

Each brawler has around 40 angles from which you can see them, plus extra angles when shooting, and extra angles while moving and shooting. This quickly stacks up.

How did I manage to get all the assets:
There are no assets for this online so I had to the following:

  1. Code a script which goes into a game and makes screenshots of each angle of the brawler.
  2. Train a model to crop out the brawler from the image
  3. Train a model to segment the brawler out of the image
  4. Get other in-game assets - Will skip this as it's just me manually going over every single map and becoming a Photoshop dev
  5. Build a dataset generator with the assets

Let's begin explaining each one!

1. Script for screenshotting:
In the game there is an Skin catalog which lets us try every skin in the game for free. It looks like this:
The brawlers are in rows of 6 and the skins in rows of 3.
1.1 The logic:
On emulators, we can set a keybind to be at a specific position and when pressed, it clicks. The keybinds zxcvb were placed on top of each brawler in the first row of brawlers and asd on the first skins in the first row.
Then a simple python algorithm goes thru all skins and rows.
When in a game the logic is the following:

  1. Shoot in all possible angles so the character looks this way after the shooting animation is over.
  2. Screenshot before shooting, while shooting, after shooting
  3. Crop only the brawler using AEP model. (Main in-game model)
  4. After all angles are covered, repeat 1., 2., and 3. while the character is moving Forwards, Backwards, Diagonally each side.

Now we have images of all brawlers which look like this:

2. Model to crop the brawlers
There is specific section for this model. Click to navigate to it

3. Segment the images
First i tried using a popular tool called rembg. Results were looking like this:

Which was useful because the less complicated the object I need to crop out is, the better.
To train a model to crop out the brawler without a circle, I decided it's best to manually annotate some images and see if it works because if it does, I will save a lot of time.
I tried 50, 100, 150 and on 200 annotated images the results were perfect. Annotating looks like this:

I used roboflow.

Now that the model is ready, it was time to segment all images. It took some time but the result was the following for each brawler:

4. Dataset Generator
This part has a lot of details to it, so it will be put into a separate section

5. Training process:
The model was trained for 150 epochs on a 100k dataset on a rented RTX 4090. It cost around $35 USD.

Dataset Generator

The generator is split into the following classes

1. NameGenerator:
Generates an random username with B.S.'s font.

2. LoadData
Loads all data

3. RandomEngine
Picks the random assets / positions for the generation of example.

4. ObstacleEngine
Generates shapes of obstacles

5. Annotator
Creates an .txt annotation file with the generated example in YOLOv8 format

6. Generate
Main class, connects all classes above and generates a new example.

Start screen

Detects: All brawlers with all possible skins in start screen
Reliability: 10/10
Maintenance: Medium

What it does:
Detects brawlers in starting screen with all possible skins.

There are 6 boxes in the start always in the same position so they are cropped out and given to the model.
The training process was identical as Specific in-game data model but required a lot less data as the objects are close to static.

Brawler Selection

Detects: All brawlers in brawler selection
Reliability: 10/10
Maintenance:: Very low

What it does:
Detects brawlers in brawlers selection menu.
This model is not used however because it's easier to use an OCR and look for the text in the boxes.

Main in-game data

Detects: Ally, Enemy, Player, Walls, Bushes
Reliability: 8.5/10
Maintenance: Very low

What it does:
Detects main in-game data.