Alternative to other entries. Cached value

[Lewiscowles1986]

This is naughty, but also simple to reason about:

• We simply add the fields we want to our table, using Laravel (could also use DBMS specific features).
• Retrieve the cached (non-live) value.
• In migration we seed the table with values.
• In Scheduled task we keep the ratings data fresh, ignoring recently updated entries.

This would better suit a queue-task based system with a lot of rows.

I Just wanted to present something that wasn't playing with the query builder or abandoning eloquent, but was also simple for most levels to get their head around.

References

• https://laravel.com/docs/8.x/scheduling
• https://laravel.com/docs/8.x/migrations#migration-structure
• https://laravel.com/docs/8.x/eloquent#chunking-results

So we are paying the cost of the original queries, but only when needed, which opens to door to other optimisations.

No matter how much you optimise the queries, you'll be performing them 1 time every page visit.

Another thing you could do is add cache, which is simpler, although that seems to avoid the eloquent requirement, which is why I avoided.

cursor could become lazy I think for some small win in some situations (we only ever fetch top 100 results here).

423 lines of this are composer noise so I could rollback to main and continue seeding until I iterated past editing the migration. The PHP is 67 net lines.

[Lewiscowles1986]

[Lewiscowles1986]

Dropped composer changes to make this clearer. Solving php artisan migrate:rollback should be another PR

[rm-yakovenko]

You can also add an index on rating field so sorting will work much faster.

[Lewiscowles1986]

Sure. The thing about indexes are that they are a trade-off. Do you have a branch with the index added, so that you can show me a query plan or debug-bar to demonstrate?

[rm-yakovenko]

rm-yakovenko/Laravel-Challenge-Movie-Table-Eloquent@main...rm-yakovenko:pr-36

There are several changes.

1. Fix precision in migration and add an index on ratings.movie_id.
2. Instead of command, update movie's rating on rating created/updated/deleted. This should keep the cache live without delays.
3. Create 20k movies and 50k reviews. Seeding data can take some time.

Benchmarking
ab -n 100 <url>

Method	Time per request (lower is better)	Diff baseline
Caching with index	18.097	0.00%
Caching no index	24.52	35.49%
Povilas' solution	137.473	659.65%
Caching with index (40k movies)	18.59	2.72%

Adding new movies almost doesn't change overall time per request. I think that caching is the best option if you're going to have lots of data.

Feel free to ask any questions.

[Lewiscowles1986]

I broadly really like it. Thanks for showing figures. To get an explain you can toSql the query and then prepend explain, although it does look much faster and the model observer is a nice touch.

[Lewiscowles1986]

@rm-yakovenko I updated this to use your suggestion of an index for the rating field, but missed out several of your other changes.

• seeder alterations
• moving limit / pagination into model
• decimal size 8,2 (I did change to 4,2 as 2,2 did seem to have issues). I Believe that might be better addressed by having DBMS be more permissive to truncate the extra data it receives, but 4,2 seems to avoid any complications.
• model observer (this will mean although we limit reads, writes come out of our control. For that approach I might sooner use raw SQL and be more efficient using a CTE. For movie ratings, I don't think writes would be so often, but it is still an amplification and coupling on movies, which I haven't yet come around to).

I Did alter the controller method to get 110 models, instead of 1010 by playing with your query; although I am not thrilled with the get needing to be passed down to the controller, I really wanted to avoid the need to ->get() in that controller, but it looks like efficiency has led to coordination. In Ruby on Rails, the get is implicit. I wonder if there is a way to get this using PHP & Laravel. Implicit Get on access of a builder as an iterator?

[rm-yakovenko]

I broadly really like it. Thanks for showing figures. To get an explain you can toSql the query and then prepend explain, although it does look much faster and the model observer is a nice touch.

explain select * from `movies` order by `rating` desc limit 100

There are 20k movies in my local DB.

With index

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	movies	index		movies_rating_index	4		100

Without index

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
1	SIMPLE	movies	ALL					20112	Using filesort

With index DB only iterates over 100 rows.

[rm-yakovenko]

I wonder if there is a way to get this using PHP & Laravel. Implicit Get on access of a builder as an iterator?

I think no. You should call the get method explicitly to get an iterator.