The obvious thing to do is to keep a computed average score for each user - either in a separate table which you update based on changes to words_moves, which you could do with a trigger, or using a materialized view which you refresh at suitable intervals (this has the drawback that the data will not be immediately up-to-date).
Combining these two changes should get you to under 100ms, maybe.
I was considering creating a cronjob, but now I will better create a trigger as you have suggested
Чтобы сделать работу с сайтом удобнее, мы используем cookie и аналитический сервис «Яндекс.Метрика». Продолжая пользоваться сайтом, вы соглашаетесь с их использованием.