The obvious thing to do is to keep a computed average score for each user - either in a separate table which you update based on changes to words_moves, which you could do with a trigger, or using a materialized view which you refresh at suitable intervals (this has the drawback that the data will not be immediately up-to-date).
Combining these two changes should get you to under 100ms, maybe.
I was considering creating a cronjob, but now I will better create a trigger as you have suggested