Huge shared hit for small table

Hello all,

We are trying to debug some slow performance in our production environment (Amazon RDS, Postgresql 9.6.11), and we’re looking at a particular EXPLAIN node that seems… weird.  This is a very large query involving a number of joins, but it performs pretty well in our staging environment (which has roughly the same data set as production, with a few tweaks).  However, there is one node in the EXPLAIN plan that is wildly different:

In the staging environment, we get this:

Index Scan using "programPK" on public.program prog  (cost=0.29..0.35 rows=1 width=16) (actual time=0.002..0.003 rows=1 loops=21965)

      Output: prog.id, prog.version, prog.active, prog.created_date, prog.last_modified_date, prog.created_by, prog.last_modified_by, prog.client_id, prog.scheme_id, prog.name, prog.legacy_group_id, prog.custom_fields, prog.setup_complete, prog.setup_messages, prog.legacy_program_type

      Index Cond: (prog.id = per.program)

      Buffers: shared hit=87860

In the production environment, we get this:

Index Scan using "programPK" on public.program prog  (cost=0.29..0.36 rows=1 width=16) (actual time=0.017..4.251 rows=1 loops=21956)

       Output: prog.id, prog.version, prog.active, prog.created_date, prog.last_modified_date, prog.created_by, prog.last_modified_by, prog.client_id, prog.scheme_id, prog.name, prog.legacy_group_id, prog.custom_fields, prog.setup_complete, prog.setup_messages, prog.legacy_program_type

       Index Cond: (prog.id = per.program)

       Buffers: shared hit=25437716

The tables in both environments are about the same size (18MB) and the indexes are about the same size (360kb/410kb) – and the shared hits are pretty much the same on the other nodes of the query between the two environments. 

Lastly, if we take out the join to the “program” table, the query performs much faster in production and the timing between staging and production is similar. 

This has happened one time before, and we did a “REINDEX” on the program table – and that made the problem mostly go away.  Now it seems to be back, and I’m not sure what to make of it. 

Thanks in advance for any help you can offer!

Scott

SCOTT RANKIN
VP, Technology

Motus, LLC
Two Financial Center, 60 South Street, Boston, MA 02111 
617.467.1900 (O) | srankin@motus.com

Follow us on LinkedIn | Visit us at motus.com