Re: [PERFORM] Odd sudden performance degradation related to tempobject churn

Поиск
Список
Период
Сортировка
От Jeremy Finzel
Тема Re: [PERFORM] Odd sudden performance degradation related to tempobject churn
Дата
Msg-id CAMa1XUg1MErrJifeeTNse+TQdbEZjs3t+tzOvnu7KCW9tUqcsA@mail.gmail.com
обсуждение исходный текст
Ответ на Re: [PERFORM] Odd sudden performance degradation related to tempobject churn  (Scott Marlowe <scott.marlowe@gmail.com>)
Ответы Re: [PERFORM] Odd sudden performance degradation related to tempobject churn  (Mark Kirkwood <mark.kirkwood@catalyst.net.nz>)
Список pgsql-performance
On Tue, Aug 15, 2017 at 12:07 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
So do iostat or iotop show you if / where your disks are working
hardest? Or is this CPU overhead that's killing performance?

Sorry for the delayed reply.  I took a look in more detail at the query plans from our problem query during this incident.  There are actually 6 plans, because there were 6 unique queries.  I traced one query through our logs, and found something really interesting.  That is that all of the first 5 queries are creating temp tables, and all of them took upwards of 500ms each to run.  The final query, however, is a simple select from the last temp table, and that query took 0.035ms!  This really confirms that somehow, the issue had to do with writing to the SAN, I think.  Of course this doesn't answer a whole lot, because we had no other apparent issues with write performance at all.

I also provide some graphs below.

7pm-3am on 8/10 (first incidents were around 10:30pm, other incidents ~1am, 2am):

Local Disk IO:

Screen Shot 2017-08-18 at 8.20.06 AM.png

SAN IO:

Screen Shot 2017-08-18 at 8.16.59 AM.png

CPU:

Screen Shot 2017-08-18 at 8.20.58 AM.png

7-9pm on 8/10 (controlled attempts starting a little after 7):

CPU:

Screen Shot 2017-08-18 at 8.43.35 AM.png

Write IO on SAN:

Screen Shot 2017-08-18 at 8.44.32 AM.png

Read IO on Local disk:

Screen Shot 2017-08-18 at 8.46.27 AM.png

Write IO on Local disk:

Screen Shot 2017-08-18 at 8.46.58 AM.png


В списке pgsql-performance по дате отправления:

Предыдущее
От: Claudio Freire
Дата:
Сообщение: Re: [PERFORM] performance problem on big tables
Следующее
От: kimaidou
Дата:
Сообщение: [PERFORM] Query plan for views and WHERE clauses, Luke is not using the index