Marking shared buffer lookup table as HASH_FIXED_SIZE

Поиск
Список
Период
Сортировка
От Ashutosh Bapat
Тема Marking shared buffer lookup table as HASH_FIXED_SIZE
Дата
Msg-id CAExHW5v0jh3F_wj86yC=qBfWk0uiT94qy=Z41uzAHLHh0SerRA@mail.gmail.com
обсуждение исходный текст
Список pgsql-hackers
Hi All,
StrategyInitialize() calls InitBufTable() passing the maximum number
of possible entries in the shared buffer lookup table. The table can
not have more entries than the number of available shared buffers +
number of partitions as explained in the comment in
StrategyInitialize(). If there are more than those entries something
is wrong with the hash table like mapping two different pages to the
same buffer. InitBufTable() invokes ShmemInitHash() with the same init
and max size. This means that the maximum number of entries are
allocated right from the beginning. So there should not be any need to
allocate more entries after initial setup. Thus the hash table
qualifies to be marked as HASH_FIXED_SIZE, so that we don't end up
with a corrupted hash table in case of a bug.

Here's a patch to do so.

We are discussing resizing shared buffers without a restart in [1].
When testing those patches, I noticed that the buffer lookup table was
getting enlarged even though the underlying memory segment didn't have
enough memory causing other errors. Marking the shared buffer lookup
table as fixed size will be good to detect such errors earlier. As a
side note, we will have to somehow tweak the HASH_FIXED_SIZE flag if
we have to resize the shared buffer table while resizing the shared
buffers table. But that's the problem for that patch set, not this.

I looked at other callers of ShmemInitHash(). Shared tables in lock.c
and wait_event.c use different initial and maximum table sizes. So I
think they can not be marked fixed sized. Those in predicate.c are
already marked HASH_FIXED_SIZE. pg_stat_statement.c has a shared hash
table with the same initial and maximum size. It can be marked as
HASH_FIXED_SIZE, but it doesn't seem to be worth the trouble in case
it requires resizing in corner cases. Shmem index sets maximum and
initial size as the same, but the comment there says that it's a soft
limit. So didn't add HASH_FIXED_SIZE to that hash table. So it's only
the shared buffer lookup table, that is worth marking as fixed sized.

I ran pgbench, scale 100, with shared_buffers = 128kB and 128MB resp.
for 10 minutes. I saw  "pgbench: error: client 7 script 0 aborted in
command 5 query 0: ERROR:  no unpinned buffers available" in case of
128kB. I think that is expected with such a small buffer pool and
large scale. No other error, crash or Assert failure was seen. The
whole buffer cache was used all the time in both the cases. The change
looks safe.

It will be good to get this committed early to detect any lingering
bugs which cause the hash table to grow beyond the initial size.

[1] https://www.postgresql.org/message-id/my4hukmejato53ef465ev7lk3sqiqvneh7436rz64wmtc7rbfj%40hmuxsf2ngov2

-- 
Best Wishes,
Ashutosh Bapat

Вложения

В списке pgsql-hackers по дате отправления: