Обсуждение: Initializing LWLock Array from Server Code
Hi,
I have created a shared hash table in partitioned mode inside the postgres server code. In order to guard the partitions, I'm trying to initialize an array of LWLocks. The code that I'm trying to use for that is
I'm not sure where exactly should this code be called from the server code. So I had placed it in
void CreateSharedMemoryAndSemaphores(bool makePrivate, int port);
within ipic.c. However, I'm getting the following error message when starting the database:
FATAL: requested tranche is not registered
So at this point, I'm a little confused as to where the methods should be called from inside the server code. Any pointers would be appreciated.
Thanks,
-SB
PS: I saw that that are two variables controlling access to RequestNamedLWLockTranche()
which implies that IsUnderPostmaster must be false and lock_named_request_allowed should be true. Thus, I had invoked RequestNamedLWLockTranche before the first call to LWLockShmemSize which sets lock_named_request_allowed = true and GetNamedLWLockTranche later. This works in the single user mode but fails when I start the server explicitly through postgres -D ...
Hi Robert,
Thank you for your reply and sorry that I couldn't reply earlier.
Since I didn't get any response within a couple of days, I took the longer route -- changed the lwlock.h and lwlock.c
for accommodating the lw locks for the shared hash table.
I'll describe what I modified in the lwlock.h and lwlock.c and it seems to be working fine. Although I haven't got an opportunity
to test it extensively. If you could let me know if I missed out anything that might cause problems later that would be great.
In lwlock.h, the following modifications were made:
/* the number of partitions of the shared hash table */
#define NUM_FILTERHASH_PARTITIONS 64
/* Offsets for various chunks of preallocated lwlocks. */
#define BUFFER_MAPPING_LWLOCK_OFFSET NUM_INDIVIDUAL_LWLOCKS
#define LOCK_MANAGER_LWLOCK_OFFSET \
(BUFFER_MAPPING_LWLOCK_OFFSET + NUM_BUFFER_PARTITIONS)
#define PREDICATELOCK_MANAGER_LWLOCK_OFFSET \
(LOCK_MANAGER_LWLOCK_OFFSET + NUM_LOCK_PARTITIONS)
#define BUFFER_MAPPING_LWLOCK_OFFSET NUM_INDIVIDUAL_LWLOCKS
#define LOCK_MANAGER_LWLOCK_OFFSET \
(BUFFER_MAPPING_LWLOCK_OFFSET + NUM_BUFFER_PARTITIONS)
#define PREDICATELOCK_MANAGER_LWLOCK_OFFSET \
(LOCK_MANAGER_LWLOCK_OFFSET + NUM_LOCK_PARTITIONS)
/* offset for shared filterhash lwlocks in the MainLWLockArray */
#define FILTERHASH_OFFSET \
(PREDICATELOCK_MANAGER_LWLOCK_OFFSET + NUM_PREDICATELOCK_PARTITIONS)
(PREDICATELOCK_MANAGER_LWLOCK_OFFSET + NUM_PREDICATELOCK_PARTITIONS)
/* modified NUM_FIXED_LOCKS */
#define NUM_FIXED_LWLOCKS \
(FILTERHASH_OFFSET + NUM_FILTERHASH_PARTITIONS)
(FILTERHASH_OFFSET + NUM_FILTERHASH_PARTITIONS)
/* added LWTRANCHE_FILTERHASH in the BuiltinTrancheIds */
typedef enum BuiltinTrancheIds
{
LWTRANCHE_CLOG_BUFFERS = NUM_INDIVIDUAL_LWLOCKS,
LWTRANCHE_COMMITTS_BUFFERS,
LWTRANCHE_SUBTRANS_BUFFERS,
LWTRANCHE_MXACTOFFSET_BUFFERS,
LWTRANCHE_MXACTMEMBER_BUFFERS,
LWTRANCHE_ASYNC_BUFFERS,
LWTRANCHE_OLDSERXID_BUFFERS,
LWTRANCHE_WAL_INSERT,
LWTRANCHE_BUFFER_CONTENT,
LWTRANCHE_BUFFER_IO_IN_PROGRESS,
LWTRANCHE_REPLICATION_ORIGIN,
LWTRANCHE_REPLICATION_SLOT_IO_IN_PROGRESS,
LWTRANCHE_PROC,
LWTRANCHE_BUFFER_MAPPING,
LWTRANCHE_LOCK_MANAGER,
LWTRANCHE_PREDICATE_LOCK_MANAGER,
LWTRANCHE_FILTERHASH,
LWTRANCHE_PARALLEL_HASH_JOIN,
LWTRANCHE_PARALLEL_QUERY_DSA,
LWTRANCHE_SESSION_DSA,
LWTRANCHE_SESSION_RECORD_TABLE,
LWTRANCHE_SESSION_TYPMOD_TABLE,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_TBM,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
{
LWTRANCHE_CLOG_BUFFERS = NUM_INDIVIDUAL_LWLOCKS,
LWTRANCHE_COMMITTS_BUFFERS,
LWTRANCHE_SUBTRANS_BUFFERS,
LWTRANCHE_MXACTOFFSET_BUFFERS,
LWTRANCHE_MXACTMEMBER_BUFFERS,
LWTRANCHE_ASYNC_BUFFERS,
LWTRANCHE_OLDSERXID_BUFFERS,
LWTRANCHE_WAL_INSERT,
LWTRANCHE_BUFFER_CONTENT,
LWTRANCHE_BUFFER_IO_IN_PROGRESS,
LWTRANCHE_REPLICATION_ORIGIN,
LWTRANCHE_REPLICATION_SLOT_IO_IN_PROGRESS,
LWTRANCHE_PROC,
LWTRANCHE_BUFFER_MAPPING,
LWTRANCHE_LOCK_MANAGER,
LWTRANCHE_PREDICATE_LOCK_MANAGER,
LWTRANCHE_FILTERHASH,
LWTRANCHE_PARALLEL_HASH_JOIN,
LWTRANCHE_PARALLEL_QUERY_DSA,
LWTRANCHE_SESSION_DSA,
LWTRANCHE_SESSION_RECORD_TABLE,
LWTRANCHE_SESSION_TYPMOD_TABLE,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_TBM,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
In lwlock.c, the following modifications were made:
In function, static void InitializeLWLocks(void), the following lines were added:
/* Initialize filterhash LWLocks in main array */
lock = MainLWLockArray + NUM_INDIVIDUAL_LWLOCKS +
NUM_BUFFER_PARTITIONS + NUM_LOCK_PARTITIONS + NUM_FILTERHASH_PARTITIONS;
for (id = 0; id < NUM_FILTERHASH_PARTITIONS; id++, lock++)
LWLockInitialize(&lock->lock, LWTRANCHE_FILTERHASH);
lock = MainLWLockArray + NUM_INDIVIDUAL_LWLOCKS +
NUM_BUFFER_PARTITIONS + NUM_LOCK_PARTITIONS + NUM_FILTERHASH_PARTITIONS;
for (id = 0; id < NUM_FILTERHASH_PARTITIONS; id++, lock++)
LWLockInitialize(&lock->lock, LWTRANCHE_FILTERHASH);
In function void RegisterLWLockTranches(void), the following line was added:
LWLockRegisterTranche(LWTRANCHE_FILTERHASH, "filter_hash");
All of this was done after allocating the required of space in the shared memory.
Thanks,
-SB
On Mon, Apr 29, 2019 at 1:59 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Fri, Apr 26, 2019 at 2:58 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:
> I have created a shared hash table in partitioned mode inside the postgres server code. In order to guard the partitions, I'm trying to initialize an array of LWLocks. The code that I'm trying to use for that is
>
> void RequestNamedLWLockTranche(const char *tranche_name, int num_lwlocks);
> LWLockPadded *GetNamedLWLockTranche(const char *tranche_name);
>
> I'm not sure where exactly should this code be called from the server code. So I had placed it in
>
> void CreateSharedMemoryAndSemaphores(bool makePrivate, int port);
>
> within ipic.c. However, I'm getting the following error message when starting the database:
>
> FATAL: requested tranche is not registered
>
> So at this point, I'm a little confused as to where the methods should be called from inside the server code. Any pointers would be appreciated.
RequestNamedLWLockTranche() changes the behavior of LWLockShmemSize()
and CreateLWLocks(), so must be called before either of those.
GetNamedLWLockTranche() must be called after CreateLWLocks().
This machinery works for pg_stat_statements, so see that as an example
of how to make this work from an extension. If you're hacking the
core server code, then look at the places where the corresponding bits
of pg_stat_statements code get called. IIRC, _PG_init() gets called
from process_shared_preload_libraries(), so you might look at the
placement of the calls to that function.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Apr 30, 2019 at 5:52 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote: > Thank you for your reply and sorry that I couldn't reply earlier. > Since I didn't get any response within a couple of days, I took the longer route -- changed the lwlock.h and lwlock.c > for accommodating the lw locks for the shared hash table. > > I'll describe what I modified in the lwlock.h and lwlock.c and it seems to be working fine. Although I haven't got an opportunity > to test it extensively. If you could let me know if I missed out anything that might cause problems later that would begreat. Well, I can't really vouch for your code on a quick look, but it doesn't look insane. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Replying to myself to resend to the list, since my previous attempt seems to have been eaten by a grue. On Mon, Apr 29, 2019 at 1:59 PM Robert Haas <robertmhaas@gmail.com> wrote: > > On Fri, Apr 26, 2019 at 2:58 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote: > > I have created a shared hash table in partitioned mode inside the postgres server code. In order to guard the partitions,I'm trying to initialize an array of LWLocks. The code that I'm trying to use for that is > > > > void RequestNamedLWLockTranche(const char *tranche_name, int num_lwlocks); > > LWLockPadded *GetNamedLWLockTranche(const char *tranche_name); > > > > I'm not sure where exactly should this code be called from the server code. So I had placed it in > > > > void CreateSharedMemoryAndSemaphores(bool makePrivate, int port); > > > > within ipic.c. However, I'm getting the following error message when starting the database: > > > > FATAL: requested tranche is not registered > > > > So at this point, I'm a little confused as to where the methods should be called from inside the server code. Any pointerswould be appreciated. > > RequestNamedLWLockTranche() changes the behavior of LWLockShmemSize() > and CreateLWLocks(), so must be called before either of those. > GetNamedLWLockTranche() must be called after CreateLWLocks(). > > This machinery works for pg_stat_statements, so see that as an example > of how to make this work from an extension. If you're hacking the > core server code, then look at the places where the corresponding bits > of pg_stat_statements code get called. IIRC, _PG_init() gets called > from process_shared_preload_libraries(), so you might look at the > placement of the calls to that function. > > -- > Robert Haas > EnterpriseDB: http://www.enterprisedb.com > The Enterprise PostgreSQL Company -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company