Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3

Поиск
Список
Период
Сортировка
От Raj kumar
Тема Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3
Дата
Msg-id CACxU--XrvwT6Ju5=WhNtoWE=fUYTLjnuFFr=NDEc_NwzAdv7KQ@mail.gmail.com
обсуждение исходный текст
Ответы Re: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3  (Raj kumar <rajkumar820999@gmail.com>)
Список pgsql-admin
Дерево обсуждения

AWS DMS supports change data capture (CDC) using logical replication. I'm using PostgreSQL V12 as the DMS Source and migrating data to S3 DataLake through DMS.

After creating replication slot for pglogical and enabling CDC from Postgresql RDS to S3, the Storage space consumed heavily and increased by 1TB in a day and filled up the storage completely. What could be the reason for this? How can I avoid this? Please advice.

To enable logical replication of a self-managed PostgreSQL source database, I followed the below steps.
  1. Set wal_level = logical.
  2. Set max_replication_slots to a value greater than 1.
  3. Set the max_replication_slots value to 10
  4. Set max_wal_senders to a value greater than 10.
  5. Set wal_sender_timeout =0.
GRANT USAGE on SCHEMA pglogical to DATALAKE_USER;

GRANT ALL on ALL TABLES in SCHEMA pglogical to DATALAKE_USER;

SELECT pglogical.create_node(node_name := 'postgresdb',dsn := 'host=postgresinstancedb.ap-south-1.rds.amazonaws.com port=5432 dbname=masterpoc');

select pglogical.create_replication_set('pgdms_replication_slot_datalake', true, false, false, true);

Thanks & Regards,
Raj Kumar Narendiran.

В списке pgsql-admin по дате отправления:

Предыдущее
От: Mladen Gogala
Дата:
Сообщение: Re: package conversion error from oracle to postgresql database !!!!!!!!!
Следующее
От: Raj kumar
Дата:
Сообщение: Re: Storage filled up heavily after enabling pglogical for CDC from AWS RDS to S3