Re: Moving delta data faster

On 4/4/24 13:42, yudhi s wrote:
> 
> On Thu, Apr 4, 2024 at 9:04 PM Adrian Klaver <adrian.klaver@aklaver.com 
> <mailto:adrian.klaver@aklaver.com>> wrote:
> 
>     On 4/3/24 22:24, yudhi s wrote:
>      >
>      > On Thu, Apr 4, 2024 at 10:16 AM Adrian Klaver
>     <adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>
>      > <mailto:adrian.klaver@aklaver.com
>     <mailto:adrian.klaver@aklaver.com>>> wrote:
> 
>     S3 is not a database. You will need to be more specific about '... then
>     from the S3 it will be picked and gets merged to the target postgres
>     database.'
> 
> 
> The data from S3 will be dumped into the stage table and then the 
> upsert/merge from that table to the actual table.

The S3 --> staging table would be helped by having the data as CSV and 
then using COPY. The staging --> final table step could be done as 
either ON CONFLICT or MERGE, you would need to test in your situation to 
verify which works better.

-- 
Adrian Klaver
adrian.klaver@aklaver.com