Обсуждение: COPY FROM how long should take ?
Hello, today I try to restore on of my tables with copy from file that I made before. The file is 2.4 GB, only integers ... took me 3 hours and 30 min and I hit CTRL+C on i7 processor with 8 GB memory, sata 2 hard drive. I modify some psql conf file values and increase memory, work, wal, temp, check point segments to 55 Is that normal ? Whole file is 37 mil lines. When I hit enter it was on line as logs says 26 million. I run it twice and second time after 45 min I again hit CTRL+C and in logs I see it was again on 26 million line. Well, the line number is approximately. Is everything normal ? Postgresql 9.0.4 -- Regards, Condor
> Date: 08/26/2011 05:40 PM
> Subject: [GENERAL] COPY FROM how long should take ?
> Sent by: pgsql-general-owner@postgresql.org
>
> Hello,
> today I try to restore on of my tables with copy from file that I made
> before.
> The file is 2.4 GB, only integers ... took me 3 hours and 30 min and I
> hit
> CTRL+C on i7 processor with 8 GB memory, sata 2 hard drive. I modify
> some
> psql conf file values and increase memory, work, wal, temp, check point
> segments to 55
> Is that normal ? Whole file is 37 mil lines. When I hit enter it was on
> line
> as logs says 26 million. I run it twice and second time after 45 min I
> again hit
> CTRL+C and in logs I see it was again on 26 million line. Well, the
> line number
> is approximately. Is everything normal ?
>
> Postgresql 9.0.4
'It was on 26 million line' - does this mean there were 26 million records in the table? I was migrating data from Oracle to PostgreSQL, using ora2pg with COPY option and in under 3 hours it easily copied a table that was 10 GB in size. Are you seeing any errors in postgres log file? In my case, the number of records in postgresql table stopped increasing after some time and I realized data transfer was failing. I checked the log file for errors,corrected the data issues in the source table (Oracle) and it finished the load without issues.
Regards,
Jayadevan
DISCLAIMER:
"The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect."
> Subject: [GENERAL] COPY FROM how long should take ?
> Sent by: pgsql-general-owner@postgresql.org
>
> Hello,
> today I try to restore on of my tables with copy from file that I made
> before.
> The file is 2.4 GB, only integers ... took me 3 hours and 30 min and I
> hit
> CTRL+C on i7 processor with 8 GB memory, sata 2 hard drive. I modify
> some
> psql conf file values and increase memory, work, wal, temp, check point
> segments to 55
> Is that normal ? Whole file is 37 mil lines. When I hit enter it was on
> line
> as logs says 26 million. I run it twice and second time after 45 min I
> again hit
> CTRL+C and in logs I see it was again on 26 million line. Well, the
> line number
> is approximately. Is everything normal ?
>
> Postgresql 9.0.4
'It was on 26 million line' - does this mean there were 26 million records in the table? I was migrating data from Oracle to PostgreSQL, using ora2pg with COPY option and in under 3 hours it easily copied a table that was 10 GB in size. Are you seeing any errors in postgres log file? In my case, the number of records in postgresql table stopped increasing after some time and I realized data transfer was failing. I checked the log file for errors,corrected the data issues in the source table (Oracle) and it finished the load without issues.
Regards,
Jayadevan
DISCLAIMER:
"The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect."
On Fri, 26 Aug 2011 17:49:35 +0530, Jayadevan M wrote:
> Date: 08/26/2011 05:40 PM
> Subject: [GENERAL] COPY FROM how long should take ?
> Sent by: pgsql-general-owner@postgresql.org
>
> Hello,
> today I try to restore on of my tables with copy from file that I made
> before.
> The file is 2.4 GB, only integers ... took me 3 hours and 30 min and I
> hit
> CTRL+C on i7 processor with 8 GB memory, sata 2 hard drive. I modify
> some
> psql conf file values and increase memory, work, wal, temp, check point
> segments to 55
> Is that normal ? Whole file is 37 mil lines. When I hit enter it was on
> line
> as logs says 26 million. I run it twice and second time after 45 min I
> again hit
> CTRL+C and in logs I see it was again on 26 million line. Well, the
> line number
> is approximately. Is everything normal ?
>
> Postgresql 9.0.4
'It was on 26 million line' - does this mean there were 26 million records in the table? I was migrating data from Oracle to PostgreSQL, using ora2pg with COPY option and in under 3 hours it easily copied a table that was 10 GB in size. Are you seeing any errors in postgres log file? In my case, the number of records in postgresql table stopped increasing after some time and I realized data transfer was failing. I checked the log file for errors,corrected the data issues in the source table (Oracle) and it finished the load without issues.
Regards,
Jayadevan
Whole file is approximately 32 million lines = 32 million records. I check log file no any errors.
-- Regards, Condor
You can add a trigger and send message every 100000 rows -- ------------ pasman