Jack:
On Sun, 14 Apr 2024 at 22:18, jack <jack4pg@a7q.com> wrote:
> The CSV files are being produced by another system, a WIndows app on a Windows machine. I then copy them to a USB key
andcopy them onto the ubuntu machine. The data is then imported via the COPY command.
>
> COPY master (field01,field02..fieldX) FROM '/data/file.text' DELIMITER E'\t'
> The fields are tab delimited.
Then call them tab-delimited, not CSV. It may be important. You have a
tendency of sending your interpretation of information which hampers
people trying to help you.
As an example, only about three messages above did we see the "signal
11" in the server logs which indicates a probable bug. Signal 11 is
normally due to stray pointers or corrupted memory ( although faulty
ram can lead to it too ), which hints at a bug, not a resource
problem.
> But importing the data works. I can get all the data into a single table without any problems. The issue is only when
Istart to update the single table. And that is why I started using smaller temporary tables for each CSV file, to do
theupdates in the smaller tables before I move them all to a single large table.
> After all the data is loaded and updated, I run php programs on the large table to generate reports. All of which
workswell EXCEPT for performing the updates on the data. And I do not want to use perl or any outside tool. I want it
allone in SQL because I am required to document all my steps so that someone else can take over, so everything needs to
beas simple as possible.
I suggested perl ( or any similar thing, php should be equally fine )
as it normally makes your importing / debugging much easier / lighter.
On the documentation side, you can have the data and some
transformations in a big sql file, but having some steps in a php file
will have them documented too. But if you want to do text processing
in SQL, go ahead, the only problems are going to be making it harder
to debug and probably harder to document. Now people know the signal
11 stuff and the problem can probably be tracked.
Francisco Olarte.