Обсуждение: continuous data from stdin
Hello, everybody!
I think it's a common question, though I couldn't find any information aboout it so far. I've got a program that puts its logs into stdout. What is the best solution to insert these logs into a table as they occur?
Thanks,
Alexey
On Tue, 18 Feb 2003 13:09:45 -0500, ÷ÅÒÅÔÅÎÎÉËÏ× áÌÅËÓÅÊ wrote: > I think it's a common question, though I couldn't find any information > aboo= ut it so far. I've got a program that puts its logs into stdout. > What is th= e best solution to insert these logs into a table as they > occur? I think you're going to have to write a program (Perl using DBI would be my choice) to use this output as input, and do INSERT statements as each record is received. (The COPY command in SQL won't do the trick, if that's what you're thinking of.) -- Jeff Boes vox 269.226.9550 ext 24 Database Engineer fax 269.349.9076 Nexcerpt, Inc. http://www.nexcerpt.com ...Nexcerpt... Extend your Expertise
If you can route the stuff to syslog, consider the modular syslog daemon (msyslogd). It has a Postgresql output module that puts the log data directly into a database. I also have modified the module for one specific log stream to parse out additional fields for entry into the database. It has been quite robust (I have my syslog->postgres connection up for months at a time and even then I'm only breaking the connection due to restarting msyslogd after code tweaking). Cheers, Steve On Tuesday 18 February 2003 10:34 am, Jeff Boes wrote: > On Tue, 18 Feb 2003 13:09:45 -0500, ÷ÅÒÅÔÅÎÎÉËÏ× áÌÅËÓÅÊ wrote: > > I think it's a common question, though I couldn't find any information > > aboo= ut it so far. I've got a program that puts its logs into stdout. > > What is th= e best solution to insert these logs into a table as they > > occur? > > I think you're going to have to write a program (Perl using DBI would be > my choice) to use this output as input, and do INSERT statements as each > record is received. (The COPY command in SQL won't do the trick, if > that's what you're thinking of.)
On Tue, 18 Feb 2003 18:09:45 +0000 (UTC),
Alexey.Veretennikov@comline.uu.ru ("???????????? ???????") wrote:
>Hello, everybody!
>
>I think it's a common question, though I couldn't find any information aboo=
>ut it so far. I've got a program that puts its logs into stdout. What is th=
>e best solution to insert these logs into a table as they occur?
>
It rather depends on the volume of these logs, but a reasonable
first pass, if the volume isn't too big, would be to write a Perl prog
to listen on stdin and write to the database.
If it was me I'd use tee as part of the pipe so I could have an
alternative record if something didn't work.
Regards,
Harry.
On Tue, 18 Feb 2003 18:09:45 +0000 (UTC),
Alexey.Veretennikov@comline.uu.ru ("???????????? ???????") wrote:
>Hello, everybody!
>
>I think it's a common question, though I couldn't find any information aboo=
>ut it so far. I've got a program that puts its logs into stdout. What is th=
>e best solution to insert these logs into a table as they occur?
>
It rather depends on the volume of these logs, but a reasonable
first pass, if the volume isn't too big, would be to write a Perl prog
to listen on stdin and write to the database.
If it was me I'd use tee as part of the pipe so I could have an
alternative record if something didn't work.
Regards,
Harry.
On Tue, 2003-02-18 at 13:34, Jeff Boes wrote: > I think you're going to have to write a program (Perl using DBI would be > my choice) to use this output as input, and do INSERT statements as each > record is received. (The COPY command in SQL won't do the trick, if > that's what you're thinking of.) Actually, COPY should work fine, AFAICS. Write a client app to open a connection to PostgreSQL, execute "COPY xxx FROM stdin;", and then push lines of COPY input over the wire to PostgreSQL using PQputline(), and end the COPY using PQendcopy(). That assumes you're using libpq -- other interfaces should provide a similar way to do the same thing. Cheers, Neil -- Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC