I've just committed a log-parsing front-end for mk-query-digest, a
tool included in Maatkit, so it can parse and analyze Postgres log
files. I encourage people to try this functionality and give feedback
by replying here, adding to
http://code.google.com/p/maatkit/issues/detail?id=535, or jumping on
the Maatkit mailing list. A brief mental Q&A of things I expect
people to ask:
Q: Why another log analysis tool? Isn't pgfouine good enough?
A: The mk-query-digest log analysis tool already existed, and lots of
work has gone into making it very good. All I did was add a
pg-capable parser to it. I am not trying to play king-of-the-mountain
with any other log analysis tool, but in brief browsing around I saw
complaints that a mere GB or three of log files was taking hours to
parse in pgfouine. I don't have real-life files that big myself, but
mk-query-digest has been extensively tuned for performance and has no
problems with many gigs of files in other formats (but please do watch
out for memory consumption; Perl hogs RAM. Run this on a non-critical
server, please.) Additionally, mk-query-digest has some nice
properties: just download-and-go with no installation necessary;
written in Perl with minimal dependencies so no PHP install or CPAN
libraries are necessary; and lots more. Finally, it's not a me-too
tool; it is different from pgfouine and different might be a good
thing.
Q: How do I get it?
A: At this point, it's unreleased, but you can get the latest SVN
trunk in the usual Maatkit way: "wget
http://www.maatkit.org/trunk/mk-query-digest".
Q: How do I run it?
A: Simple: "perl mk-query-digest --type pglog /path/to/logfile" should
produce a report on the most important queries.
Q: Where is the documentation?
A: Maatkit's documentation is always embedded within the tools
themselves. Use "perldoc mk-query-digest" and search for "pglog" to
find docs on this specific feature, or use the --help option to get an
overview of the tool in general. This is a complex and powerful tool,
and I encourage you to learn more about what you can do with it. The
default is to do something useful, as you should see by running the
command above.
Q: What's the development status?
A: Dozens of PG-specific unit and integration test cases all pass
cleanly. But I need real-life testing and bug reports, feature
requests, etc. Also, the reporting format and perhaps some other
functionality is not yet PG-aware. There are helpful little
copy-paste ready shortcuts for things like examining the structure of
tables found within queries; right now these are MySQL-centric. But
that will change.
That's all for now -- let me know what you think!
- Baron