Обсуждение: Machine learning top of db logs

I have been thinking about how to apply NLP work to logs. The problem is that logs are structured data. It would depend on what you are looking for, what your outcomes are, and how you want to generate it. The other problem is that your problems are going to be very rare events. Rare event detection is its own special mess. Like, if you have 1000 users generating 1 million line log files, how do you pick out the 10 that are bad? You also need to make those rare events representative of very rare events so you might need say, 100 million line log files with a know 100 or 1000 issues to start getting models to work.