Hello Again Jason. It's been a while and I'm finally able to take some time to get this working again. I'll let you know what I come up with. The requirement has grown to be able to search all types of log files (Solaris Audit, Windows Event Logs, Application Logs, Etc).
I do have a greater understanding of the appliances target market, but I have to try and shoe-horn the appliance into the solution so prove value. (wonderful, isn't it)
One option I'm looking at is to parse the data to XML and feed a DB that is walked by the appliance (like you said earlier). The downside is the volume of data. Thousands of log/audit files.
Unless there are connectors for these types of log files now.
> One option I'm looking at is to parse the data to XML
> and feed a DB that is walked by the appliance (like you
> said earlier). The downside is the volume of data.
> Thousands of log/audit files.
If you have your own program that parses the data, you can have THAT program shove it directly in the appliance with our Data Load API. We provide an interface so that arbitrary records can be pushed in to the HTML table.
Already there, look in the docs for "DataLoad API".
It's essentially a POST http request with an XML body attached to it. The docs tell you where URL to submit it to, the format for the request, and what to expect from the response.
One thing to keep in mind is that the appliance is geared towards "web searches", so each record you push in has a URL associated with it that will be presented in the search results as the "hit" to link to when that data matches.
If you're linking to 100M log files, be sure that users understand what they're clicking on.