On Thu, Aug 20, 2009 at 2:45 AM, Merlin Moncure<mmoncure@gmail.com> wrote:
>
> I think your first step is to stand back, think about the problem as a
> whole and see if you can't define your requirements better:
>
> *) how much data comes in each day? how much rolls out?
Need to get some data...will have this tomorrow.
> *) what types of analysis of the data would you like to do?
Basically, what I need to do is feed the tick data to the CEP engine
as a data stream. The CEP engine creates various data bars streams say
2 minute OHLC (open/high/low/close), 3 min, 5 min, 15 min, 30 min or
20 tick, 2000 tick, 5000 tick etc. & so on. These are to be fed to
strategies which will trade. At least this is how it is done if data
arrives from the online data provider.
I am sure that this could be done on database. I had originally
thought of populated views of prebuilt data bars so building the bars
should not be required. They should be built using triggers as soon as
the data arrives in postgres, but then everyone advised that it was
exactly due to slow results in this type of data that CEP engines were
built. I was told that if I used this type of thing, so many triggers
would be running that I would have my processors and RAM maxing out,
not to say the exponential increase in disk space requirements.
And this is for one symbol. On an average 2000 plus scrips need to be analysed.
Again the analysis is backtesting trade simulation results according
to various strategies, so that means at least three years plus data,
if one wants to be a stickler 5-6 years but one year is the bare
minimum.
> *) what are your performance expectations?
>
Well, needfully the simulations should finish overnight with three
years worth of data.
> For example, try and describe what you would like to do in detail.
> The data you would import, how you would like it structured, and a
> typical query. While the amount of price data a stock market produces
> is immense, these types of problems tend to scale very well
> horizontally. So let's define the basic amount of work you'd like to
> do, then build off that.
>
I'll get back on this by tomorrow. Thanks.
With best regards.
Sanjay.