I too like the idea of multiple files for the data to allow data updates while running. But it's a complicate scenario that still doesn't help you out if you lose your internet connection temporarily during a particular session.
To that end I intend to create a DB back end for data storage. A quick investigation has me leaning towards Berkeley DB (open to other suggestions, BDB may be extreme overkill here, but it's the best documented free non-SQL DB I found). My initial implementation plans are for 1 db file per month per symbol, which is much nicer than 1 large file for backup purposes especially with a backup system like Mozy. The key benefit of using a DB on the back end is that it will allow concurrent updates to the data, which would enable data back fill on any time frame instead of an arbitrary time since the last file split being locked out. Once that back fill completes a chart refresh would get you new data. I plan to extend GomFileConverter for back fill capability (which basically comes for free once the recorder indicator is updated).
The following user says Thank You to danjurgens for this post:
Favorite Futures: Gameplay KlownbineŽ Trading of Globex
Posts: 1,276 since Jul 2009
Thanks: 1,227 given,
database for historical bid ask data
danjurgens, I am extremely interested in this and would like to collaborate with you.
I have quite a bit of experience with RDBMS including architecture, administration, coding of front end applications, etc. Have worked with Access, SQLServer, and Oracle. It's been a while, but I eat this stuff up.
I think I'll get a chance to sit down and take my first stab at this on the weekend. For this project I don't really see the benefit of the RDMS since the relational aspects are pretty one dimensional. Symbol(1)->Ticks(N). Berkeley DB should provide all that is needed with Key->Data storage a mapping of symbols to different databases. It manages caching (configurable) and concurrent access, which are the performance and data integrity sticking points.
My background is in embedded real time development so the C# learning curve will slow me down a bit, but I hope to have a prototype working sometime this weekend. I'll post what I come up with as soon as I have it.
On a side note, it would be nice to have a version management system for the Gom files, has anyone considered setting something up on sourceforge or anything? I have a pretty mixed collection where I get some files from NT's forums, updates from here, etc.
Do you mean a chart with 3 CL, each with its own session time settings ?
Depending on which session the recorder is running, it won't send ticks outside of the associated session. For instance if you have a GomCD associated with the CL running in the 9:00-2:30 session, it won't send ticks outside of this session.
Sorry.Probably OT,but gomi presence can be useful for some suggestions.
I' m interested in having in the same panel some CD of the different 'size players' (i.e. on five level <5,6<x<49,50<x<99,100<x<299,x>300 - kind of Volume Splitter on Cumulative Delta with different lines color for different players).
Is there any way to achieve without big revolution on the code?
Can gomi or some other CD expert programmer (Zondor?) give me some hints?
Last edited by paolfili; November 7th, 2010 at 04:59 PM.
I've started working on the DB back-end for gomRecorderIndicator. It got a little more complicated when I figured out that bringing up the DB had to be serialized, that added a lot of complexity if one wanted to keep it as an NT indicator, so I'm taking a slightly different route. I'm in the process of creating it as stand alone server app. I've got the server side up and running, although not really debugged at all. I'm working on the client side now.
The downside is I probably won't have anything I can distribute until next weekend.
Once I do that I'll start a new thread instead of continuing to hijack this one!