Using OnMarketData() on Historical data with a recording engine - futures io
futures io



Using OnMarketData() on Historical data with a recording engine


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one gomi with 34 posts (87 thanks)
    2. looks_two aviat72 with 10 posts (2 thanks)
    3. looks_3 danjurgens with 9 posts (3 thanks)
    4. looks_4 Michael.H with 8 posts (0 thanks)
      Best Posters
    1. looks_one gomi with 2.6 thanks per post
    2. looks_two Zondor with 1 thanks per post
    3. looks_3 danjurgens with 0.3 thanks per post
    4. looks_4 aviat72 with 0.2 thanks per post
    1. trending_up 36,098 views
    2. thumb_up 104 thanks given
    3. group 28 followers
    1. forum 115 posts
    2. attach_file 23 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Using OnMarketData() on Historical data with a recording engine

(login for full post details)
  #1 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Hi guys,

Just wanted to say if you need to be able to call OnMarketData() on historical charts, you can use the GomRecorderIndicator like a base indicator you derive your indicator from.

This base indicator includes a recording/replay engine that acts like a virtual tick data source on historical data.

Then you only have to use GomInitialize, GomOnMarketData and GomOnBarUpdate instead of NT standard ones and you have an indicator that receives historical (and real time) tick info.

It was originally developped with GomCD indicator to access historical Cumulative Delta. I finally managed to make a package out of it ; it contains a very quickly converted version of VolumeProfile that shows how you can get a historical Volume Profile.

See All GomRecorder Indicators - NinjaTrader Support Forum

Edit : Millisec version on topic 31

Edit : Correct time filter version on topic 67

Attached Thumbnails
Click image for larger version

Name:	gompackage.png
Views:	863
Size:	85.1 KB
ID:	5156  
Started this thread Reply With Quote
The following 14 users say Thank You to gomi for this post:

Can you help answer these questions
from other members on futures io?
Indicator from a HTF on a LTF chart
NinjaTrader
Multichart NET set signal parameters different than the …
MultiCharts
need help writing easylanguage radarscreen weekly perfor …
EasyLanguage Programming
1-min Intrabar indicator
MultiCharts
Trading platform with volume profile and delta for free?
Platforms and Indicators
 
Best Threads (Most Thanked)
in the last 7 days on futures io
How much do you know about Bitcoin?
113 thanks
I finally blew up an account
44 thanks
FIO Journal Challenge - April 2021 w/Jigsaw Trading
38 thanks
Big Mike in Ecuador
27 thanks
The tiyfTradePlanFactory indicator
23 thanks
 
(login for full post details)
  #2 (permalink)
 toulouse-lautrec 
Europe
 
Experience: Beginner
Platform: NinjaTrader
 
Posts: 73 since Jun 2009
Thanks: 11 given, 53 received

Gomi, you rock!

Reply With Quote
 
(login for full post details)
  #3 (permalink)
 Big Mike 
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
 
Big Mike's Avatar
 
Posts: 50,093 since Jun 2009
Thanks: 32,554 given, 98,581 received


gomi,

Yes, this is a fantastic set of tools.
Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/

Visit other sites? Please spread the word about your experience with our community!
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #4 (permalink)
 cclsys 
Sydney, NS
 
Experience: Intermediate
Platform: Ninja
Broker: Zen-Fire
Trading: TF,S,GC
 
cclsys's Avatar
 
Posts: 607 since Nov 2009
Thanks: 248 given, 392 received


gomi View Post
Hi guys,

Just wanted to say if you need to be able to call OnMarketData() on historical charts, you can use the GomRecorderIndicator like a base indicator you derive your indicator from.

This base indicator includes a recording/replay engine that acts like a virtual tick data source on historical data.

Then you only have to use GomInitialize, GomOnMarketData and GomOnBarUpdate instead of NT standard ones and you have an indicator that receives historical (and real time) tick info.

It was originally developped with GomCD indicator to access historical Cumulative Delta. I finally managed to make a package out of it ; it contains a very quickly converted version of VolumeProfile that shows how you can get a historical Volume Profile.

See All GomRecorder Indicators - NinjaTrader Support Forum

Gomi: are you saying you can use some of your code in other indicators so that they too can record? If so, could you please provide a snippet example so I can understand. Or are you saying that if you run Gom Recorder in a workspace that all other indicators can access the file somehow? (Please excuse my confusion.)

Also: is there anywhere where your indicators are explained, i.e. what are they saying? Or is it just something you figure out intuitively by using them regularly. For example: I do not really understand 'delta volume' yet. Clearly I can do some study using Google etc. but perhaps you could point me to a thread where you discussed this in the context of your indicators, or perhaps you know of a good basic source for me to start with. Now that I have your tools, I want to spend some time getting into market profile type stuff. In fact, the reason I stumbled on this forum a short while ago was because of the thread about 'using volume in trading' which is what interests me most right now.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #5 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


cclsys View Post
Gomi: are you saying you can use some of your code in other indicators so that they too can record?

Yes. Basically you only have to derive your indicator from GomRecorderIndicator and use GomInitialize, GomOnMarketData and GomOnBarUpdate and you're done. That's how I converted NT's VolumeProfile into GomVolumeProfile



cclsys View Post
If so, could you please provide a snippet example so I can understand.

You can check GomVWAP, it's very basic



cclsys View Post
Or are you saying that if you run Gom Recorder in a workspace that all other indicators can access the file somehow? (Please excuse my confusion.)

All indicators derived from GomRecorder share the same files. You have to set write=true on the one that will have writing privileges, and all the others can read from the file



cclsys View Post
Also: is there anywhere where your indicators are explained, i.e. what are they saying? Or is it just something you figure out intuitively by using them regularly. For example: I do not really understand 'delta volume' yet. Clearly I can do some study using Google etc. but perhaps you could point me to a thread where you discussed this in the context of your indicators, or perhaps you know of a good basic source for me to start with. Now that I have your tools, I want to spend some time getting into market profile type stuff. In fact, the reason I stumbled on this forum a short while ago was because of the thread about 'using volume in trading' which is what interests me most right now.

All the stuff have more or less to do with the delta calculation. You can check fulcrumtrader on google for info on using cumulative delta, and of course market delta for ladder stuff.

Started this thread Reply With Quote
The following 3 users say Thank You to gomi for this post:
 
(login for full post details)
  #6 (permalink)
 dnof 
san francisco, ca
 
Experience: Intermediate
Platform: NinjaTrader
 
Posts: 45 since Nov 2009
Thanks: 42 given, 27 received

Gomi,

A quick question on the difference between the two onMarket methods. What is the functional difference between the two - does the first one have +/- value indicated only where as the second method has the tick timestamp? - I guess I am trying to understand the functional difference?

Dean

Reply With Quote
 
(login for full post details)
  #7 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

There is no difference in the data.

With the withtime method, you get the time of the tick with the other data .

The recorder bars are reconstructed using the close time of the bar, so the recorder engine sends all the ticks that have time stamp < time[0]. So for ninja, all ticks arrive at time[0]. You can use the withtime method if you need more detailed tick time information.

Other people have used it to implement a more-precise-than-1-second tick engine by using the Flat format and storing milliseconds information.
Delta Buy/Sell Volume - Page 4 - NinjaTrader Support Forum , message 591 and later

Started this thread Reply With Quote
 
(login for full post details)
  #8 (permalink)
surfer
Charkov
 
 
Posts: 4 since Jan 2010
Thanks: 2 given, 0 received

Can you say your idia to calculate Ask/Bid using only ticks?

Reply With Quote
 
(login for full post details)
  #9 (permalink)
 dnof 
san francisco, ca
 
Experience: Intermediate
Platform: NinjaTrader
 
Posts: 45 since Nov 2009
Thanks: 42 given, 27 received

Gomi,

I am sure that I am missing something basic, but I have tried converting a indicator I use today, and it seems to function fine (it calculates, displays on charts as expected), but I get no data file created. I see "Recording Binary OK" but no file is created in the GOMFOLDER path specified in my environment settings (personal env).

What else needs to be done or looked at to evaluate?

(BTW - I implemented indicator using the GomRecorderIndicator as you describe with the associated changes to the methods).




gomi View Post
Yes. Basically you only have to derive your indicator from GomRecorderIndicator and use GomInitialize, GomOnMarketData and GomOnBarUpdate and you're done. That's how I converted NT's VolumeProfile into GomVolumeProfile



You can check GomVWAP, it's very basic




All indicators derived from GomRecorder share the same files. You have to set write=true on the one that will have writing privileges, and all the others can read from the file




All the stuff have more or less to do with the delta calculation. You can check fulcrumtrader on google for info on using cumulative delta, and of course market delta for ladder stuff.


Reply With Quote
 
(login for full post details)
  #10 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


by default the files are stored in your My Documents folder. You can use GOMFOLDER to change that location but to be effective you have to restart ninja trader.

Started this thread Reply With Quote
 
(login for full post details)
  #11 (permalink)
 cclsys 
Sydney, NS
 
Experience: Intermediate
Platform: Ninja
Broker: Zen-Fire
Trading: TF,S,GC
 
cclsys's Avatar
 
Posts: 607 since Nov 2009
Thanks: 248 given, 392 received

Gomi or anyone else. I have tried to apply the Recorder to EdsVolPriceBars but it won't accept the GomOnMarketUpdate so I am stumped. It runs as is (without it inserted) but doesn't record.

The relevant line (I think) is:

"
protected override void OnMarketData(MarketDataEventArgs e)

"

I have tried putting in:
"
protected override void GomOnMarketData(MarketDataEventArgs e)
"
to no avail.

Any suggestions?

Of old the skilled first made themselves invincible to await the enemy's vincibility.
Invincibility lies in oneself. Vincibility lies in the enemy.
Thus the skilled can make themselves invincible.
They cannot cause the enemy's vincibility.
Thus it is said: 'Victory can be known; it cannot be made.'
Attached Files
Register to download File Type: zip EDSVolPriceAsh.zip (13.6 KB, 42 views)
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to cclsys for this post:
 
(login for full post details)
  #12 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Method signature is different : it's

protected virtual void GomOnMarketData(TickTypeEnum tickType,double price,int volume,bool firstTickOfBar)

ticktypeenum is
{ BelowBid,AtBid,BetweenBidAsk,AtAsk,AboveAsk,Unknown}

Started this thread Reply With Quote
 
(login for full post details)
  #13 (permalink)
 cclsys 
Sydney, NS
 
Experience: Intermediate
Platform: Ninja
Broker: Zen-Fire
Trading: TF,S,GC
 
cclsys's Avatar
 
Posts: 607 since Nov 2009
Thanks: 248 given, 392 received

Thanks.

(wow. what a fast response!)

But what do I do with TickTypeEnum. Where does it go?

Of old the skilled first made themselves invincible to await the enemy's vincibility.
Invincibility lies in oneself. Vincibility lies in the enemy.
Thus the skilled can make themselves invincible.
They cannot cause the enemy's vincibility.
Thus it is said: 'Victory can be known; it cannot be made.'
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #14 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

you have to do your bid/ask stuff with it.

example :

 
Code
int delta=0;
            
if ((tickType==TickTypeEnum.BelowBid)||(tickType==TickTypeEnum.AtBid))
     delta =-volume;
else if ((tickType==TickTypeEnum.AboveAsk)||(tickType==TickTypeEnum.AtAsk))
      delta=volume;

Started this thread Reply With Quote
 
(login for full post details)
  #15 (permalink)
 cclsys 
Sydney, NS
 
Experience: Intermediate
Platform: Ninja
Broker: Zen-Fire
Trading: TF,S,GC
 
cclsys's Avatar
 
Posts: 607 since Nov 2009
Thanks: 248 given, 392 received

Thanks, Gomi. I'll see if I can work that sort of thing into the existing code, but I think it's going to be beyond my paygrade since his code is for a different enum altogether and I doubt I'll be able to transliterate it all.

Have had a real hard time with the recording engine in general with your indies. Sometimes it seems to work and others it doesn't but I haven't been sharp enough to notice what the difference in my settings is between one and t'other. I hate losing the plots if/when I make a change to a chart.

Of old the skilled first made themselves invincible to await the enemy's vincibility.
Invincibility lies in oneself. Vincibility lies in the enemy.
Thus the skilled can make themselves invincible.
They cannot cause the enemy's vincibility.
Thus it is said: 'Victory can be known; it cannot be made.'
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #16 (permalink)
ptd26
Portland, Oregon
 
 
Posts: 27 since Jan 2010
Thanks: 3 given, 2 received

Is there an indicator that can be run on historical data which simply dumps every tick into a text file? Basically looking for a way to extract the market data outside of Ninja's proprietary logging format. If so, what's the granularity/resolution of the timestamps on the ticks? Are the ticks raw of aggregated?

Reply With Quote
 
(login for full post details)
  #17 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

You must be sure you have "Recording OK".
And remember Gompackage1.1 is very more robust to "changes to the chart"



cclsys View Post
Thanks, Gomi. I'll see if I can work that sort of thing into the existing code, but I think it's going to be beyond my paygrade since his code is for a different enum altogether and I doubt I'll be able to transliterate it all.

Have had a real hard time with the recording engine in general with your indies. Sometimes it seems to work and others it doesn't but I haven't been sharp enough to notice what the difference in my settings is between one and t'other. I hate losing the plots if/when I make a change to a chart.


Started this thread Reply With Quote
 
(login for full post details)
  #18 (permalink)
 cclsys 
Sydney, NS
 
Experience: Intermediate
Platform: Ninja
Broker: Zen-Fire
Trading: TF,S,GC
 
cclsys's Avatar
 
Posts: 607 since Nov 2009
Thanks: 248 given, 392 received

Gomi, thanks for reply. Just want to make something clear: your work on those indies was exceptional. The fact that I can't play with certain aspects might be personally frustrating, but I do not expect you to be able to provide customer service for them! That said, if anyone else wants to get EDS pricevolume bars 'Gomified' (i.e. recordable), feel free! Definitely beyond my pay grade programming-wise.

As to the recording, Gomi, I have package 1.1 but it seems to elude me the recording business. It's no big deal. Because I am running 32 bit OS it seems that using these indies for more than a few (2-3) hours is not advisable anyway because of memory/processing problems so I have reluctantly phased out of loading them and am back to more simple CalconClose indies which are less stress on the system (but which also seem to start overtaxing NT after 4-6 hours). One day when I grow up and have 24 GIG ram.....

Of old the skilled first made themselves invincible to await the enemy's vincibility.
Invincibility lies in oneself. Vincibility lies in the enemy.
Thus the skilled can make themselves invincible.
They cannot cause the enemy's vincibility.
Thus it is said: 'Victory can be known; it cannot be made.'
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #19 (permalink)
 Big Mike 
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
 
Big Mike's Avatar
 
Posts: 50,093 since Jun 2009
Thanks: 32,554 given, 98,581 received

I'm just now having some time to start working with this.

I have a question.

We can use this for backtesting yes? So for instance, if I normally want to do a MarketIfTouch scenario and had to use OnMarketData(), I can now use your method, and provided the recorded data is present on my system, I can recreate the proper events even during a backtest, yes?

If so, I need to setup another VM and start recording a bunch of instruments using GomRecorder... so that in a year I can use this

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/

Visit other sites? Please spread the word about your experience with our community!
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #20 (permalink)
 subterfuge 
uk
 
Experience: Intermediate
Platform: ninja
Broker: amp
Trading: ES
 
Posts: 169 since Aug 2009
Thanks: 112 given, 80 received

Hi Gom. I've seen a few screenshots of people using 'gomvolumeladder'. I cant seem to find it though. Is it in the ninjatrader thread you linked to in your OP somewhere? thanks

Reply With Quote
 
(login for full post details)
  #21 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Hi,

It's here in VIP section


subterfuge View Post
Hi Gom. I've seen a few screenshots of people using 'gomvolumeladder'. I cant seem to find it though. Is it in the ninjatrader thread you linked to in your OP somewhere? thanks


Started this thread Reply With Quote
 
(login for full post details)
  #22 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Yes I'm pretty sure this will work. I can post my current ES 0310 if you want to test.



Big Mike View Post
I'm just now having some time to start working with this.

I have a question.

We can use this for backtesting yes? So for instance, if I normally want to do a MarketIfTouch scenario and had to use OnMarketData(), I can now use your method, and provided the recorded data is present on my system, I can recreate the proper events even during a backtest, yes?

If so, I need to setup another VM and start recording a bunch of instruments using GomRecorder... so that in a year I can use this

Mike


Started this thread Reply With Quote
The following user says Thank You to gomi for this post:
 
(login for full post details)
  #23 (permalink)
 dnof 
san francisco, ca
 
Experience: Intermediate
Platform: NinjaTrader
 
Posts: 45 since Nov 2009
Thanks: 42 given, 27 received


Big Mike View Post
I'm just now having some time to start working with this.

I have a question.

We can use this for backtesting yes? So for instance, if I normally want to do a MarketIfTouch scenario and had to use OnMarketData(), I can now use your method, and provided the recorded data is present on my system, I can recreate the proper events even during a backtest, yes?

If so, I need to setup another VM and start recording a bunch of instruments using GomRecorder... so that in a year I can use this

Mike

Gomi,

Is there any reason we could not use replay data to "backfill" some of this? It would be good to be able to do this.

I am also thinking of a slightly different approach but still in the concept/investigation phase so once I get a bit more crystalized I'll post back.

Reply With Quote
 
(login for full post details)
  #24 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Backfill with market replay works but the recorder won't write data previous to the last date/time written in the file. So it can't fill gaps.

But for that you can
  • convert the file to a human readable format with the file converter (flat format)
  • using large file editor like emeditor, you edit the file and remove the end of the file, until beginning of the gap (remember Time Zone is UTC)
  • populate the file with a market replay
  • reconvert it to binary (if necessary) and reload

Started this thread Reply With Quote
 
(login for full post details)
  #25 (permalink)
 GoldStandard 
arizona
 
Experience: Intermediate
Platform: rolling my own
Trading: ES,CL,GC,6E
 
GoldStandard's Avatar
 
Posts: 211 since Oct 2009
Thanks: 324 given, 189 received


gomi View Post
Yes I'm pretty sure this will work. I can post my current ES 0310 if you want to test.

Does this mean we don't absolutely need NT to fix their bid/ask ordering in order to code strategies that use bid/ask data for indicators and filling orders more accurately..... Is it possible we could use COMCD data for this?

Recording all the data and making it available would be a challenge, but perhaps not as insurmountable a challenge as getting NT to change course of their development.

Reply With Quote
 
(login for full post details)
  #26 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


GoldStandard View Post
Does this mean we don't absolutely need NT to fix their bid/ask ordering in order to code strategies that use bid/ask data for indicators and filling orders more accurately..... Is it possible we could use COMCD data for this?

Recording all the data and making it available would be a challenge, but perhaps not as insurmountable a challenge as getting NT to change course of their development.

After some thinking I don't think this will work, because I don't think you can get intrabar fills on historical backtest. It would be logical you are always filled on next bar, so I don't think you can simulate intrabar fills.

Or you could completely monitor trades by yourself without EnterLong() or EnterShort(), outputting PnL in the console. But bye bye fancy stats and optimizer.
Sorry :-(

Started this thread Reply With Quote
The following user says Thank You to gomi for this post:
 
(login for full post details)
  #27 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Hi guys,

Just wanted to mention I updated the package to 1.3 on ninja web site.
Some bug correction, a few upgrades and added a delta momentum indicator

Cheers

Attached Thumbnails
Click image for larger version

Name:	gompackage1.2.png
Views:	203
Size:	77.4 KB
ID:	10176  
Started this thread Reply With Quote
The following user says Thank You to gomi for this post:
 
(login for full post details)
  #28 (permalink)
 syxforex 
British Columbia
 
Experience: Advanced
Platform: NINJA
Broker: ZEN
Trading: Crude
 
Posts: 1,091 since May 2010

Hi Gomi,

First let me just say thank you for your generosity in sharing your work with everybody in the NT world. I don't know if I can figure out how to use these codes with NT so I can't even imagine how long it took you to build them. I just switched over to NT7 from MT4 and I'm quite surprised at how long it takes to load charts when all I have on them is a few exponential stochastics and bollinger bands, wow, could I do it with a calculator that fast.

I don't think I'm going to risk the rather fragile and seemingly shaky state of the platform at present. It seems she can't take much more already. Adding custom codes sounds like a lot of fun and I'm wishing I could check it out. I did a demo of Market Delta and was quite impressed, is the Gomi CD similar? And why does the recorder need to be on 24/7 if Zen Fire saves tick data already. What does one do if they experience a power or inet outage, as I do on a regular basis?

Many Thanks,




gomi View Post
Hi guys,

Just wanted to mention I updated the package to 1.3 on ninja web site.
Some bug correction, a few upgrades and added a delta momentum indicator

Cheers


Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #29 (permalink)
 syxforex 
British Columbia
 
Experience: Advanced
Platform: NINJA
Broker: ZEN
Trading: Crude
 
Posts: 1,091 since May 2010

Is this thread still going? I have some big areas of missing data in my 6E Gomi Folder. Can't seem to find Ztrader's post about backfilling the GomFolder with the replay function. Can anybody point me in the right direction?

thanks..



dnof View Post
Gomi,

Is there any reason we could not use replay data to "backfill" some of this? It would be good to be able to do this.

I am also thinking of a slightly different approach but still in the concept/investigation phase so once I get a bit more crystalized I'll post back.


Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #30 (permalink)
 ZTR 
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Here it is:



Fixed this to show up in a downloads search & added keywords. Thought I did this before the webinar???

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to ZTR for this post:
 
(login for full post details)
  #31 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Here's a new version of the Recorder.

Added

File Formats


* Ability to read directy QCollector , QCollectorIQ and IRT files
* Millisec format, which will try to use the current clock to record millisecond timestamps. The recorder will try to adapt the lag between the exchange and the PC to keep timestamps matched (see for an indy that monitors the lag). And thanks Zondor, there was indeed a bug ;-)

Methods

Added GomOnBarUpdateDone method to allow better performance.

So now the whenever OnBarUpdate is called on the indicator, it calls first GomOnBarUpdate, then it does its internal stuff like reading Gom files, and when everything is finished it calls GomOnBarUpdateDone

Ideally, you should put in GomOnBarUpdate the code thas is used to initialize each new bar, then in GomOnMarketData the code that updates your own data objects, and in GomOnBarUpdateDone the code that plots your internal objects by setting values to time series.

See enclosed GomDeltaVolume to see example implementation.

Attached Thumbnails
Click image for larger version

Name:	millisec.png
Views:	306
Size:	39.2 KB
ID:	15178  
Attached Files
Register to download File Type: cs GomRecorderIndicator.cs (67.3 KB, 130 views)
Register to download File Type: cs GomDeltaVolume.cs (8.7 KB, 112 views)
Started this thread Reply With Quote
The following 12 users say Thank You to gomi for this post:
 
(login for full post details)
  #32 (permalink)
 zikonc 
san ramon
 
Experience: Beginner
Platform: ninja
Broker: zen
Trading: es, 6e
 
Posts: 136 since Oct 2009
Thanks: 98 given, 76 received


gomi View Post

So now the whenever OnBarUpdate is called on the indicator, it calls first GomOnBarUpdate, then it does its internal stuff like reading Gom files, and when everything is finished it calls GomOnBarUpdateDone.

Hi Gomi, would you say that this new function GomOnBarUpdateDone would help VolumeLadder?.......if so, perhaps you can tell us where in the code you would add it ...... thanks a bunch .....

Reply With Quote
 
(login for full post details)
  #33 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

no the ladder uses it own plot methods. It's more for indicators that use standard plotting methods and Ninja dataseries.

The problem is that if you put your plotting code in GomOnMarketData you will end plotting too often. So it's best to put it in GomOnBarUpdateDone.

Started this thread Reply With Quote
The following user says Thank You to gomi for this post:
 
(login for full post details)
  #34 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

my chart doesn't say gom millisecond recorder on it. Im not using the delta volume indicator, just the gom cd. I know before you didn't have to actually load the volume recorder on the chart, but do you have to on this one?

Reply With Quote
 
(login for full post details)
  #35 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

no, but did you launch a recompilation using F5 on any indicator ?

Started this thread Reply With Quote
 
(login for full post details)
  #36 (permalink)
 rsi77 
Chicago
 
Experience: Intermediate
Platform: Ninja Trader & TradeStation
Broker: Amp Futures/Zen-Fire
Trading: ES
 
Posts: 56 since Jun 2009
Thanks: 17 given, 48 received

Hi Gomi,
Will new indicators need to be written or updated for this recorder, or will the current GOM package benefit from this recorder. (probably a stupid question but I don't quite understand how the GOM stuff works...I'm just glad it does!)

Ron

Reply With Quote
 
(login for full post details)
  #37 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

I actually went to edit ninjascripts, and did a compile manually. I then did f5 on all my charts, still won't say millisecond recorder. its says using binary on the side. Here's a pic


Gomi.. sorry.. stupid mistake. I didn't choose milisecond on the bottom. Few question...
1. will it replace the old bid/ask data if i switch formats to millisecond? found out it creates a new file... so disregard.
2. Is there a file converter that allows you to convert old binary data to be ported to millisecond data so it can use it as a lookback?
3. Also, if I have q collector, will it read it as soon as q collector is finished downloading it( after hitting F5 of course)? I don't have q collector yet. Reason i ask is simply because technically, it takes a few seconds to download data, and with this feature, there's no longer a need to leave computer on running 24/5
4.If we use q collector to backfill, can it ammend to millisecond data or is it a totally different file format (meaning you download for missing data, then as new data comes in using the recorder, that picks off where the qcollector left of)?

Is there a file format that you actually prefer or think works best?




this is truely amazing, you managed to overcome ninja's greatest problem, by yourself.....
I was talking to tim about this.... amazing how one person can do so much, but a team of coders at ninja couldn't fix this....

Attached Thumbnails
Click image for larger version

Name:	Capture.PNG
Views:	179
Size:	19.5 KB
ID:	15284  
Reply With Quote
 
(login for full post details)
  #38 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


rsi77 View Post
Hi Gomi,
Will new indicators need to be written or updated for this recorder, or will the current GOM package benefit from this recorder. (probably a stupid question but I don't quite understand how the GOM stuff works...I'm just glad it does!)

Ron

No nothing needs to be rewritten. All current indicators can benefit from this one.

Started this thread Reply With Quote
 
(login for full post details)
  #39 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


Michael.H View Post
1. will it replace the old bid/ask data if i switch formats to millisecond? found out it creates a new file... so disregard.
2. Is there a file converter that allows you to convert old binary data to be ported to millisecond data so it can use it as a lookback?
3. Also, if I have q collector, will it read it as soon as q collector is finished downloading it( after hitting F5 of course)? I don't have q collector yet. Reason i ask is simply because technically, it takes a few seconds to download data, and with this feature, there's no longer a need to leave computer on running 24/5
4.If we use q collector to backfill, can it ammend to millisecond data or is it a totally different file format (meaning you download for missing data, then as new data comes in using the recorder, that picks off where the qcollector left of)?

Is there a file format that you actually prefer or think works best?

2. no. Millisecond data has to be written real time. It may actually work though if you use an input file (IRT, QCollector etc ) and your input file has milliseconds timestamps.
3. YOu will have a problem because the indicators keep a lock on the file so it won't be updateable by QCollector
I wanted to update the indicator with all the nice job of gooni ( ) but actually forgot;-)
Anyway, for now, you can't update your file using QCollector while it's used by any Gom indicator
4. I don't think QCollector does millisec.
5. File format preference : this millisec stuff is for guys that want to play around with millisec data. It may not be usable at all(check your network jitter with then monitoring app). If you don't need it you can stick to binary

Careful anyway using directyl QCollector files : in my case with esignal, 1 week is 500Mo, so parsing of the file is very long. Binary uses a few Mo for the same week, so you can read contract-long data with not too much pain.

Started this thread Reply With Quote
 
(login for full post details)
  #40 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Here's a new Converter that handles Millisec format.

You also have a command line mode using
GomFileConverter.exe inputfile inputformat outputformat ticksize

I also included the VS2010 project

Attached Files
Register to download File Type: zip GomFileConverterProject.zip (7.86 MB, 261 views)
Register to download File Type: zip GomFileConverter.zip (17.2 KB, 200 views)
Started this thread Reply With Quote
The following 8 users say Thank You to gomi for this post:
 
(login for full post details)
  #41 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

so if we use the millisecond version, there's no improvement in any way when compared to the binary format? The updates that you mentioned, that works with the binary format as well ?
( ie GomOnBarUpdateDone method to allow better performance).


I'm assuming based on your reply that the gom millisecond data is only to check for network jitter?

Sorry if these questions seem redundant, just try to figure out the best way to go at it. I also see the new converters on there, thank you.

For people playing around with the q collector version file format, i talked to tim and he said you can have it update every 5 seconds, but its not suitable for day trading. Its also a bit slow.. sorry guys.

Reply With Quote
 
(login for full post details)
  #42 (permalink)
 ZTR 
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Hi Gomi,

BigMike is having a programmer put together a new downloads module. I would like your permission, once it is complete, to begin putting all of your great work into this as my test/debugging project. The new downloads section will be thread like and have a lot of easy to use features. Including auto update notification and places for comments.

Am trying to envision a way to combined this with Sam BMCodex Project as a way to organize large coding projects.

More to come as I begin to get familiar with the new GUI.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #43 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


Michael.H View Post
so if we use the millisecond version, there's no improvement in any way when compared to the binary format?

No. It's only useful if you design millisecond indicators.


Michael.H View Post
The updates that you mentioned, that works with the binary format as well ?
( ie GomOnBarUpdateDone method to allow better performance).

Yes


Michael.H View Post
I'm assuming based on your reply that the gom millisecond data is only to check for network jitter?

no, it records millisecond timestamps, but it is only useful if you have a more or less constant network delay (so low jitter)


ZTR View Post
I would like your permission, once it is complete, to begin putting all of your great work into this as my test/debugging project.

Yes, no problem. I'm honored ;-)

Started this thread Reply With Quote
The following 2 users say Thank You to gomi for this post:
 
(login for full post details)
  #44 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

Gotcha, after reviewing some videos, the indicator shows a roughly a negative ~15000-16800 ms lag on es and ym with my dtn feed and internet connection. Is this something that the indicator might be useful for, or is that about standard in your opinion? My pc lock is pretty accurate. I've read your other posts, and noticed no delay( at least so far)


Thanks again.

Reply With Quote
 
(login for full post details)
  #45 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received

I tried to use IRT format but does not working nor in reading nor in recording. I have tried anything but still not working.

What is the prob?

Thanks.

Luke.

Take your Pips, go out and Live.
Luke.
Attached Thumbnails
Click image for larger version

Name:	IRT.png
Views:	139
Size:	40.7 KB
ID:	15404  
Reply With Quote
 
(login for full post details)
  #46 (permalink)
 slowpork 
Los Angeles, CA
 
Experience: Beginner
Platform: Nintendo Wii w/ Nunchuk Controller, MarketDelta, Ninja
Broker: MBtrading/Mirus/IQFeed
Trading: Ibanez, Gibson, GBP/JPY, 6E, ES
 
Posts: 4 since May 2010
Thanks: 2 given, 8 received

hi Luke.

Not sure if you try these steps from MarketDelta.

- go to File / Export / History Data
- choose the symbol, select range and export
- save as filename "ES 09-10.IRT.txt" (or "ES 06-10.IRT.txt") under "My Documents"

For IRT, it's read-only. So to have data all in one place is to use Gom file convertor. *(i'm testing)

Slowpork

Reply With Quote
 
(login for full post details)
  #47 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received


slowpork View Post
hi Luke.

Not sure if you try these steps from MarketDelta.

- go to File / Export / History Data
- choose the symbol, select range and export
- save as filename "ES 09-10.IRT.txt" (or "ES 06-10.IRT.txt") under "My Documents"

For IRT, it's read-only. So to have data all in one place is to use Gom file convertor. *(i'm testing)

Slowpork


Ja, I done it first.

Thanks.

Luke.

Take your Pips, go out and Live.
Luke.
Reply With Quote
 
(login for full post details)
  #48 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

While trying to develop something similar to your Volume Ladder, I felt that CalculateOnBarClose should be true, since all the live data is based on MarketData event. However, when rebuilding the information, I had an off by one error since the bar on which the data-was to be plotted was already completed before the Gom functions were call. I offset it by one, but then faced a problem when transitioning from historical to live data. The bar which is currently being built was part historical and part real-time.

Will the new updates help in the transition? I am not sure whether the setting for the CalculateOnBarClose would make any difference in overall performance for the Gom based indicators but wanted to optimize as much as I could.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #49 (permalink)
 Prtester 
SE-Asia
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Amp
Trading: ES
 
Posts: 148 since Jun 2009
Thanks: 64 given, 122 received


gomi View Post
Here's a new version of the Recorder.

Added

File Formats


* Millisec format, which will try to use the current clock to record millisecond timestamps. The recorder will try to adapt the lag between the exchange and the PC to keep timestamps matched (see for an indy that monitors the lag). And thanks Zondor, there was indeed a bug ;-)

Was so busy that i just don't see this before, cool, I play with it when have time, Thanks!

Reply With Quote
 
(login for full post details)
  #50 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

The millisec file format has time stamps with millisecond resolution instead of one second resolution. For most indicators that use the GomRecorder, this is not necessary, and unless otherwise stated, the preferred file format is generally binary.

The millisecond time stamps are helpful for indicators that do things such as counting the number of trades per unit of time. If the unit of time is less than one second, or for that matter less than a few seconds, one second time stamps would be too coarse to give meaningful results. For example, trying to determine the average volume traded within each of the 0.8 second time periods within a bar. So this is a great feature for indicators that measure TRADE INTENSITY.

For best results it's important to keep your computer clock synchronized with a reference clock using an NTP client such as Meinberg.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #51 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

I went on their website, but there's no executable file. I know that this doesn't have to do with the gom recorder, but is anyone willing to share how its setup?

In terms of Gomi's work, it doesn't seem i can use it anymore anyways because of all the issues with ninja. Im forced to use a different platform for now.

Reply With Quote
 
(login for full post details)
  #52 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


aviat72 View Post
While trying to develop something similar to your Volume Ladder, I felt that CalculateOnBarClose should be true, since all the live data is based on MarketData event. However, when rebuilding the information, I had an off by one error since the bar on which the data-was to be plotted was already completed before the Gom functions were call. I offset it by one, but then faced a problem when transitioning from historical to live data. The bar which is currently being built was part historical and part real-time.

Will the new updates help in the transition? I am not sure whether the setting for the CalculateOnBarClose would make any difference in overall performance for the Gom based indicators but wanted to optimize as much as I could.

I think you should keep CalculateOnBarClose to false... If you override Plot() method then new version is of no importance, but fo r best performance(on historiocla data) if you use OnBarUpdate-like code you should put bar init code in GomOnBarUpdate() and plotting code in GomOnBarUpdateDone()

Started this thread Reply With Quote
 
(login for full post details)
  #53 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


LukeGeniol View Post
Ja, I done it first.

Thanks.

Luke.

Does it look like
-SYMBOL DATE PRICE TICKVOL BID ASK
@ES# 04/20/2009 16:30:00 831.50000 50 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 32 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 30 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 25 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 25 831.50000 831.50000

Started this thread Reply With Quote
 
(login for full post details)
  #54 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received


gomi View Post
Does it look like
-SYMBOL DATE PRICE TICKVOL BID ASK
@ES# 04/20/2009 16:30:00 831.50000 50 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 32 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 30 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 25 831.50000 831.50000
@ES# 04/20/2009 16:30:00 831.50000 25 831.50000 831.50000


ja but the date is in different format M/D/Y.

Luke.

Take your Pips, go out and Live.
Luke.
Reply With Quote
 
(login for full post details)
  #55 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

So it should work then.... Dunno if anybody else has the problem...

Started this thread Reply With Quote
 
(login for full post details)
  #56 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received


gomi View Post
So it should work then.... Dunno if anybody else has the problem...

sorry, I see now that the date is in the same format.....ops

Thanks.

Luke.

Take your Pips, go out and Live.
Luke.
Reply With Quote
 
(login for full post details)
  #57 (permalink)
 ZTR 
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received

Is there a way to force the Gom Recorder to write all buffered data before a shut down?

I was sending Gom Data to a collaborator and then restated & see this. If I do not restart then the file stored has a much earlier time stamp & was uncertain if all data accumulated would be included in the sent file.

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Attached Thumbnails
Click image for larger version

Name:	CL 08-10 (2048 Volume)  7_8_2010_1905.jpg
Views:	165
Size:	238.8 KB
ID:	16707  
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #58 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Hi,

I don't know why this happens because the recorder doesn't buffer data : it flushes after every tick
bw.Flush();

Started this thread Reply With Quote
The following user says Thank You to gomi for this post:
 
(login for full post details)
  #59 (permalink)
 FBJS 
Toronto
 
Experience: Advanced
Platform: NinjaTrader
 
Posts: 108 since Sep 2009
Thanks: 30 given, 191 received


cclsys View Post
Gomi or anyone else. I have tried to apply the Recorder to EdsVolPriceBars but it won't accept the GomOnMarketUpdate so I am stumped. It runs as is (without it inserted) but doesn't record.

The relevant line (I think) is:

"
protected override void OnMarketData(MarketDataEventArgs e)

"

I have tried putting in:
"
protected override void GomOnMarketData(MarketDataEventArgs e)
"
to no avail.

Any suggestions?

I'm just wondering: did you ever get this working, or did anyone? I am interested in a Gomi version of this particular indicator, if it exists...

Thanks!

Reply With Quote
 
(login for full post details)
  #60 (permalink)
 RJay 
Hartford, CT. USA
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG, Kinetick
Trading: RTY
 
RJay's Avatar
 
Posts: 688 since Jun 2009
Thanks: 748 given, 781 received

Hi All,

Trying out the recorder from post 31.

Its recording OK with binary setting but will not record data with millisec setting.

Does anyone have this working in NT7???

Thanks,

RJay


UPDATE:

Well, I got it working sort of.

If I record using GomCD indicator, then the historic data can be accessed with my indicators.

Don't know why... But it works.

Reply With Quote
 
(login for full post details)
  #61 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

I have a question of the millisecond time-stamp.

When reloading historical data, does NT supply the millisecond at which the bar finished in Time[0] for the new precision of the GomRecorder to be effective?

On a different note, I presume if you have IRT or MD, you can then download the data from IQFeed using that tool and use them as input for GOM files. That way you will not have to have the recorder running 24/7.

Ideally what would be ideal if you could get the files up to date real-time. NT will occasionally hang and while you are restarting, you lose the ability to recreate the data intra-day.

Now all this will not be necessary if NT could supply ticks for historical data .... But then we will not be thanking Gomi.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #62 (permalink)
tarantino
Willowbrook, IL
 
 
Posts: 32 since Sep 2010
Thanks: 71 given, 39 received


gomi View Post
Hi,

I don't know why this happens because the recorder doesn't buffer data : it flushes after every tick
bw.Flush();


Does anybody know if there is a way to convert NT Market Replay Data to Gomi Recorder format?
Or at least anybody tried to do such an utility?

Reply With Quote
 
(login for full post details)
  #63 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received


aviat72 View Post
I have a question of the millisecond time-stamp.

When reloading historical data, does NT supply the millisecond at which the bar finished in Time[0] for the new precision of the GomRecorder to be effective?
................................................

No, the resolution of NT historical data is 1 second, but GomRecorder doesn't use that data, only the recorded data.

Take your Pips, go out and Live.
Luke.
Reply With Quote
 
(login for full post details)
  #64 (permalink)
 LukeGeniol 
Italy (IT)
 
Experience: Advanced
Platform: ATAS, R|Trader, NT8
Broker: Rithmic
Trading: CL, Brent, GC, TF
 
LukeGeniol's Avatar
 
Posts: 1,501 since Apr 2010
Thanks: 408 given, 984 received


tarantino View Post
Does anybody know if there is a way to convert NT Market Replay Data to Gomi Recorder format?
Or at least anybody tried to do such an utility?

Read this post , but I haven't tried it and don't understood if it works or not, but think it doesn't work. lol .

Take your Pips, go out and Live.
Luke.
Reply With Quote
 
(login for full post details)
  #65 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

To access the correct precision you need to use the GomOnMarketDataWithTime method which will provide the millisec timestamp.


LukeGeniol View Post
No, the resolution of NT historical data is 1 second, but GomRecorder doesn't use that data, only the recorded data.


Started this thread Reply With Quote
 
(login for full post details)
  #66 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

No format is unknown( at least to me, I know fin-alg reads it but I don't know how they did it...)



tarantino View Post
Does anybody know if there is a way to convert NT Market Replay Data to Gomi Recorder format?
Or at least anybody tried to do such an utility?


Started this thread Reply With Quote
 
(login for full post details)
  #67 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Here's a new version of the recorder that corrects the time filtering problem some people had on GomMP.

Now ticks should be correcty filtered based on your session settings. You can still have ticks outside the session added on the first bar of the session by setting "Disable time filter" to true.

I promise to soon do a complete update on GomPackage so we don't have to keep chasing the latest versions....

Attached Files
Register to download File Type: cs GomRecorderIndicator.cs (67.4 KB, 90 views)
Started this thread Reply With Quote
The following 11 users say Thank You to gomi for this post:
 
(login for full post details)
  #68 (permalink)
 RJay 
Hartford, CT. USA
 
Experience: Intermediate
Platform: NinjaTrader
Broker: AMP/CQG, Kinetick
Trading: RTY
 
RJay's Avatar
 
Posts: 688 since Jun 2009
Thanks: 748 given, 781 received


gomi View Post
Here's a new version of the recorder that corrects the time filtering problem some people had on GomMP.

Now ticks should be correcty filtered based on your session settings. You can still have ticks outside the session added on the first bar of the session by setting "Disable time filter" to true.

I promise to soon do a complete update on GomPackage so we don't have to keep chasing the latest versions....


Gomi,

Can you please place header line of text in files with the date of the most recent revisions.

Thanks,

RJay

Reply With Quote
 
(login for full post details)
  #69 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

This is a reminder/response to an earlier post in this thread.

Remember that an environment variable has a NAME and a VALUE.

The GomRecorder looks for an environment variable with the name GOMFOLDER.

The VALUE can be any VALID windows path name. The path must point to a folder that actually exists.

I set the value of my GOMFOLDER environment variable to C:\GomData, because that makes the GomData folder easy to find. There must actually be a folder called GomData located directly under the C directory, so that the value of the environment variable will be a VALID path name.

(On one of my computers that has two PHYSICAL disks, the GOMFOLDER points to D:\GomData. Maybe this improves I/O performance a little... maybe not. Worth trying.)

Gomi, thanks for the new recorder !

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #70 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


Zondor View Post
(On one of my computers that has two PHYSICAL disks, the GOMFOLDER points to D:\GomData. Maybe this improves I/O performance a little... maybe not. Worth trying.)

I'm pretty sure I/O use is very low : most systems have writing cache enabled in windows, so most I/O is done in the cache, which flushed much more rarely.

This is also why you could lose tick info if your computer crashed and the write cache is not empty.

Attached Thumbnails
Click image for larger version

Name:	io.png
Views:	137
Size:	165.1 KB
ID:	24123  
Started this thread Reply With Quote
 
(login for full post details)
  #71 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

I think that the optimal setup for GomRecorderIndicators used to write data, is to put them on tick interval charts. There is no need to use writing recorders on any other type of chart than a tick chart. All of my recorders that have write data set to true are on dedicated tick interval recorder charts in one workspace that is not used for anything else, and that runs in the background.

I found that when my recording indicator and data consuming indicator are both on tick charts, I am now getting perfect matches on almost all bars between (buyvolume + sellvolume) and Volume[0]. To get there, I needed to implement RJay's FirstTickOfBar fix for NT7, and also to count the BetwenBidAsk and Unknown ticks, using Gomi's "direction" to determine whether they are buys or sells.

The file created by the GomRecorder needs to be able to be used by data consuming indicators that could be on any type of chart. Charts with intervals of ticks and seconds should work fine. On minute based charts there are some problems due to the lower precision of the time stamps. However you can get around this by using seconds intervals, such as 300 for a 5 minute chart. This forces Ninnie to use the one second time stamps. Gomi pointed this out a long time ago.

With volume interval charts there are problems when a spillover trade with volume of more than one causes the volume counted to go over the nominal interval volume of the chart. Example: We have a 100 volume chart and are up to 98 contracts traded. The next tick has a volume of 25. If all of those are included in the current bar of the volume indicator, it will have a total volume of 123, but Volume [0] will still be 100, as it is for all bars.

The child indicators such as CD, GomDeltaIndicator, etc are where the adjustments should be made on volume based charts, when one trade spills over to the next bar. I don't see how the GomRecorder itself can compensate for this, since it has no way of knowing what kind of chart it will be called upon to supply data to.

When I get around to it I will see if I can code changes to the BuySellVolume indicator and to GomDeltaIndicator to make them work correctly on constant volume charts. I am interested in the GomDeltaIndicator because it is the data source for the GomVolumeLadder, which is often used on volume interval charts.

Regarding range, Renko, etc charts, I don't even want to think about them right now.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Zondor for this post:
 
(login for full post details)
  #72 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

How many days do you load on the tick charts.. Do you set it to ETH or default? Does it have to be on a 1 tick chart?

Reply With Quote
 
(login for full post details)
  #73 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

Zondor:

I am surprised that type of chart you record on should make a difference. The recorder is based on Market Data events and those are kind of independent of the bar logic in NT. Do you have any specific observations which made you arrive at that conclusion or is it based more on empirical observations about what works and what does not?

I made a small change on the volume ladder to add the last tick to the new bar. I do not mess with the old bar since it does not matter that much.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #74 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received


Quoting 
How many days do you load on the tick charts.. Do you set it to ETH or default? Does it have to be on a 1 tick chart?

Assuming that you are asking about the charts that I use for recording: They have an interval of about 100 ticks and only go back about 50 bars, so that they will load fast. To make sure I don't miss anything I usually use the default 24/7 session template.

Before I shut down Ninnie I close all the workspaces except the Recorder workspace. Then it's the only one that loads automagically® when Ninnie restarts.

Then I start up the other workspaces one by one and leave the Recorder one running in the background. The only reason I look at it once in a while is to check that everything is Recording OK.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #75 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Hi guys,

A good news : I've been working to try to improve the tick engine of the recorder, for now I've been able to get 100% perfect match of Vol[0] as compared to gomi volume, for second charts, tick charts, range charts and volume charts (by splitting volume). I also improved the non-splitting volume chart by sending the ticks a bit differently.

A few charts : first indie is Vol[0], second indie is Volume read from Gom tick engine, and third indie is the diff : so should be 0.

Of course this is work in progress...

Attached Thumbnails
Click image for larger version

Name:	secondchart.png
Views:	116
Size:	40.9 KB
ID:	24421   Click image for larger version

Name:	tickchart.png
Views:	107
Size:	41.5 KB
ID:	24423   Click image for larger version

Name:	rangechart.png
Views:	106
Size:	37.7 KB
ID:	24425   Click image for larger version

Name:	volumesplitting.png
Views:	96
Size:	32.3 KB
ID:	24427   Click image for larger version

Name:	oldticksending.png
Views:	98
Size:	54.3 KB
ID:	24429   Click image for larger version

Name:	newticksending.png
Views:	85
Size:	36.7 KB
ID:	24431  
Started this thread Reply With Quote
The following 8 users say Thank You to gomi for this post:
 
(login for full post details)
  #76 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received


gomi View Post
Hi guys,

A good news : I've been working to try to improve the tick engine of the recorder, for now I've been able to get 100% perfect match of Vol[0] as compared to gomi volume, for second charts, tick charts, range charts and volume charts (by splitting volume). I also improved the non-splitting volume chart by sending the ticks a bit differently.

A few charts : first indie is Vol[0], second indie is Volume read from Gom tick engine, and third indie is the diff : so should be 0.

Of course this is work in progress...

Gomi:

I was also thinking of making the gom interface sensitive to the type of bar being used to work around these last tick issues; it is great you are already attempting it.

One of the biggest challenges with the gom indicators is keeping the tick data base current. I was wondering if you would consider a mode where the recorder uses one file per day of data, instead of one file for the entire history of the instrument.

It would be much easier to maintain a clean tick database; you can just download and save the data from your vendor after midnight, while recording the current data live. This will also allow you to update historical data without shutting off the recorder. The biggest challenge I face is keeping the database current, without shutting off the live recorder.

Ideally the on disk file-structure for that will involve instrument specific sub-folders within the GOMFOLDER directory.
I can pitch in some coding time if that helps, but I do not understand all the nuances of the GomRecorder to do it independently.

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to aviat72 for this post:
 
(login for full post details)
  #77 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

That's great news, Gomi.

I also like the idea of one file per day for Gom files, would make it easier to manage the data, fix gaps, etc.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Zondor for this post:
 
(login for full post details)
  #78 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

gomi:

I think we still have a problem. I have a CL template with 3 sessions (6:00PM to 9:AM, 9:00AM to 2:30PM, 2:30 PM to 5:15PM EST). Today (Friday) the last day of the week, the recorder stopped sending ticks at 9:00AM when the session rolled over the pit session.

Earlier on, I used to face problems on Monday morning ES (again a 3 session template). I think that is fixed, but the Friday problem remains.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #79 (permalink)
 Silvester17 
Market Wizard
Columbus, OH
 
Experience: None
Platform: NT 8, TOS
Trading: ES
 
Silvester17's Avatar
 
Posts: 3,587 since Aug 2009
Thanks: 5,079 given, 11,392 received


aviat72 View Post
gomi:

I think we still have a problem. I have a CL template with 3 sessions (6:00PM to 9:AM, 9:00AM to 2:30PM, 2:30 PM to 5:15PM EST). Today (Friday) the last day of the week, the recorder stopped sending ticks at 9:00AM when the session rolled over the pit session.

Earlier on, I used to face problems on Monday morning ES (again a 3 session template). I think that is fixed, but the Friday problem remains.

I had the exact same problem with TF and 3 session template. today is the first day that everything is working fine. don't know if it has something to do with NT7 rc1 version. are you using the same version?

Reply With Quote
 
(login for full post details)
  #80 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received


Silvester17 View Post
I had the exact same problem with TF and 3 session template. today is the first day that everything is working fine. don't know if it has something to do with NT7 rc1 version. are you using the same version?

gomi made some changes in the recorder file. That fixed the monday bug but there is this Friday issue now.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #81 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

I too like the idea of multiple files for the data to allow data updates while running. But it's a complicate scenario that still doesn't help you out if you lose your internet connection temporarily during a particular session.

To that end I intend to create a DB back end for data storage. A quick investigation has me leaning towards Berkeley DB (open to other suggestions, BDB may be extreme overkill here, but it's the best documented free non-SQL DB I found). My initial implementation plans are for 1 db file per month per symbol, which is much nicer than 1 large file for backup purposes especially with a backup system like Mozy. The key benefit of using a DB on the back end is that it will allow concurrent updates to the data, which would enable data back fill on any time frame instead of an arbitrary time since the last file split being locked out. Once that back fill completes a chart refresh would get you new data. I plan to extend GomFileConverter for back fill capability (which basically comes for free once the recorder indicator is updated).

Reply With Quote
The following user says Thank You to danjurgens for this post:
 
(login for full post details)
  #82 (permalink)
 Zondor 
Portland Oregon, United States
 
Experience: Beginner
Platform: Ninjatrader®
Broker: CQG, Kinetick
Trading: Gameplay Klownbine® Trading of Globex
 
Zondor's Avatar
 
Posts: 1,347 since Jul 2009
Thanks: 1,245 given, 2,721 received

danjurgens, I am extremely interested in this and would like to collaborate with you.

I have quite a bit of experience with RDBMS including architecture, administration, coding of front end applications, etc. Have worked with Access, SQLServer, and Oracle. It's been a while, but I eat this stuff up.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #83 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

I think I'll get a chance to sit down and take my first stab at this on the weekend. For this project I don't really see the benefit of the RDMS since the relational aspects are pretty one dimensional. Symbol(1)->Ticks(N). Berkeley DB should provide all that is needed with Key->Data storage a mapping of symbols to different databases. It manages caching (configurable) and concurrent access, which are the performance and data integrity sticking points.

My background is in embedded real time development so the C# learning curve will slow me down a bit, but I hope to have a prototype working sometime this weekend. I'll post what I come up with as soon as I have it.

On a side note, it would be nice to have a version management system for the Gom files, has anyone considered setting something up on sourceforge or anything? I have a pretty mixed collection where I get some files from NT's forums, updates from here, etc.

Dan

Reply With Quote
 
(login for full post details)
  #84 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received


aviat72 View Post
gomi:

I think we still have a problem. I have a CL template with 3 sessions (6:00PM to 9:AM, 9:00AM to 2:30PM, 2:30 PM to 5:15PM EST). Today (Friday) the last day of the week, the recorder stopped sending ticks at 9:00AM when the session rolled over the pit session.

Earlier on, I used to face problems on Monday morning ES (again a 3 session template). I think that is fixed, but the Friday problem remains.

Do you mean a chart with 3 CL, each with its own session time settings ?
Depending on which session the recorder is running, it won't send ticks outside of the associated session. For instance if you have a GomCD associated with the CL running in the 9:00-2:30 session, it won't send ticks outside of this session.

Or am I not understanding the problem ?

Started this thread Reply With Quote
 
(login for full post details)
  #85 (permalink)
 Silvester17 
Market Wizard
Columbus, OH
 
Experience: None
Platform: NT 8, TOS
Trading: ES
 
Silvester17's Avatar
 
Posts: 3,587 since Aug 2009
Thanks: 5,079 given, 11,392 received


gomi View Post
Do you mean a chart with 3 CL, each with its own session time settings ?
Depending on which session the recorder is running, it won't send ticks outside of the associated session. For instance if you have a GomCD associated with the CL running in the 9:00-2:30 session, it won't send ticks outside of this session.

Or am I not understanding the problem ?

hi gomi,

I have the same problem. 1 chart, 1 instrument and a 3 session template. (template with different sessions, from 20:00 - 09:30, 09:30 - 16:15 and 16:15 - 18:00)

Attached Thumbnails
Click image for larger version

Name:	2010-11-03_1523.png
Views:	77
Size:	32.0 KB
ID:	25243  
Reply With Quote
 
(login for full post details)
  #86 (permalink)
 gomi 
Paris
 
Experience: None
Platform: NinjaTrader
 
Posts: 1,248 since Oct 2009
Thanks: 275 given, 4,421 received

Thanks.

Not sure this is supported by ninja : I tried on the ES and I don't get Session Break bars, ans Bars.SessionBreak never gets true.

If you want all ticks you can always set "disable tick filter" to true.

Started this thread Reply With Quote
 
(login for full post details)
  #87 (permalink)
tarantino
Willowbrook, IL
 
 
Posts: 32 since Sep 2010
Thanks: 71 given, 39 received


gomi View Post
Thanks.

Not sure this is supported by ninja : I tried on the ES and I don't get Session Break bars, ans Bars.SessionBreak never gets true.

If you want all ticks you can always set "disable tick filter" to true.

How about if(Bars.FirstBarOfSession) ?

Reply With Quote
 
(login for full post details)
  #88 (permalink)
 paolfili 
italy
 
Experience: Intermediate
Platform: Ninjatrader
Broker: Zen-Fire
Trading: ES
 
Posts: 33 since Nov 2009
Thanks: 51 given, 11 received

Sorry.Probably OT,but gomi presence can be useful for some suggestions.
I' m interested in having in the same panel some CD of the different 'size players' (i.e. on five level <5,6<x<49,50<x<99,100<x<299,x>300 - kind of Volume Splitter on Cumulative Delta with different lines color for different players).
Is there any way to achieve without big revolution on the code?
Can gomi or some other CD expert programmer (Zondor?) give me some hints?
Thanks

Reply With Quote
 
(login for full post details)
  #89 (permalink)
 ZTR 
 
Experience: Advanced
Platform: NinjaTrader7
Broker: Mirus RCG/Zen-Fire
Trading: CL & 6e, looking at ES, ZB and AU again.
 
ZTR's Avatar
 
Posts: 2,099 since Nov 2009
Thanks: 1,099 given, 1,392 received


pfx111 View Post
Sorry.Probably OT,but gomi presence can be useful for some suggestions.
I' m interested in having in the same panel some CD of the different 'size player' (i.e. on five level <5,6<x<49,50<x<99,100<x<299,x>300 - kind of Volume Splitter on Cumulative Delta with different lines color for different players).
Is there any way to achieve without big revolution on the code?
Can gomi or some other CD expert programmer (Zondor?) give me some hints?
Thanks

Quick way: You could have more than one Gom CD in the lower panels with different size filters. I have posted ladders with the GOM CD, both filtered by size.

There are differences...

R.I.P. Andy Zektzer (ZTR), 1960-2010.
Please visit this thread for more information.
Attached Thumbnails
Click image for larger version

Name:	Vol_Filter.jpg
Views:	94
Size:	74.6 KB
ID:	25439  
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #90 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

I've started working on the DB back-end for gomRecorderIndicator. It got a little more complicated when I figured out that bringing up the DB had to be serialized, that added a lot of complexity if one wanted to keep it as an NT indicator, so I'm taking a slightly different route. I'm in the process of creating it as stand alone server app. I've got the server side up and running, although not really debugged at all. I'm working on the client side now.

The downside is I probably won't have anything I can distribute until next weekend.

Once I do that I'll start a new thread instead of continuing to hijack this one!

Reply With Quote
 
(login for full post details)
  #91 (permalink)
 Big Mike 
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: Custom solution
Trading: Futures & Crypto
 
Big Mike's Avatar
 
Posts: 50,093 since Jun 2009
Thanks: 32,554 given, 98,581 received


danjurgens View Post
I've started working on the DB back-end for gomRecorderIndicator. It got a little more complicated when I figured out that bringing up the DB had to be serialized, that added a lot of complexity if one wanted to keep it as an NT indicator, so I'm taking a slightly different route. I'm in the process of creating it as stand alone server app. I've got the server side up and running, although not really debugged at all. I'm working on the client side now.

The downside is I probably won't have anything I can distribute until next weekend.

Once I do that I'll start a new thread instead of continuing to hijack this one!

Looking forward to your new thread. Sooner rather than later I hope, because I already have questions

I've been trying to get a mysql backend for recording tick data for a long time, this may be exactly that. But I wanted to make it publicly accessible (futures.io (formerly BMT) Elite only). So maybe we can work together to make that happen, the general idea was a central tick repository that others can pull from.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/

Visit other sites? Please spread the word about your experience with our community!
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #92 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

What I have so far is a long way from something that I'd ever expose to the internet. Robustness has taken a backseat to getting something that works fairly quickly, but the underpinnings are probably there. Design problem I can foresee for something that serves data to a lot of clients are: 1. That it spawns a new thread for each client, so it might not scale well over a hundred or so concurrent clients. 2. The overly simple protocol has no way to deal with faulty requests or anything.

I opted against a relational database, and am using Berkeley DB (which is what MySQL used to use for it's internal storage). BDB stores data via key/data pairs, this works great for ticks, with a key of (symbol,time) and data of (price, bid, ask, volume), duplicate keys are allowed. For simplicity in implementation I just have 1 database right now, but it will be easy to change to have numerous internal DBs for symbols, or symbols and subsets of the time like year/month, but that'd be completely transparent to the client.

The socket interface is very simple, 3 requests read, realtime write, and back fill. Only one response from the server, containing data from a read request (1..N per request). Real time and back fill are the same except backfill first deletes all the data for that exact symbol and time stamp. Read requests take a symbol and start time stamp, it then streams all the data after that time stamp for the given symbol in blocks that each contain all the ticks for that exact time stamp.

This design pushes some buffering responsibility to a back fill writer, because it has to send all the ticks for a unique time stamp at once or it deletes it's previously sent data. Readers get N responses each one with all the data for a given time stamp.

Right now I'm working on shoehorning a client into a GomFileManager subclass so I can do some debugging and testing of the server. It takes a bit of kludging because the recordTick interface to that class only provides the price and tick type, so I have to rebuild fake ask and bid prices. I also need to understand the read back use case a little better, and that might lead to changes in the read request/response interface.

Reply With Quote
The following user says Thank You to danjurgens for this post:
 
(login for full post details)
  #93 (permalink)
 Michael.H 
CA
 
Experience: Master
Platform: Marketdelta and Ninja
Broker: Velocity
Trading: NQ
 
Posts: 670 since Apr 2010
Thanks: 64 given, 526 received

I truly think ninja should come in here and kiss the feet of gomi and other programmers that are literally fixing their buggy crap software for them for free.

Reply With Quote
 
(login for full post details)
  #94 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

It is quite amazing all the hoops that we have to jump through because they won't add 2 column to table!

Reply With Quote
 
(login for full post details)
  #95 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received


danjurgens View Post
It is quite amazing all the hoops that we have to jump through because they won't add 2 column to table!

It is worse than that.

They store the bid-ask-last tick data, but store it separately so the sequence of ticks is not preserved. I presume that is because NT can draw the charts with either of these values, and some one made the architectural decision to tie the db to the application!

They could very easily fix this so that a single tick db has all the three types of ticks with the proper sequential time-stamp with a very simple filter but they chose not too. So the tick data downloaded by NT is not useful to backfill.

HOWEVER, they also have the infrastructure for replay which DOES store and capture tick data (both L1 and L2) in the right sequence. However that infrastructure can be used only for replay and can not drive your indicators.

I presume they are simply overloaded and understaffed and different teams did their own thing without regards to the big picture. NT is only software I know off which maintains two separate mutually incompatible dbs for tick data. But they still can not offer chart backfill capability (OnMarketData) for historical values.
--------------------------------------------------------------------------------------------------

On your project: The biggest challenge I have is that you can not update your database while you are collecting the ticks. So you have to find a time-window when the ticks are not coming to fill gaps in your gomi data base.

There is a similar issue with your db also. How will you guarantee that you do not create duplicate entries when you try to update the db with historical data for those times your tick collector was not working.? One way to approach this would be to define sectors (say 5 minutes of tick data), and use that as the minimum unit which can be updated. So when you are reloading historical data, your application should send data in chunks of 5 min (or multiples of 5min) and then the db will treat that data as the golden copy and over write any existing data for those 5 minutes with the new data. On the other hand the live tick data will not check for duplicates and simply insert.

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to aviat72 for this post:
 
(login for full post details)
  #96 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received


gomi View Post
Thanks.

Not sure this is supported by ninja : I tried on the ES and I don't get Session Break bars, ans Bars.SessionBreak never gets true.

If you want all ticks you can always set "disable tick filter" to true.

OK. Will try that. At least for the charts I am interested in, it needs the complete session.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #97 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

There will be no issue writing backfill and real time to the DB at anytime. All updates and reads from the DB will be by unique time stamp (seconds from the market, although I see gomi has a MS option). Berkeley DB supports transactions, readers will have a consistent view of the data on a per time stamp basis as they'll lock each block while reading it to prevent a write to that data, so writers block until readers release their lock.

I have two thoughts about concurrent real time writers. Either the DB server will only accept real time writes from the first connected client to request real time writing and other requests will be dumped as No-Ops. Or it will always delete the data for a key (symbol/time stamp) before writing the new data. Plan 2 would require the writer to buffer all ticks for a time stamp before sending, that way multiple writers would just overwrite each others data until the last ones data persists, not the most efficient, but workable. Whatever route I go will all be encapsulated in the client side interface I'm exposing to GomFileManager.

The use case of this DB is frequent writes to the end with rare burst reads of large amounts data across all keys, and rare burst writes. The only place a real time writer and reader conflict is at the last time stamp, so that should keep lock blocking quite minimal. Doing a simultaneous backfill while reading could have higher contention, but should still be quite reasonable since the amount of locks needing to be acquired to write or read is small.

Reply With Quote
 
(login for full post details)
  #98 (permalink)
 aviat72 
San Francisco Bay Area
 
Experience: Intermediate
Platform: NT,TOS,IB
Trading: ES,CL,TF
 
Posts: 280 since Jun 2010
Thanks: 155 given, 271 received

The issue is that time stamps are not guaranteed to be unique. You can not assume that two trade ticks with exactly the same time-stamp are one and the same trade; they may correspond to different trades. Of course if the data-vendor supplies some unique tick identification mechanism you should be able to avoid the conflict. So you may end up with duplicate copies for the same tick if you have a live writer and also a backfill application which dumps historical data asynchronously.

This of course depends on the time-stamp resolution. With the 1second resolution you are virtually guaranteed to have non-unique time-stamps; even with the ms resolution, the time-stamps are not guaranteed to be unique for separate trades.

That is why I felt that backfills should over-write live data and should also be executed with some minimum chunk size.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #99 (permalink)
 danjurgens 
Austin, TX
 
Experience: Advanced
Platform: TradeStation, TWS
Broker: TradeStation, IB .. for now
Trading: 6E, ES
 
Posts: 28 since May 2010
Thanks: 0 given, 28 received

BDB accepts duplicate keys and duplicate key/data pairs just fine. You just have to use a construct they call a cursor to access all the data with duplicate keys. The conflict between realtime writers and backfill writers resolves itself when you force the backfill writer to provide all the tick data for each unique time stamp at once. Then you just delete and replace all the data for each time stamp with the back filled data. It's theoretically possible you would lose some ticks if you back fill up to the current real time second, but I don't see that being possible in my initial implementation. To download back fill for the current partial second and get it processed and to the server while a real time writer is still writing that tick is race the user can avoid by not back filling up to right now, just go a second back.

Reply With Quote
 
(login for full post details)
  #100 (permalink)
 FBJS 
Toronto
 
Experience: Advanced
Platform: NinjaTrader
 
Posts: 108 since Sep 2009
Thanks: 30 given, 191 received


danjurgens View Post
It is quite amazing all the hoops that we have to jump through because they won't add 2 column to table!

I agree... if you take a look at how MultiCharts is dealing with their customers (setting up a site to ask for suggestions regarding their new DOM feature), you can tell that they are going to eat NinjaTrader's lunch in the not too distant future, once their feature set catches up. NT has a suggestion board too, too bad they rarely listen or do anything that their customers actually suggest... it's a lot of "my way or the highway" with them.

NT is only getting away with their ridiculously bad level of customer responsiveness and horribly delayed development timelines because there hasn't been a decently priced competitor for a while... hopefully MultiCharts can change that. I don't blame Big Mike for switching, as soon as they add some discretionary trading features I am thinking about it myself.

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > Using OnMarketData() on Historical data with a recording engine


Last Updated on March 6, 2011


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing

NEW BlackBird Features + FOREX Support w/Jeremy Tang @ SharkIndicators

Elite only
 

Our 12-year anniversary w/ $$,$$$ prizes (check soon)

June
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts