NexusFi: Find Your Edge


Home Menu

 





how can I export ES indiv contracts (from NT7) as 1 continuous csv?


Discussion in NinjaTrader

Updated
    1. trending_up 1,676 views
    2. thumb_up 0 thanks given
    3. group 3 followers
    1. forum 7 posts
    2. attach_file 0 attachments




 
Search this Thread

how can I export ES indiv contracts (from NT7) as 1 continuous csv?

  #1 (permalink)
 Wil70 
SF (CA/USA)
 
Experience: Beginner
Platform: NT / Proprietary
Broker: IB / may be Kinetic
Trading: ES, CL, NQ, YM, 6E
Posts: 32 since Nov 2015
Thanks Given: 169
Thanks Received: 10

Hello, how can I export NT7 tick data from none continuous contract as 1 csv continuous contract ?

Note: I downloaded some continuous and none continuous contract, I can export both but I found a bug, I notice as I download new tick data the old data are being removed (from the continuous contract)....so now I Can not rely on my ES ##-## as it is missing some older data I did have, and if I export the last current ES individual contract then this is all I have in the file, I would like to export all ticks I have from ES as 1 continuous contract.

thanks for your help
w

Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
Futures True Range Report
The Elite Circle
Ninja Mobile Trader VPS (ninjamobiletrader.com)
Trading Reviews and Vendors
Build trailing stop for micro index(s)
Psychology and Money Management
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
60 thanks
Funded Trader platforms
37 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #2 (permalink)
 Wil70 
SF (CA/USA)
 
Experience: Beginner
Platform: NT / Proprietary
Broker: IB / may be Kinetic
Trading: ES, CL, NQ, YM, 6E
Posts: 32 since Nov 2015
Thanks Given: 169
Thanks Received: 10

My goal is to export 1 continuous contracts file with only ticks.
As there is a known bug when downloading a continuous contract (like they overwrite older data) with NT UI, I decided to download individual contracts, then to export their ticks by running a backtest (with a slightly modified version of the SampleStreamwritter code from the documentation so it can handle ticks)
But i always end up having this:


"Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process.
Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process."

 
Code
        protected override void Initialize()
        {
            CalculateOnBarClose	= true;
			path= Cbi.Core.UserDataDir.ToString() + this.Instrument.FullName +".txt";
            Console.Out.WriteLine(path);
            sw = File.AppendText(path);
        }
		
        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {

        }
		
        protected override void OnMarketData(MarketDataEventArgs e)
        {
            if (e.MarketDataType == MarketDataType.Last)				
            {
				sw.WriteLine(ToDay(e.Time) + "," + e.Price);
             }
        }
		// Necessary to call in order to clean up resources used by the StreamWriter object
		protected override void OnTermination() 
		{
			// Disposes resources used by the StreamWriter
			if (sw != null)
			{
				sw.Dispose();
				sw = null;
			}
		}
What other process is locking the file?
Is there another way to export tick data out of NT7 in a csv file?

Thanks.

Started this thread Reply With Quote
  #3 (permalink)
 
sam028's Avatar
 sam028 
Site Moderator
 
Posts: 3,765 since Jun 2009
Thanks Given: 3,825
Thanks Received: 4,629


Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Success requires no deodorant! (Sun Tzu)
Follow me on Twitter Reply With Quote
  #4 (permalink)
 Wil70 
SF (CA/USA)
 
Experience: Beginner
Platform: NT / Proprietary
Broker: IB / may be Kinetic
Trading: ES, CL, NQ, YM, 6E
Posts: 32 since Nov 2015
Thanks Given: 169
Thanks Received: 10


sam028 View Post
Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

Started this thread Reply With Quote
  #5 (permalink)
 
sam028's Avatar
 sam028 
Site Moderator
 
Posts: 3,765 since Jun 2009
Thanks Given: 3,825
Thanks Received: 4,629


Wil70 View Post
Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Success requires no deodorant! (Sun Tzu)
Follow me on Twitter Reply With Quote
  #6 (permalink)
 Wil70 
SF (CA/USA)
 
Experience: Beginner
Platform: NT / Proprietary
Broker: IB / may be Kinetic
Trading: ES, CL, NQ, YM, 6E
Posts: 32 since Nov 2015
Thanks Given: 169
Thanks Received: 10


sam028 View Post
You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.

Started this thread Reply With Quote
  #7 (permalink)
 
sam028's Avatar
 sam028 
Site Moderator
 
Posts: 3,765 since Jun 2009
Thanks Given: 3,825
Thanks Received: 4,629


Wil70 View Post
Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.

Dispose() should close the file too, but if you want to add both...
Correct, OnMarketData will be called after each order book updates, it's useless in your case.

Success requires no deodorant! (Sun Tzu)
Follow me on Twitter Reply With Quote
  #8 (permalink)
 
shodson's Avatar
 shodson 
OC, California, USA
Quantoholic
 
Experience: Advanced
Platform: IB/TWS, NinjaTrader, ToS
Broker: IB, ToS, Kinetick
Trading: stocks, options, futures, VIX
Posts: 1,976 since Jun 2009
Thanks Given: 533
Thanks Received: 3,709

You should wrap your File.AppendText() calls around a using block. This will handle all of the disposing and resource releasing automatically once the block is exited. See the example code at https://msdn.microsoft.com/en-us/library/system.io.file.appendtext.aspx

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(ToDay(e.Time) + "," + e.Price);
}
And you never have to call dispose()

If that feels like too much file opening and closing/releasing, consider using a StringBuilder and writing it out to the file OnTermination()

 
Code
private StringBuilder sb = new StringBuilder();
then in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
Then in OnTermination()

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(sb.ToString());
}
if you're writing lots of data that string could get really big, maybe check the size of it and write it to disk and clear it mid-stream like every megabyte, in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
if (sb.Length > 1024 * 1024) {
   using (StreamWriter sw = File.AppendText(path))
   {
      sw.WriteLine(sb.ToString());
   }
   sb.Clear();
}

Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote




Last Updated on November 7, 2016


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts