how can I export ES indiv contracts (from NT7) as 1 continuous csv? - NinjaTrader | futures.io
futures.io futures trading
 

Go Back   futures.io

> Futures Trading, News, Charts and Platforms > Platforms and Indicators > NinjaTrader


how can I export ES indiv contracts (from NT7) as 1 continuous csv?
Started:October 29th, 2016 (10:50 PM) by Wil70 Views / Replies:261 / 7
Last Reply:November 7th, 2016 (09:03 PM) Attachments:0

Welcome to futures.io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike
     

Reply
 
Thread Tools Search this Thread

how can I export ES indiv contracts (from NT7) as 1 continuous csv?

Old October 29th, 2016, 10:50 PM   #1 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 19 since Nov 2015
Thanks: 140 given, 5 received

how can I export ES indiv contracts (from NT7) as 1 continuous csv?

Hello, how can I export NT7 tick data from none continuous contract as 1 csv continuous contract ?

Note: I downloaded some continuous and none continuous contract, I can export both but I found a bug, I notice as I download new tick data the old data are being removed (from the continuous contract)....so now I Can not rely on my ES ##-## as it is missing some older data I did have, and if I export the last current ES individual contract then this is all I have in the file, I would like to export all ticks I have from ES as 1 continuous contract.

thanks for your help
w

Reply With Quote
     

Old October 31st, 2016, 04:28 PM   #2 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 19 since Nov 2015
Thanks: 140 given, 5 received

My goal is to export 1 continuous contracts file with only ticks.
As there is a known bug when downloading a continuous contract (like they overwrite older data) with NT UI, I decided to download individual contracts, then to export their ticks by running a backtest (with a slightly modified version of the SampleStreamwritter code from the documentation so it can handle ticks)
But i always end up having this:


"Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process.
Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process."

 
Code
        protected override void Initialize()
        {
            CalculateOnBarClose	= true;
			path= Cbi.Core.UserDataDir.ToString() + this.Instrument.FullName +".txt";
            Console.Out.WriteLine(path);
            sw = File.AppendText(path);
        }
		
        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {

        }
		
        protected override void OnMarketData(MarketDataEventArgs e)
        {
            if (e.MarketDataType == MarketDataType.Last)				
            {
				sw.WriteLine(ToDay(e.Time) + "," + e.Price);
             }
        }
		// Necessary to call in order to clean up resources used by the StreamWriter object
		protected override void OnTermination() 
		{
			// Disposes resources used by the StreamWriter
			if (sw != null)
			{
				sw.Dispose();
				sw = null;
			}
		}
What other process is locking the file?
Is there another way to export tick data out of NT7 in a csv file?

Thanks.

Reply With Quote
     

Old October 31st, 2016, 04:48 PM   #3 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,231 since Jun 2009
Thanks: 3,453 given, 3,807 received


Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Usually in trading, those who know don't talk, and those who talk don't know. (Al Brooks)
success requires no deodorant! (Sun Tzu)
Reply With Quote
     

Old October 31st, 2016, 05:46 PM   #4 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 19 since Nov 2015
Thanks: 140 given, 5 received


sam028 View Post
Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

Reply With Quote
     

Old October 31st, 2016, 06:33 PM   #5 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,231 since Jun 2009
Thanks: 3,453 given, 3,807 received

Answer
This post has been selected as an answer to the original posters question Answer

Futures Edge on FIO

Are you a NinjaTrader user?

 

Wil70 View Post
Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Usually in trading, those who know don't talk, and those who talk don't know. (Al Brooks)
success requires no deodorant! (Sun Tzu)
Reply With Quote
     

Old October 31st, 2016, 06:56 PM   #6 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 19 since Nov 2015
Thanks: 140 given, 5 received


sam028 View Post
You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.


Last edited by Wil70; October 31st, 2016 at 07:02 PM.
Reply With Quote
     

Old October 31st, 2016, 07:32 PM   #7 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,231 since Jun 2009
Thanks: 3,453 given, 3,807 received


Wil70 View Post
Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.

Dispose() should close the file too, but if you want to add both...
Correct, OnMarketData will be called after each order book updates, it's useless in your case.

Usually in trading, those who know don't talk, and those who talk don't know. (Al Brooks)
success requires no deodorant! (Sun Tzu)

Last edited by sam028; October 31st, 2016 at 07:33 PM. Reason: typo
Reply With Quote
     

Old November 7th, 2016, 09:03 PM   #8 (permalink)
 Vendor: www.probabletrades.com 
OC, California, USA
 
Futures Experience: Advanced
Platform: IB/TWS, NinjaTrader, thinkorswim
Favorite Futures: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,835 since Jun 2009
Thanks: 471 given, 3,167 received

You should wrap your File.AppendText() calls around a using block. This will handle all of the disposing and resource releasing automatically once the block is exited. See the example code at https://msdn.microsoft.com/en-us/library/system.io.file.appendtext(v=vs.110).aspx

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(ToDay(e.Time) + "," + e.Price);
}
And you never have to call dispose()

If that feels like too much file opening and closing/releasing, consider using a StringBuilder and writing it out to the file OnTermination()

 
Code
private StringBuilder sb = new StringBuilder();
then in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
Then in OnTermination()

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(sb.ToString());
}
if you're writing lots of data that string could get really big, maybe check the size of it and write it to disk and clear it mid-stream like every megabyte, in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
if (sb.Length > 1024 * 1024) {
   using (StreamWriter sw = File.AppendText(path))
   {
      sw.WriteLine(sb.ToString());
   }
   sb.Clear();
}

Reply With Quote
     

Reply



futures.io > Futures Trading, News, Charts and Platforms > Platforms and Indicators > NinjaTrader > how can I export ES indiv contracts (from NT7) as 1 continuous csv?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)
 

Trading Technologies: ADL hands-on

Dec 13

Normal webinar schedule resumes after the holidays

January

Ernie Chan: Risk Management

Elite only

Dedicated Trading Servers: Advantages/Disadvantages w/sam028

Elite only

An Afternoon with FIO member Massive I

Elite only

Leo Murphy: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
NinjaTrader 7 Continuous Contracts grausch NinjaTrader 2 September 19th, 2016 08:33 AM
Continuous Contracts steve2222 Terms (Glossary) 0 July 4th, 2016 04:04 AM
Export chart data plotted from signal (Excel/CSV/etc)? cestnoir MultiCharts 1 April 12th, 2016 06:23 AM
ZigZag indicator export to txt/csv PK007 EasyLanguage Programming 2 January 21st, 2015 02:49 PM
multitimeframe live CSV export rirhasiu NinjaTrader 5 May 16th, 2013 03:15 PM


All times are GMT -4. The time now is 06:45 AM.

Copyright © 2016 by futures.io. All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
 
no new posts
Page generated 2016-12-10 in 0.16 seconds with 35 queries on phoenix via your IP 54.166.37.177