how can I export ES indiv contracts (from NT7) as 1 continuous csv? - NinjaTrader | futures io social trading
futures io futures trading


how can I export ES indiv contracts (from NT7) as 1 continuous csv?
Created: by Wil70 Views / Replies:354 / 7
Updated: Attachments:0

Welcome to futures io.

Welcome, Guest!

This forum was established to help traders (especially futures traders) by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

We are fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike

Reply
 
Thread Tools Search this Thread
 

how can I export ES indiv contracts (from NT7) as 1 continuous csv?

  #1 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 23 since Nov 2015
Thanks: 156 given, 5 received

how can I export ES indiv contracts (from NT7) as 1 continuous csv?

Hello, how can I export NT7 tick data from none continuous contract as 1 csv continuous contract ?

Note: I downloaded some continuous and none continuous contract, I can export both but I found a bug, I notice as I download new tick data the old data are being removed (from the continuous contract)....so now I Can not rely on my ES ##-## as it is missing some older data I did have, and if I export the last current ES individual contract then this is all I have in the file, I would like to export all ticks I have from ES as 1 continuous contract.

thanks for your help
w

Reply With Quote
 
  #2 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 23 since Nov 2015
Thanks: 156 given, 5 received

My goal is to export 1 continuous contracts file with only ticks.
As there is a known bug when downloading a continuous contract (like they overwrite older data) with NT UI, I decided to download individual contracts, then to export their ticks by running a backtest (with a slightly modified version of the SampleStreamwritter code from the documentation so it can handle ticks)
But i always end up having this:


"Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process.
Failed to call method 'Initialize' for indicator 'SampleStreamWriter': The process cannot access the file 'C:\Users\Wilhelm\Documents\NinjaTrader 7\MyTestFile.txt' because it is being used by another process."

 
Code
        protected override void Initialize()
        {
            CalculateOnBarClose	= true;
			path= Cbi.Core.UserDataDir.ToString() + this.Instrument.FullName +".txt";
            Console.Out.WriteLine(path);
            sw = File.AppendText(path);
        }
		
        /// <summary>
        /// Called on each bar update event (incoming tick)
        /// </summary>
        protected override void OnBarUpdate()
        {

        }
		
        protected override void OnMarketData(MarketDataEventArgs e)
        {
            if (e.MarketDataType == MarketDataType.Last)				
            {
				sw.WriteLine(ToDay(e.Time) + "," + e.Price);
             }
        }
		// Necessary to call in order to clean up resources used by the StreamWriter object
		protected override void OnTermination() 
		{
			// Disposes resources used by the StreamWriter
			if (sw != null)
			{
				sw.Dispose();
				sw = null;
			}
		}
What other process is locking the file?
Is there another way to export tick data out of NT7 in a csv file?

Thanks.

Reply With Quote
 
  #3 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,309 since Jun 2009
Thanks: 3,522 given, 3,916 received


Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
 
  #4 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 23 since Nov 2015
Thanks: 156 given, 5 received


sam028 View Post
Maybe another Ninja thread, if you forgot the Dispose() in a previous version?
The easier is to export from Tools -> Historical Data Manager , I don't see why you'll like to use a strategy to do this.

Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

Reply With Quote
 
  #5 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,309 since Jun 2009
Thanks: 3,522 given, 3,916 received

Answer
This post has been selected as an answer to the original posters question Answer

futures io on Twitter

Wil70 View Post
Thanks Sam, my goal is to export all my ZB ticks from jan 2015 to now in a csv file.
Tools -> Historical Data Manager won't work because this imply I use the continuous contract of ZB (to have a nice back adjusted serie), but when I download new data for ZB ##-## from Continuum it always keep 1.5 years of date and overwrite my older data in ZB ##-## (like let say I have Jan2015 to June 2016, then I decide to download all the data missing from now to June 2016, it will erase Jan 2015 to April2015, so I will have left 1.5 years of tick data (from now to May 2015))...basically older data disappear with ZB##-## but they do not disappear in the individual contracts.
So I'm basically looking for a way to export those ticks as one continuous contract by using the individual contracts so I do not have the issue of disappearance of older ticks...The idea was to backtest with a little script from ZB 12-16 so I got a nice continuous contract that goes back to jan 2015. But I have this error of file locked by another process, even after restarting the NT and the machine

You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
 
  #6 (permalink)
Elite Member
SF (CA/USA)
 
Futures Experience: Beginner
Platform: NT / Proprietary
Broker/Data: IB / may be Kinetic
Favorite Futures: ES, CL, NQ, YM, 6E
 
Posts: 23 since Nov 2015
Thanks: 156 given, 5 received


sam028 View Post
You can find which process is using this file in opening the Windows Resource Monitor, from the CPU tab. Select all the processes, then take a look at the "Associated Handles", sort by "Handle name" and search for your .txt file.

I did something similar in the past and it worked, I used a sw.Close() instead of sw.Dispose() but it should do the same thing.

Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.


Last edited by Wil70; October 31st, 2016 at 06:02 PM.
Reply With Quote
 
  #7 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,309 since Jun 2009
Thanks: 3,522 given, 3,916 received


Wil70 View Post
Yeah working! Thanks Sam, I had 2 mistakes:
1- I didn't use close as you did, I just used Dispose, now I use both...working!
2- I move the export code in OnBarUpdate instead of OnMarketData because when you backtest, even if it is tick by tick, it will create bars of 1 tick.....(OnMarketData is never call)
Thanks again
Best
w

Note to future reader: if you know a better way to do it, please do not hesitate to post here. thanks.

Dispose() should close the file too, but if you want to add both...
Correct, OnMarketData will be called after each order book updates, it's useless in your case.

Success requires no deodorant! (Sun Tzu)

Last edited by sam028; October 31st, 2016 at 06:33 PM. Reason: typo
Reply With Quote
 
  #8 (permalink)
 Vendor: www.probabletrades.com 
OC, California, USA
 
Futures Experience: Advanced
Platform: IB/TWS, NinjaTrader, thinkorswim
Favorite Futures: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,845 since Jun 2009
Thanks: 473 given, 3,222 received

You should wrap your File.AppendText() calls around a using block. This will handle all of the disposing and resource releasing automatically once the block is exited. See the example code at https://msdn.microsoft.com/en-us/library/system.io.file.appendtext(v=vs.110).aspx

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(ToDay(e.Time) + "," + e.Price);
}
And you never have to call dispose()

If that feels like too much file opening and closing/releasing, consider using a StringBuilder and writing it out to the file OnTermination()

 
Code
private StringBuilder sb = new StringBuilder();
then in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
Then in OnTermination()

 
Code
using (StreamWriter sw = File.AppendText(path))
{
   sw.WriteLine(sb.ToString());
}
if you're writing lots of data that string could get really big, maybe check the size of it and write it to disk and clear it mid-stream like every megabyte, in OnMarketData()

 
Code
sb.AppendLine(ToDay(e.Time) + "," + e.Price);
if (sb.Length > 1024 * 1024) {
   using (StreamWriter sw = File.AppendText(path))
   {
      sw.WriteLine(sb.ToString());
   }
   sb.Clear();
}

Reply With Quote

Reply



futures io > > > > how can I export ES indiv contracts (from NT7) as 1 continuous csv?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)
 

Market Analysis w/Dave Forss @ IQFeed

Apr 25
 

Introducing iSystems with Stage 5 Trading

Apr 27

*NEW* FIO Journal Challenge featuring NinjaTrader ($2,000+ of prizes)

May

Prototyping Python Strategies (part 2) w/Ran Aroussi

Elite only

Ask Me Anything w/Raymond Deux @ NinjaTrader

May 3

An Afternoon with FIO member Softsoap

Elite only
 

Machine Learning & Data Mining Bias w/Kris Longmore @ Robot Wealth

Elite only

Brannigan Barrett (TBA)

Elite only

FIO member Gomi (TBA)

Elite only

FuturesTrader71 (TBA)

Elite only

EasyLanguage Programming w/Chris @ ABC Trading Group

Elite only

Portfolio Diversification w/Brendon Delate

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
NinjaTrader 7 Continuous Contracts grausch NinjaTrader 2 September 19th, 2016 07:33 AM
Continuous Contracts steve2222 Terms (Glossary) 0 July 4th, 2016 03:04 AM
Export chart data plotted from signal (Excel/CSV/etc)? cestnoir MultiCharts 1 April 12th, 2016 05:23 AM
ZigZag indicator export to txt/csv PK007 EasyLanguage Programming 2 January 21st, 2015 01:49 PM
multitimeframe live CSV export rirhasiu NinjaTrader 5 May 16th, 2013 02:15 PM


All times are GMT -4. The time now is 07:54 AM.

no new posts
Page generated 2017-04-26 in 0.13 seconds with 35 queries on phoenix via your IP 54.198.73.111