Data converter from a format to another format - Platforms and Indicators | futures io social day trading
futures io futures trading


Data converter from a format to another format
Updated: Views / Replies:4,286 / 22
Created: by Nicolas11 Attachments:9

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 9  
 
Thread Tools Search this Thread
 

Data converter from a format to another format

  #11 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #12 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,352 given, 83,223 received


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

I've just been busy with my own projects. But I'm sure your files will help many others.

Thx

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #13 (permalink)
Patient Discipline
Bellingham, WA USA
 
Futures Experience: Beginner
Platform: NT
Broker/Data: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Favorite Futures: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 426 given, 191 received



Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

This is just the program I was looking for, as I am interested in converting the QCollector data file formats from the GOM/QCollector thread (https://futures.io/elite-circle/7279-official-gomrecorder-qcollector-tick-data-sharing-thread.html) to the standard NT7 tick import format, which Mike has already pointed out and that can be found here (NinjaTrader Version 7). I will kick the tires on this and let you know how it works.

Thanks for your programming wizardry!

All best,

Aventeren

Reply With Quote
 
  #14 (permalink)
Patient Discipline
Bellingham, WA USA
 
Futures Experience: Beginner
Platform: NT
Broker/Data: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Favorite Futures: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 426 given, 191 received


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

So the program works perfectly. However, I have a couple of suggestions for potential improvements that are based on how the QCollector files are downloaded from the QCOllector thread.

When one downloads the QCollector data from the QCollector thread, the files are all housed in an instrument specific folder. For instance, with the January 2011 QCollector file, the folder contains all of the CL contracts that traded volume during January 2011--and in January 2011, the following CL contracts were traded:

1. CL ##-##.txt
2. CL 01-12.txt
3. CL 02-11.txt
4. CL 02-11.txt
5. CL 03-11.txt
6. CL 04-11.txt
7. CL 05-11.txt
8. CL 06-11.txt
9. CL 07-11.txt
10. CL 08-11.txt
11. CL 09-11.txt
12. CL 10-11.txt
13. CL 11-11.txt
14. CL 12-11.txt

So the procedure to use your tool, would be to drop in the .jar file into the folder, rename the file that you'd like to convert, execute the .jar file, which will in turn create a new file called "input.txt, which will then need to be renamed to the NT7 import requirement of "<instrument name> <mm-yy>.txt" (ie, "CL ##-##.txt" or "CL 02-11.txt"). The tool clearly works, it just requires a few steps that if I could wave my magic wand would be incorporated into the .jar program. So, given the above, here are my magic wand requests:

Request #1: Have the program be placed in a folder with numerous .txt files and ran once. The program would automatically note the file name, which is in the correct format for NT formatting but with data that is not formatted correctly, and copy the original file name. The program would then rename the original file with an "_original" in it and create a new file with the original file name (and which would have the correctly formatted data and file name to import into NT7). For example, the "CL 02-11.txt" original file would be renamed to "CL 02-11_original.txt" and the new file with the properly formatted data would be called "CL 02-11.txt"

Request #2: The newly formatted output files, which would have the correct file name and data formatting, would then be placed in a new folder that would contain all of the converted files.

By making the two magic wand wishes come true, one would then be able to process an entire month's worth of data with a single mouse click--albeit over a considerable amount of time. My thinking here is that this process would just run in the background, and when it was done one could start on the next round of conversions.

At this point it is probably worth noting what I am hoping to accomplish and provide back to the futures.io (formerly BMT) community. If I can make this happen, I plan on posting all of the newly formatted data into the appropriate threads so that other futures.io (formerly BMT) members will just have to download the data instead of going through the process that I've asked you to help out with above--and my time associated with converting all of the available QCollector data for the instruments I am interested in. We're all in this together, and I am hoping that together we can provide something of use to the other members.

Please let me know if you have any questions. I really appreciate the effort that you have put forth thus far on the converter program, and I am hopeful that some facsimile of the above requests can be achieved.

Thank you,

Aventeren

Reply With Quote
 
  #15 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #16 (permalink)
Patient Discipline
Bellingham, WA USA
 
Futures Experience: Beginner
Platform: NT
Broker/Data: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Favorite Futures: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 426 given, 191 received


Nicolas11 View Post
@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Excellent; you are correct that the file name would not have to be renamed if the newly converted files were dropped into a new sub directory.

I can't wait to see what you come up with.

Please let me know if you have any questions.

Thanks and all best,

Aventeren

Reply With Quote
 
  #17 (permalink)
Elite Member
Netherlands
 
Futures Experience: Intermediate
Platform: Ninjatrader
Favorite Futures: 6E
 
Zwaen's Avatar
 
Posts: 247 since Dec 2010
Thanks: 762 given, 221 received

Hi Nicolas11,

great program, I was looking for this. Like BigMike posted in request 2, I was also very interested in a converter for Qcollector data to NT-format. This is a horror in excel, besides excel has not enough rows.

I wanted to check the output, and did what you said, put the Qcollector data and the Jar file in the same directory, and try to run the Jar file. Somehow, nothing happens when I double click or run the Jar file:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Java is correctly installed:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


Can you see what I do wrong?

Thanks!

One of my worst enemies are my own false assumptions
Reply With Quote
 
  #18 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

@Zwaen ,

I do not know where this problem comes from. I have tested on my side just now, and the output file is produced.

Maybe you can try...

Trick 1: right-click on the jar and execute it with administrator privileges

Trick 2: launch the jar from the DOS command line (after having used cd to move the right directory)
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).



Nicolas

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #19 (permalink)
Elite Member
Netherlands
 
Futures Experience: Intermediate
Platform: Ninjatrader
Favorite Futures: 6E
 
Zwaen's Avatar
 
Posts: 247 since Dec 2010
Thanks: 762 given, 221 received

So the problem is just my computer. I do not see the option to open the file as administrator, but then I will figure it out, or just try the same on another computer /laptop.

Thanks for your help!


Edit: computersavvy as I am , I renamed the Qcollector file "input.txt" as it already was a .txt file. Renaming the file in "input" did the job.

One of my worst enemies are my own false assumptions

Last edited by Zwaen; September 6th, 2013 at 05:06 AM. Reason: Solution found
Reply With Quote
 
  #20 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received


@aventeren ,

Could you test the following?

I propose to use a Windows batch file to apply the jar to the various .txt files of the directory.

1. In the directory of your CL .txt files, make sure that there is :
- no "input.txt"
- no "output.txt"
- no ConvertedFiles sub-directory

2. In the directory of your CL .txt files, copy the jar file

3. Copy/paste the code below into NotePad (or similar) and save it in the CL .txt folder under the name "convert.bat" (no .txt suffix)

4. At this stage, your directory should like:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


5. Launch the batch console from the Window "Accessories" Menu.
Navigate to the above-mentioned directory with "cd"
Type "convert"
It should launch the process, ending with something like:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


The ConvertedFiles sub-directory should contain the output files.

Nicolas

 
Code
@echo OFF

rem | Batch Conversion
rem | v1
rem | September 5th, 2013
rem | by Nicolas11 @ futures.io (formerly BMT)

echo Batch conversion (v1)
echo ---------------------

if exist ConvertedFiles (
	echo ERROR: Directory ConvertedFiles already exist
	goto End
)

if exist "input.txt" (
	echo ERROR: input.txt already exits
	goto End
)

if exist "output.txt" (
	echo ERROR: output.txt already exits
	goto End
)

echo Creation of ConvertFiles directory
mkdir ConvertedFiles

for /f "delims=" %%a in ('dir /b *.txt') do (
	echo Conversion of %%a
	rename "%%a" "input.txt"
	DataConverter_for_BigMike_req02_v01.jar
	move "output.txt" "ConvertedFiles\%%a" > del123.tmp
	rename "input.txt" "%%a"
)

del del123.tmp

:End
echo End.

Reply With Quote
The following 3 users say Thank You to Nicolas11 for this post:

Reply



futures io > > > Data converter from a format to another format

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
possible to convert downloaded historical data to replay format (.ntm) gztanwei NinjaTrader Programming 10 August 24th, 2012 12:37 AM
here's way to convert tick data from NT to comma-separated format, with Mac or Linux theorist Platforms and Indicators 1 July 1st, 2012 05:00 AM
software with milliseconds and L2 in data format bidask201 Platforms and Indicators 2 May 13th, 2012 08:10 PM
ideal data format for MC mea109 MultiCharts 3 September 9th, 2011 04:36 AM
What should I have as 'days back' under 'Format Data Series' eone Traders Hideout 5 February 24th, 2010 07:38 AM


All times are GMT -4. The time now is 06:44 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-13 in 0.14 seconds with 20 queries on phoenix via your IP 54.90.92.204