Data converter from a format to another format - futures io
futures io futures trading



Data converter from a format to another format


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one Nicolas11 with 10 posts (24 thanks)
    2. looks_two aventeren with 5 posts (2 thanks)
    3. looks_3 Big Mike with 4 posts (0 thanks)
    4. looks_4 Zwaen with 2 posts (0 thanks)
    1. trending_up 7,132 views
    2. thumb_up 26 thanks given
    3. group 7 followers
    1. forum 22 posts
    2. attach_file 9 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Data converter from a format to another format

(login for full post details)
  #1 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

This thread is a possible place to obtain a small software allowing the conversion of data from a format to another format.

For the avoidance of doubt, it’s free.

For instance, let’s suppose that you have a 1-Go text file with data formatted like:
05/24/2012 23:45 716,50 713,00 714,25 715,75 12563
… and you want to convert into a text file with the following format:
1120525,0045,716.50,713.00,714.25,715.75,12563
(time in output file shall be one hour later than in the input file).

Then...

1. In this thread, post:
For input file:
- the 5 first lines of your data file
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- has the data file a first line with titles of the fields or not
- size of the input file
For output file:
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- should the data file have a first line with titles of the fields or not

An example of such request is given in the following message.

2. Then, after some time (typically: from 1 day to 1 week), I will post in this thread:
- a JAR file (it is executable, same as EXE file),
- source code in Java.
The source code is useless to execute the JAR file. It is just given for reference.

3. Then, you will have to:
- save the JAR file in the same directory as your data file
- rename your data file as "input.txt"
- execute the JAR file (by double-clicking on it)

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximatively the size of the input file.

An example of such JAR file and source code is given in a following message.

4. Obviously, you will have to check the content of output.txt

5. These instructions may be updated in the "Quick Summary" below.

6. I do not promise to provide this service forever. If there are too many requests or if it is not possible any more for me to continue for any reason, I will kindly ask Big Mike to close this thread.

In case of problems in the execution of the JAR file, I will do my best to help, but in limited time. If we do not find a solution quickly, I will perhaps have to give up. As a reminder, this service is free.

Nicolas

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to Nicolas11 for this post:

Journal Challenge February 2021 results (so far):
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(166 thanks from 30 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(116 thanks from 33 posts)
looks_3Wisdom is Emptinessby Mtype
(68 thanks from 25 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(31 thanks from 17 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received


This message is an example of request.

Hi,

Could you please provide with an executable file allowing the following conversion?

Input file:
Five first lines:

Quoting 
Date Time High Low Open Close Vol
05/24/2012 22:30 716,50 713,00 714,25 715,75 26540
05/24/2012 22:45 716,50 713,00 714,25 715,75 12563
05/24/2012 23:00 716,50 713,00 714,25 715,75 13562
05/24/2012 23:15 716,50 713,00 714,25 715,75 9856

Field delimiter is: tabulation
Fields are: date MM/DD/YYY, hour HH:mm, H, L, O, C, volume
The file contains a first line with the titles of the fields
The size of the input file is 1Go

Output file:
1120525,0045,716.50,713.00,714.25,715.75,12563
Field delimiter is: comma
Field are: date YYYMMDD, hour HHmm, H, L, O, C , volume
Time is in output file shall be one hour later than in the input file
Note: commas within the numbers shall be replaced by dots
No first line with the titles of the fields

Thanks in advance.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #4 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

This message is an example of answer.

Hi,

In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).

Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)

Note: the source code (.java file) is useless for the conversion, and is just given for reference.

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.

It is your responsibility to check the content of output.txt

The 5 first lines of the input file:

Quoting 
Date Time High Low Open Close Vol
05/24/2012 22:30 716,50 713,00 714,25 715,75 26540
05/24/2012 22:45 716,50 713,00 714,25 715,75 12563
05/24/2012 23:00 716,50 713,00 714,25 715,75 13562
05/24/2012 23:15 716,50 713,00 714,25 715,75 9856

have been converted as follows:

Quoting 
1120524,2330,716.5,713,714.25,715.75,26540
1120524,2345,716.5,713,714.25,715.75,12563
1120525,0000,716.5,713,714.25,715.75,13562
1120525,0015,716.5,713,714.25,715.75,9856

On my computer, the conversion of a 1 Go input file has taken 25 minutes.

Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_Nicolas11_v01.zip (4.8 KB, 59 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #5 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,747 since Jun 2009
Thanks: 32,295 given, 97,495 received

Continuing from

Source file is MultiCharts export:

 
Code
"Date","Time","Price","Volume"
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
Includes header line

Output file should be standard NinjaTrader yyyyMMdd HHmmss;price;volume

Are there any file size limits? I need to know if I need to break up to < 2GB or < 4GB files.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #6 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,747 since Jun 2009
Thanks: 32,295 given, 97,495 received

Request 2

Source is QCollector:
 
Code
06/01/2012	05:17:21	12100.00	1	1	12060.00	12100.00	4202	0	0
06/05/2012	09:14:20	12000.00	1	1	12000.00	12018.00	999	0	0
06/07/2012	22:02:33	12442.00	5	5	12442.00	12450.00	2445	0	0
06/10/2012	17:00:00	12524.00	1	1	0.00	12524.00	84	0	0
06/11/2012	06:35:54	12500.00	1	2	12500.00	12552.00	635	0	0
06/11/2012	08:15:50	12498.00	1	4	12498.00	12561.00	671	0	0
No header line. Tab delimited.
Date, Time hh:mm:ss, price, volume. The remaining columns need to be discarded.

Output is standard NinjaTrader.

Again, thank you.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #7 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

Hi @Big Mike,

Thanks for giving me the opportunity to test this.

Let's first just handle your "request 1" to see if it works.

I have made three assumptions:
(i) date format in MultiCharts export is dd/MM/yyyy and not MM/dd/yyyy (I cannot check now).
(ii) you do not want header in the output file
(iii) as you wrote, Ninja Trader format is "yyyyMMdd HHmmss" with no ";" in the middle (I do not know Ninja Trader)

If one of these assumptions is wrong, just tell me.

Normally, "size does not matter". So no need to break down the files. However, the conversion can take some time. So perhaps you would want to test on a small file first.

In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).

Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)

Note: the source code (.java file) is useless for the conversion, and is just given for reference.

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.

The 5 first lines of the input file:


Quoting 
"Date","Time","Price","Volume"
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1

have been converted as follows:


Quoting 
20100301 170000;1113.75;2
20100301 170000;1113.75;2
20100301 170000;1113.75;1
20100301 170000;1113.75;1
20100301 170000;1113.75;1



Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req01_v01.zip (4.4 KB, 51 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #8 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,747 since Jun 2009
Thanks: 32,295 given, 97,495 received


Nicolas11 View Post
Hi @Big Mike,

Thanks for giving me the opportunity to test this.

Let's first just handle your "request 1" to see if it works.

I have made three assumptions:
(i) date format in MultiCharts export is dd/MM/yyyy and not MM/dd/yyyy (I cannot check now).
(ii) you do not want header in the output file
(iii) as you wrote, Ninja Trader format is "yyyyMMdd HHmmss" with no ";" in the middle (I do not know Ninja Trader)

Sorry about that. Request 1:

It is m/d/yyyy. Take note, not mm/dd/yyyy.

No header in output file.

Correct, the first field for ninja has a space between date and time.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #9 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

For request 1...

Updated output:

Quoting 
20100103 170000;1113.75;2
20100103 170000;1113.75;2
20100103 170000;1113.75;1
20100103 170000;1113.75;1
20100103 170000;1113.75;1

And updated executable enclosed (v02).

Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req01_v02.zip (4.4 KB, 43 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to Nicolas11 for this post:
 
(login for full post details)
  #10 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received


For "request 2"...

Enclosed please find a dedicated executable (_req02_v01).

The following input lines
 
Code
06/01/2012	05:17:21	12100.00	1	1	12060.00	12100.00	4202	0	0
06/05/2012	09:14:20	12000.00	1	1	12000.00	12018.00	999	0	0
06/07/2012	22:02:33	12442.00	5	5	12442.00	12450.00	2445	0	0
06/10/2012	17:00:00	12524.00	1	1	0.00	12524.00	84	0	0
06/11/2012	06:35:54	12500.00	1	2	12500.00	12552.00	635	0	0
06/11/2012	08:15:50	12498.00	1	4	12498.00	12561.00	671	0	0
have been converted into:
 
Code
20120601 051721;12100.00;1
20120605 091420;12000.00;1
20120607 220233;12442.00;5
20120610 170000;12524.00;1
20120611 063554;12500.00;1
20120611 081550;12498.00;1
Nicolas

PS - Mike, for some reasons, I will have to make a break on this conversion subject between 16:00 EST today and Wednesday morning. If you wish some changes today, do not hesitate! Anyway, we could resume on Wednesday.

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req02_v01.zip (4.4 KB, 80 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 5 users say Thank You to Nicolas11 for this post:
 
(login for full post details)
  #11 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #12 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,747 since Jun 2009
Thanks: 32,295 given, 97,495 received


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

I've just been busy with my own projects. But I'm sure your files will help many others.

Thx

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #13 (permalink)
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 428 given, 201 received


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

This is just the program I was looking for, as I am interested in converting the QCollector data file formats from the GOM/QCollector thread ( ) to the standard NT7 tick import format, which Mike has already pointed out and that can be found here (NinjaTrader Version 7). I will kick the tires on this and let you know how it works.

Thanks for your programming wizardry!

All best,

Aventeren

Reply With Quote
 
(login for full post details)
  #14 (permalink)
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 428 given, 201 received


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

So the program works perfectly. However, I have a couple of suggestions for potential improvements that are based on how the QCollector files are downloaded from the QCOllector thread.

When one downloads the QCollector data from the QCollector thread, the files are all housed in an instrument specific folder. For instance, with the January 2011 QCollector file, the folder contains all of the CL contracts that traded volume during January 2011--and in January 2011, the following CL contracts were traded:

1. CL ##-##.txt
2. CL 01-12.txt
3. CL 02-11.txt
4. CL 02-11.txt
5. CL 03-11.txt
6. CL 04-11.txt
7. CL 05-11.txt
8. CL 06-11.txt
9. CL 07-11.txt
10. CL 08-11.txt
11. CL 09-11.txt
12. CL 10-11.txt
13. CL 11-11.txt
14. CL 12-11.txt

So the procedure to use your tool, would be to drop in the .jar file into the folder, rename the file that you'd like to convert, execute the .jar file, which will in turn create a new file called "input.txt, which will then need to be renamed to the NT7 import requirement of "<instrument name> <mm-yy>.txt" (ie, "CL ##-##.txt" or "CL 02-11.txt"). The tool clearly works, it just requires a few steps that if I could wave my magic wand would be incorporated into the .jar program. So, given the above, here are my magic wand requests:

Request #1: Have the program be placed in a folder with numerous .txt files and ran once. The program would automatically note the file name, which is in the correct format for NT formatting but with data that is not formatted correctly, and copy the original file name. The program would then rename the original file with an "_original" in it and create a new file with the original file name (and which would have the correctly formatted data and file name to import into NT7). For example, the "CL 02-11.txt" original file would be renamed to "CL 02-11_original.txt" and the new file with the properly formatted data would be called "CL 02-11.txt"

Request #2: The newly formatted output files, which would have the correct file name and data formatting, would then be placed in a new folder that would contain all of the converted files.

By making the two magic wand wishes come true, one would then be able to process an entire month's worth of data with a single mouse click--albeit over a considerable amount of time. My thinking here is that this process would just run in the background, and when it was done one could start on the next round of conversions.

At this point it is probably worth noting what I am hoping to accomplish and provide back to the futures.io (formerly BMT) community. If I can make this happen, I plan on posting all of the newly formatted data into the appropriate threads so that other futures.io (formerly BMT) members will just have to download the data instead of going through the process that I've asked you to help out with above--and my time associated with converting all of the available QCollector data for the instruments I am interested in. We're all in this together, and I am hoping that together we can provide something of use to the other members.

Please let me know if you have any questions. I really appreciate the effort that you have put forth thus far on the converter program, and I am hopeful that some facsimile of the above requests can be achieved.

Thank you,

Aventeren

Reply With Quote
 
(login for full post details)
  #15 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #16 (permalink)
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 428 given, 201 received


Nicolas11 View Post
@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Excellent; you are correct that the file name would not have to be renamed if the newly converted files were dropped into a new sub directory.

I can't wait to see what you come up with.

Please let me know if you have any questions.

Thanks and all best,

Aventeren

Reply With Quote
 
(login for full post details)
  #17 (permalink)
Netherlands, Blaricum
 
Experience: Intermediate
Platform: Excel, Python, R
Broker: IB
Trading: Options
 
Zwaen's Avatar
 
Posts: 250 since Dec 2010
Thanks: 848 given, 237 received

Hi Nicolas11,

great program, I was looking for this. Like BigMike posted in request 2, I was also very interested in a converter for Qcollector data to NT-format. This is a horror in excel, besides excel has not enough rows.

I wanted to check the output, and did what you said, put the Qcollector data and the Jar file in the same directory, and try to run the Jar file. Somehow, nothing happens when I double click or run the Jar file:


Java is correctly installed:


Can you see what I do wrong?

Thanks!

One of my worst enemies are my own false assumptions
Reply With Quote
 
(login for full post details)
  #18 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

@Zwaen ,

I do not know where this problem comes from. I have tested on my side just now, and the output file is produced.

Maybe you can try...

Trick 1: right-click on the jar and execute it with administrator privileges

Trick 2: launch the jar from the DOS command line (after having used cd to move the right directory)



Nicolas

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
(login for full post details)
  #19 (permalink)
Netherlands, Blaricum
 
Experience: Intermediate
Platform: Excel, Python, R
Broker: IB
Trading: Options
 
Zwaen's Avatar
 
Posts: 250 since Dec 2010
Thanks: 848 given, 237 received

So the problem is just my computer. I do not see the option to open the file as administrator, but then I will figure it out, or just try the same on another computer /laptop.

Thanks for your help!


Edit: computersavvy as I am , I renamed the Qcollector file "input.txt" as it already was a .txt file. Renaming the file in "input" did the job.

One of my worst enemies are my own false assumptions
Reply With Quote
 
(login for full post details)
  #20 (permalink)
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
 
Nicolas11's Avatar
 
Posts: 1,071 since Aug 2011
Thanks: 2,232 given, 1,751 received

@aventeren ,

Could you test the following?

I propose to use a Windows batch file to apply the jar to the various .txt files of the directory.

1. In the directory of your CL .txt files, make sure that there is :
- no "input.txt"
- no "output.txt"
- no ConvertedFiles sub-directory

2. In the directory of your CL .txt files, copy the jar file

3. Copy/paste the code below into NotePad (or similar) and save it in the CL .txt folder under the name "convert.bat" (no .txt suffix)

4. At this stage, your directory should like:


5. Launch the batch console from the Window "Accessories" Menu.
Navigate to the above-mentioned directory with "cd"
Type "convert"
It should launch the process, ending with something like:


The ConvertedFiles sub-directory should contain the output files.

Nicolas

 
Code
@echo OFF

rem | Batch Conversion
rem | v1
rem | September 5th, 2013
rem | by Nicolas11 @ futures.io (formerly BMT)

echo Batch conversion (v1)
echo ---------------------

if exist ConvertedFiles (
	echo ERROR: Directory ConvertedFiles already exist
	goto End
)

if exist "input.txt" (
	echo ERROR: input.txt already exits
	goto End
)

if exist "output.txt" (
	echo ERROR: output.txt already exits
	goto End
)

echo Creation of ConvertFiles directory
mkdir ConvertedFiles

for /f "delims=" %%a in ('dir /b *.txt') do (
	echo Conversion of %%a
	rename "%%a" "input.txt"
	DataConverter_for_BigMike_req02_v01.jar
	move "output.txt" "ConvertedFiles\%%a" > del123.tmp
	rename "input.txt" "%%a"
)

del del123.tmp

:End
echo End.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to Nicolas11 for this post:
 
(login for full post details)
  #21 (permalink)
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 428 given, 201 received


Nicolas11 View Post
@aventeren ,

Could you test the following?

I propose to use a Windows batch file to apply the jar to the various .txt files of the directory.

1. In the directory of your CL .txt files, make sure that there is :
- no "input.txt"
- no "output.txt"
- no ConvertedFiles sub-directory

2. In the directory of your CL .txt files, copy the jar file

3. Copy/paste the code below into NotePad (or similar) and save it in the CL .txt folder under the name "convert.bat" (no .txt suffix)

4. At this stage, your directory should like:


5. Launch the batch console from the Window "Accessories" Menu.
Navigate to the above-mentioned directory with "cd"
Type "convert"
It should launch the process, ending with something like:


The ConvertedFiles sub-directory should contain the output files.

Nicolas

 
Code
@echo OFF

rem | Batch Conversion
rem | v1
rem | September 5th, 2013
rem | by Nicolas11 @ futures.io (formerly BMT)

echo Batch conversion (v1)
echo ---------------------

if exist ConvertedFiles (
	echo ERROR: Directory ConvertedFiles already exist
	goto End
)

if exist "input.txt" (
	echo ERROR: input.txt already exits
	goto End
)

if exist "output.txt" (
	echo ERROR: output.txt already exits
	goto End
)

echo Creation of ConvertFiles directory
mkdir ConvertedFiles

for /f "delims=" %%a in ('dir /b *.txt') do (
	echo Conversion of %%a
	rename "%%a" "input.txt"
	DataConverter_for_BigMike_req02_v01.jar
	move "output.txt" "ConvertedFiles\%%a" > del123.tmp
	rename "input.txt" "%%a"
)

del del123.tmp

:End
echo End.

I just got back from my wife's 39 week pregnancy check (all is well).

I'll run this now and report back.

Thanks!

Reply With Quote
 
(login for full post details)
  #22 (permalink)
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
 
aventeren's Avatar
 
Posts: 202 since Mar 2013
Thanks: 428 given, 201 received


Nicolas11 View Post
@aventeren ,

Could you test the following?

Well I'm be damned, the batch file approach worked. It took me less than 5 minutes to convert all of the files and then import them to NT7 as tick data. WELL DONE @Nicolas11!!!! :bmt-rocks:

Okay, now for my end. I'll start converting the data and posting it on the CL Historical Tick Data post: I'll make sure to post up my file locations by month in the Summary tab for easy reference.

At this point I am laser focused on the CL, but I may expand to the ES, etc in the future, at which point I will find a similar home for the converted data.

In all seriousness, I really appreciate your time and effort on the quick turnaround from the other side of the world. It's experiences like these that make futures.io (formerly BMT) so amazing. Thanks.

Okay, I'm off.

Reply With Quote
The following 2 users say Thank You to aventeren for this post:
 
(login for full post details)
  #23 (permalink)
London
 
Experience: Beginner
Platform: NinjaTrader
Trading: TF
 
Posts: 2 since Jun 2014
Thanks: 5 given, 1 received

Hi folks / Nicolas -

There appeared to be changes to the QCollector format from March 2013 (from a slash to a dash for date separation with the year reversed in March and again in June 2013 where time then has three further digits). As a result Nicolas' converter Request02 doesn't recognise the format. Is there a chance of a version that could convert the newly exported format(s)? (Or alternatively does anyone have NT7 last tick data for TF, ES, 6E, CL & YM) from March 2013 - Feb end 2014?.... Thanks and greatly appreciate your effort.

Darren

PS. in the meantime I'll try and hack / re-compile Nicolas' original source code and will upload if I get it to work

Reply With Quote


futures io Trading Community Platforms and Indicators > Data converter from a format to another format


Last Updated on March 30, 2015


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1500 prizes from Topstep!

February
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
 

Call Option Buying: The New Pain Trade? w/Carley Garner

Elite only
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts