Data converter from a format to another format - Platforms and Indicators | futures io social day trading
futures io futures trading


Data converter from a format to another format
Updated: Views / Replies:4,290 / 22
Created: by Nicolas11 Attachments:9

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 9  
 
Thread Tools Search this Thread
 

Data converter from a format to another format

  #1 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Data converter from a format to another format

This thread is a possible place to obtain a small software allowing the conversion of data from a format to another format.

For the avoidance of doubt, it’s free.

For instance, let’s suppose that you have a 1-Go text file with data formatted like:
05/24/2012 23:45 716,50 713,00 714,25 715,75 12563
… and you want to convert into a text file with the following format:
1120525,0045,716.50,713.00,714.25,715.75,12563
(time in output file shall be one hour later than in the input file).

Then...

1. In this thread, post:
For input file:
- the 5 first lines of your data file
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- has the data file a first line with titles of the fields or not
- size of the input file
For output file:
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- should the data file have a first line with titles of the fields or not

An example of such request is given in the following message.

2. Then, after some time (typically: from 1 day to 1 week), I will post in this thread:
- a JAR file (it is executable, same as EXE file),
- source code in Java.
The source code is useless to execute the JAR file. It is just given for reference.

3. Then, you will have to:
- save the JAR file in the same directory as your data file
- rename your data file as "input.txt"
- execute the JAR file (by double-clicking on it)

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximatively the size of the input file.

An example of such JAR file and source code is given in a following message.

4. Obviously, you will have to check the content of output.txt

5. These instructions may be updated in the "Quick Summary" below.

6. I do not promise to provide this service forever. If there are too many requests or if it is not possible any more for me to continue for any reason, I will kindly ask Big Mike to close this thread.

In case of problems in the execution of the JAR file, I will do my best to help, but in limited time. If we do not find a solution quickly, I will perhaps have to give up. As a reminder, this service is free.

Nicolas


Last edited by Nicolas11; July 11th, 2012 at 02:35 PM.
Reply With Quote
The following 6 users say Thank You to Nicolas11 for this post:
 
  #3 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Example of request


This message is an example of request.

Hi,

Could you please provide with an executable file allowing the following conversion?

Input file:
Five first lines:

Quoting 
Date Time High Low Open Close Vol
05/24/2012 22:30 716,50 713,00 714,25 715,75 26540
05/24/2012 22:45 716,50 713,00 714,25 715,75 12563
05/24/2012 23:00 716,50 713,00 714,25 715,75 13562
05/24/2012 23:15 716,50 713,00 714,25 715,75 9856

Field delimiter is: tabulation
Fields are: date MM/DD/YYY, hour HH:mm, H, L, O, C, volume
The file contains a first line with the titles of the fields
The size of the input file is 1Go

Output file:
1120525,0045,716.50,713.00,714.25,715.75,12563
Field delimiter is: comma
Field are: date YYYMMDD, hour HHmm, H, L, O, C , volume
Time is in output file shall be one hour later than in the input file
Note: commas within the numbers shall be replaced by dots
No first line with the titles of the fields

Thanks in advance.

Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #4 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Example of answer

This message is an example of answer.

Hi,

In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).

Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)

Note: the source code (.java file) is useless for the conversion, and is just given for reference.

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.

It is your responsibility to check the content of output.txt

The 5 first lines of the input file:

Quoting 
Date Time High Low Open Close Vol
05/24/2012 22:30 716,50 713,00 714,25 715,75 26540
05/24/2012 22:45 716,50 713,00 714,25 715,75 12563
05/24/2012 23:00 716,50 713,00 714,25 715,75 13562
05/24/2012 23:15 716,50 713,00 714,25 715,75 9856

have been converted as follows:

Quoting 
1120524,2330,716.5,713,714.25,715.75,26540
1120524,2345,716.5,713,714.25,715.75,12563
1120525,0000,716.5,713,714.25,715.75,13562
1120525,0015,716.5,713,714.25,715.75,9856

On my computer, the conversion of a 1 Go input file has taken 25 minutes.

Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_Nicolas11_v01.zip (4.8 KB, 52 views)

Last edited by Nicolas11; July 11th, 2012 at 03:07 PM.
Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #5 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,353 given, 83,234 received

Continuing from https://futures.io/elite-circle/21502-standalone-tick-minute-data-converter-cross-platform.html

Source file is MultiCharts export:

 
Code
"Date","Time","Price","Volume"
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
Includes header line

Output file should be standard NinjaTrader yyyyMMdd HHmmss;price;volume

Are there any file size limits? I need to know if I need to break up to < 2GB or < 4GB files.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #6 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,353 given, 83,234 received

Request 2

Source is QCollector:
 
Code
06/01/2012	05:17:21	12100.00	1	1	12060.00	12100.00	4202	0	0
06/05/2012	09:14:20	12000.00	1	1	12000.00	12018.00	999	0	0
06/07/2012	22:02:33	12442.00	5	5	12442.00	12450.00	2445	0	0
06/10/2012	17:00:00	12524.00	1	1	0.00	12524.00	84	0	0
06/11/2012	06:35:54	12500.00	1	2	12500.00	12552.00	635	0	0
06/11/2012	08:15:50	12498.00	1	4	12498.00	12561.00	671	0	0
No header line. Tab delimited.
Date, Time hh:mm:ss, price, volume. The remaining columns need to be discarded.

Output is standard NinjaTrader.

Again, thank you.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #7 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

Hi @Big Mike,

Thanks for giving me the opportunity to test this.

Let's first just handle your "request 1" to see if it works.

I have made three assumptions:
(i) date format in MultiCharts export is dd/MM/yyyy and not MM/dd/yyyy (I cannot check now).
(ii) you do not want header in the output file
(iii) as you wrote, Ninja Trader format is "yyyyMMdd HHmmss" with no ";" in the middle (I do not know Ninja Trader)

If one of these assumptions is wrong, just tell me.

Normally, "size does not matter". So no need to break down the files. However, the conversion can take some time. So perhaps you would want to test on a small file first.

In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).

Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)

Note: the source code (.java file) is useless for the conversion, and is just given for reference.

Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software

The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.

Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.

The 5 first lines of the input file:


Quoting 
"Date","Time","Price","Volume"
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,2
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1
1/3/2010,17:00:00,1113.75,1

have been converted as follows:


Quoting 
20100301 170000;1113.75;2
20100301 170000;1113.75;2
20100301 170000;1113.75;1
20100301 170000;1113.75;1
20100301 170000;1113.75;1



Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req01_v01.zip (4.4 KB, 43 views)
Reply With Quote
The following user says Thank You to Nicolas11 for this post:
 
  #8 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,353 given, 83,234 received


Nicolas11 View Post
Hi @Big Mike,

Thanks for giving me the opportunity to test this.

Let's first just handle your "request 1" to see if it works.

I have made three assumptions:
(i) date format in MultiCharts export is dd/MM/yyyy and not MM/dd/yyyy (I cannot check now).
(ii) you do not want header in the output file
(iii) as you wrote, Ninja Trader format is "yyyyMMdd HHmmss" with no ";" in the middle (I do not know Ninja Trader)

Sorry about that. Request 1:

It is m/d/yyyy. Take note, not mm/dd/yyyy.

No header in output file.

Correct, the first field for ninja has a space between date and time.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #9 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received

For request 1...

Updated output:

Quoting 
20100103 170000;1113.75;2
20100103 170000;1113.75;2
20100103 170000;1113.75;1
20100103 170000;1113.75;1
20100103 170000;1113.75;1

And updated executable enclosed (v02).

Nicolas

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req01_v02.zip (4.4 KB, 37 views)
Reply With Quote
The following 4 users say Thank You to Nicolas11 for this post:
 
  #10 (permalink)
Elite Member
near Paris, France
 
Futures Experience: Beginner
Platform: -
Favorite Futures: -
 
Nicolas11's Avatar
 
Posts: 1,070 since Aug 2011
Thanks: 2,232 given, 1,729 received


For "request 2"...

Enclosed please find a dedicated executable (_req02_v01).

The following input lines
 
Code
06/01/2012	05:17:21	12100.00	1	1	12060.00	12100.00	4202	0	0
06/05/2012	09:14:20	12000.00	1	1	12000.00	12018.00	999	0	0
06/07/2012	22:02:33	12442.00	5	5	12442.00	12450.00	2445	0	0
06/10/2012	17:00:00	12524.00	1	1	0.00	12524.00	84	0	0
06/11/2012	06:35:54	12500.00	1	2	12500.00	12552.00	635	0	0
06/11/2012	08:15:50	12498.00	1	4	12498.00	12561.00	671	0	0
have been converted into:
 
Code
20120601 051721;12100.00;1
20120605 091420;12000.00;1
20120607 220233;12442.00;5
20120610 170000;12524.00;1
20120611 063554;12500.00;1
20120611 081550;12498.00;1
Nicolas

PS - Mike, for some reasons, I will have to make a break on this conversion subject between 16:00 EST today and Wednesday morning. If you wish some changes today, do not hesitate! Anyway, we could resume on Wednesday.

Attached Files
Register to download File Type: zip DataConverter_for_BigMike_req02_v01.zip (4.4 KB, 75 views)
Reply With Quote
The following 5 users say Thank You to Nicolas11 for this post:

Reply



futures io > > > Data converter from a format to another format

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
possible to convert downloaded historical data to replay format (.ntm) gztanwei NinjaTrader Programming 10 August 24th, 2012 12:37 AM
here's way to convert tick data from NT to comma-separated format, with Mac or Linux theorist Platforms and Indicators 1 July 1st, 2012 05:00 AM
software with milliseconds and L2 in data format bidask201 Platforms and Indicators 2 May 13th, 2012 08:10 PM
ideal data format for MC mea109 MultiCharts 3 September 9th, 2011 04:36 AM
What should I have as 'days back' under 'Format Data Series' eone Traders Hideout 5 February 24th, 2010 07:38 AM


All times are GMT -4. The time now is 05:25 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-16 in 0.17 seconds with 20 queries on phoenix via your IP 54.160.245.121