This thread is a possible place to obtain a small software allowing the conversion of data from a format to another format.
For the avoidance of doubt, it’s free.
For instance, let’s suppose that you have a 1-Go text file with data formatted like:
05/24/2012 23:45 716,50 713,00 714,25 715,75 12563
… and you want to convert into a text file with the following format:
1120525,0045,716.50,713.00,714.25,715.75,12563
(time in output file shall be one hour later than in the input file).
Then...
1. In this thread, post: For input file:
- the 5 first lines of your data file
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- has the data file a first line with titles of the fields or not
- size of the input file For output file:
- format of the lines
- explanation for each field
- identification of the delimiter (tabulation, comma, etc.)
- should the data file have a first line with titles of the fields or not
2. Then, after some time (typically: from 1 day to 1 week), I will post in this thread:
- a JAR file (it is executable, same as EXE file),
- source code in Java.
The source code is useless to execute the JAR file. It is just given for reference.
3. Then, you will have to:
- save the JAR file in the same directory as your data file
- rename your data file as "input.txt"
- execute the JAR file (by double-clicking on it)
Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software
The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.
Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximatively the size of the input file.
An example of such JAR file and source code is given in a following message.
4. Obviously, you will have to check the content of output.txt
5.These instructions may be updated in the "Quick Summary" below.
6. I do not promise to provide this service forever. If there are too many requests or if it is not possible any more for me to continue for any reason, I will kindly ask Big Mike to close this thread.
In case of problems in the execution of the JAR file, I will do my best to help, but in limited time. If we do not find a solution quickly, I will perhaps have to give up. As a reminder, this service is free.
Nicolas
The following 6 users say Thank You to Nicolas11 for this post:
Could you please provide with an executable file allowing the following conversion?
Input file:
Five first lines:
Field delimiter is: tabulation
Fields are: date MM/DD/YYY, hour HH:mm, H, L, O, C, volume
The file contains a first line with the titles of the fields
The size of the input file is 1Go
Output file:
1120525,0045,716.50,713.00,714.25,715.75,12563
Field delimiter is: comma
Field are: date YYYMMDD, hour HHmm, H, L, O, C , volume Time is in output file shall be one hour later than in the input file
Note: commas within the numbers shall be replaced by dots
No first line with the titles of the fields
Thanks in advance.
The following user says Thank You to Nicolas11 for this post:
In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).
Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)
Note: the source code (.java file) is useless for the conversion, and is just given for reference.
Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software
The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.
Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.
It is your responsibility to check the content of output.txt
The 5 first lines of the input file:
have been converted as follows:
On my computer, the conversion of a 1 Go input file has taken 25 minutes.
Nicolas
The following user says Thank You to Nicolas11 for this post:
Thanks for giving me the opportunity to test this.
Let's first just handle your "request 1" to see if it works.
I have made three assumptions:
(i) date format in MultiCharts export is dd/MM/yyyy and not MM/dd/yyyy (I cannot check now).
(ii) you do not want header in the output file
(iii) as you wrote, Ninja Trader format is "yyyyMMdd HHmmss" with no ";" in the middle (I do not know Ninja Trader)
If one of these assumptions is wrong, just tell me.
Normally, "size does not matter". So no need to break down the files. However, the conversion can take some time. So perhaps you would want to test on a small file first.
In the enclosed ZIP file, you will find an executable JAR file and a source code (.java file).
Next steps on your side:
- Save the JAR file in the same directory as your data file.
- Rename your data file as "input.txt"
- Execute the JAR file (by double-clicking on it)
Note: the source code (.java file) is useless for the conversion, and is just given for reference.
Note: the execution of the JAR file requires Java to be installed on your computer. It is usually the case. Otherwise, it could be downloaded there: Download Free Java Software
The application will generate in the same directory a file called "output.txt" in the required format.
Take care: any file already named "output.txt" in the directory will be deleted and replaced.
Note: if your input file is very big, the execution could take some time (for instance: 25 minutes for a 1 Go file). But you can check that the process is progressing by refreshing your Windows Explorer window (press F5): you should see the size of the output file growing until it reaches approximately the size of the input file.
The 5 first lines of the input file:
have been converted as follows:
Nicolas
The following user says Thank You to Nicolas11 for this post:
Enclosed please find a dedicated executable (_req02_v01).
The following input lines
have been converted into:
Nicolas
PS - Mike, for some reasons, I will have to make a break on this conversion subject between 16:00 EST today and Wednesday morning. If you wish some changes today, do not hesitate! Anyway, we could resume on Wednesday.
The following 5 users say Thank You to Nicolas11 for this post:
I keep getting requests for people to share their GomRecorder data. A few threads have been started on the subject but not many are following thru it would seem.
I am collecting data from IQFeed via QCollector. All data contains bid/ask and is tick …
) to the standard NT7 tick import format, which Mike has already pointed out and that can be found here (NinjaTrader Version 7). I will kick the tires on this and let you know how it works.
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
Posts: 202 since Mar 2013
Thanks: 428 given,
201
received
Nicolas--
So the program works perfectly. However, I have a couple of suggestions for potential improvements that are based on how the QCollector files are downloaded from the QCOllector thread.
When one downloads the QCollector data from the QCollector thread, the files are all housed in an instrument specific folder. For instance, with the January 2011 QCollector file, the folder contains all of the CL contracts that traded volume during January 2011--and in January 2011, the following CL contracts were traded:
So the procedure to use your tool, would be to drop in the .jar file into the folder, rename the file that you'd like to convert, execute the .jar file, which will in turn create a new file called "input.txt, which will then need to be renamed to the NT7 import requirement of "<instrument name> <mm-yy>.txt" (ie, "CL ##-##.txt" or "CL 02-11.txt"). The tool clearly works, it just requires a few steps that if I could wave my magic wand would be incorporated into the .jar program. So, given the above, here are my magic wand requests:
Request #1: Have the program be placed in a folder with numerous .txt files and ran once. The program would automatically note the file name, which is in the correct format for NT formatting but with data that is not formatted correctly, and copy the original file name. The program would then rename the original file with an "_original" in it and create a new file with the original file name (and which would have the correctly formatted data and file name to import into NT7). For example, the "CL 02-11.txt" original file would be renamed to "CL 02-11_original.txt" and the new file with the properly formatted data would be called "CL 02-11.txt"
Request #2: The newly formatted output files, which would have the correct file name and data formatting, would then be placed in a new folder that would contain all of the converted files.
By making the two magic wand wishes come true, one would then be able to process an entire month's worth of data with a single mouse click--albeit over a considerable amount of time. My thinking here is that this process would just run in the background, and when it was done one could start on the next round of conversions.
At this point it is probably worth noting what I am hoping to accomplish and provide back to the futures.io (formerly BMT) community. If I can make this happen, I plan on posting all of the newly formatted data into the appropriate threads so that other futures.io (formerly BMT) members will just have to download the data instead of going through the process that I've asked you to help out with above--and my time associated with converting all of the available QCollector data for the instruments I am interested in. We're all in this together, and I am hoping that together we can provide something of use to the other members.
Please let me know if you have any questions. I really appreciate the effort that you have put forth thus far on the converter program, and I am hopeful that some facsimile of the above requests can be achieved.
The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?
What is the converter which works for you? "Big Mike request 2"?
I will try to have a look this week-end.
Nicolas
The following user says Thank You to Nicolas11 for this post:
great program, I was looking for this. Like BigMike posted in request 2, I was also very interested in a converter for Qcollector data to NT-format. This is a horror in excel, besides excel has not enough rows.
I wanted to check the output, and did what you said, put the Qcollector data and the Jar file in the same directory, and try to run the Jar file. Somehow, nothing happens when I double click or run the Jar file:
Java is correctly installed:
Can you see what I do wrong?
Thanks!
One of my worst enemies are my own false assumptions
So the problem is just my computer. I do not see the option to open the file as administrator, but then I will figure it out, or just try the same on another computer /laptop.
Thanks for your help!
Edit: computersavvy as I am , I renamed the Qcollector file "input.txt" as it already was a .txt file. Renaming the file in "input" did the job.
One of my worst enemies are my own false assumptions
I propose to use a Windows batch file to apply the jar to the various .txt files of the directory.
1. In the directory of your CL .txt files, make sure that there is :
- no "input.txt"
- no "output.txt"
- no ConvertedFiles sub-directory
2. In the directory of your CL .txt files, copy the jar file
3. Copy/paste the code below into NotePad (or similar) and save it in the CL .txt folder under the name "convert.bat" (no .txt suffix)
4. At this stage, your directory should like:
5. Launch the batch console from the Window "Accessories" Menu.
Navigate to the above-mentioned directory with "cd"
Type "convert"
It should launch the process, ending with something like:
The ConvertedFiles sub-directory should contain the output files.
Nicolas
The following 3 users say Thank You to Nicolas11 for this post:
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
Posts: 202 since Mar 2013
Thanks: 428 given,
201
received
Well I'm be damned, the batch file approach worked. It took me less than 5 minutes to convert all of the files and then import them to NT7 as tick data. WELL DONE @Nicolas11!!!! :bmt-rocks:
Okay, now for my end. I'll start converting the data and posting it on the CL Historical Tick Data post:
Here is over 1 year of CL crude oil tick data from Zen Fire, and the same period of minute data from Zen Fire.
It starts 1/1/2009 and ends 3/26/2010.
You can import this easily into most platforms. The time zone is Central (GMT -0600).
I include both …
I'll make sure to post up my file locations by month in the Summary tab for easy reference.
At this point I am laser focused on the CL, but I may expand to the ES, etc in the future, at which point I will find a similar home for the converted data.
In all seriousness, I really appreciate your time and effort on the quick turnaround from the other side of the world. It's experiences like these that make futures.io (formerly BMT) so amazing. Thanks.
Okay, I'm off.
The following 2 users say Thank You to aventeren for this post:
There appeared to be changes to the QCollector format from March 2013 (from a slash to a dash for date separation with the year reversed in March and again in June 2013 where time then has three further digits). As a result Nicolas' converter Request02 doesn't recognise the format. Is there a chance of a version that could convert the newly exported format(s)? (Or alternatively does anyone have NT7 last tick data for TF, ES, 6E, CL & YM) from March 2013 - Feb end 2014?.... Thanks and greatly appreciate your effort.
Darren
PS. in the meantime I'll try and hack / re-compile Nicolas' original source code and will upload if I get it to work
, I renamed the Qcollector file "input.txt" as it already was a .txt file. Renaming the file in "input" did the job.
: