NexusFi: Find Your Edge


Home Menu

 





Data converter from a format to another format


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one Nicolas11 with 10 posts (24 thanks)
    2. looks_two aventeren with 5 posts (2 thanks)
    3. looks_3 Big Mike with 4 posts (0 thanks)
    4. looks_4 Zwaen with 2 posts (0 thanks)
    1. trending_up 10,821 views
    2. thumb_up 26 thanks given
    3. group 7 followers
    1. forum 22 posts
    2. attach_file 9 attachments




 
Search this Thread

Data converter from a format to another format

  #11 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769

Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Visit my NexusFi Trade Journal Started this thread Reply With Quote
Thanked by:

Can you help answer these questions
from other members on NexusFi?
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
NexusFi Journal Challenge - April 2024
Feedback and Announcements
Better Renko Gaps
The Elite Circle
ZombieSqueeze
Platforms and Indicators
Exit Strategy
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
The Program
18 thanks
GFIs1 1 DAX trade per day journal
18 thanks
  #12 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,396 since Jun 2009
Thanks Given: 33,172
Thanks Received: 101,537


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

I've just been busy with my own projects. But I'm sure your files will help many others.

Thx

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #13 (permalink)
 
aventeren's Avatar
 aventeren 
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
Posts: 202 since Mar 2013
Thanks Given: 428
Thanks Received: 202



Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

This is just the program I was looking for, as I am interested in converting the QCollector data file formats from the GOM/QCollector thread ( ) to the standard NT7 tick import format, which Mike has already pointed out and that can be found here ( NinjaTrader Version 7). I will kick the tires on this and let you know how it works.

Thanks for your programming wizardry!

All best,

Aventeren

Reply With Quote
  #14 (permalink)
 
aventeren's Avatar
 aventeren 
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
Posts: 202 since Mar 2013
Thanks Given: 428
Thanks Received: 202


Nicolas11 View Post
Hi @Big Mike,

I'm back!
Have the two codes fulfilled their mission, or has something failed?

Nicolas

Nicolas--

So the program works perfectly. However, I have a couple of suggestions for potential improvements that are based on how the QCollector files are downloaded from the QCOllector thread.

When one downloads the QCollector data from the QCollector thread, the files are all housed in an instrument specific folder. For instance, with the January 2011 QCollector file, the folder contains all of the CL contracts that traded volume during January 2011--and in January 2011, the following CL contracts were traded:

1. CL ##-##.txt
2. CL 01-12.txt
3. CL 02-11.txt
4. CL 02-11.txt
5. CL 03-11.txt
6. CL 04-11.txt
7. CL 05-11.txt
8. CL 06-11.txt
9. CL 07-11.txt
10. CL 08-11.txt
11. CL 09-11.txt
12. CL 10-11.txt
13. CL 11-11.txt
14. CL 12-11.txt

So the procedure to use your tool, would be to drop in the .jar file into the folder, rename the file that you'd like to convert, execute the .jar file, which will in turn create a new file called "input.txt, which will then need to be renamed to the NT7 import requirement of "<instrument name> <mm-yy>.txt" (ie, "CL ##-##.txt" or "CL 02-11.txt"). The tool clearly works, it just requires a few steps that if I could wave my magic wand would be incorporated into the .jar program. So, given the above, here are my magic wand requests:

Request #1: Have the program be placed in a folder with numerous .txt files and ran once. The program would automatically note the file name, which is in the correct format for NT formatting but with data that is not formatted correctly, and copy the original file name. The program would then rename the original file with an "_original" in it and create a new file with the original file name (and which would have the correctly formatted data and file name to import into NT7). For example, the "CL 02-11.txt" original file would be renamed to "CL 02-11_original.txt" and the new file with the properly formatted data would be called "CL 02-11.txt"

Request #2: The newly formatted output files, which would have the correct file name and data formatting, would then be placed in a new folder that would contain all of the converted files.

By making the two magic wand wishes come true, one would then be able to process an entire month's worth of data with a single mouse click--albeit over a considerable amount of time. My thinking here is that this process would just run in the background, and when it was done one could start on the next round of conversions.

At this point it is probably worth noting what I am hoping to accomplish and provide back to the futures.io (formerly BMT) community. If I can make this happen, I plan on posting all of the newly formatted data into the appropriate threads so that other futures.io (formerly BMT) members will just have to download the data instead of going through the process that I've asked you to help out with above--and my time associated with converting all of the available QCollector data for the instruments I am interested in. We're all in this together, and I am hoping that together we can provide something of use to the other members.

Please let me know if you have any questions. I really appreciate the effort that you have put forth thus far on the converter program, and I am hopeful that some facsimile of the above requests can be achieved.

Thank you,

Aventeren

Reply With Quote
  #15 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769

@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Visit my NexusFi Trade Journal Started this thread Reply With Quote
Thanked by:
  #16 (permalink)
 
aventeren's Avatar
 aventeren 
Bellingham, WA USA
 
Experience: Beginner
Platform: NT
Broker: Mirus (Broker), Continuum (Data), Dorman (Clearing)
Trading: Futures
Posts: 202 since Mar 2013
Thanks Given: 428
Thanks Received: 202


Nicolas11 View Post
@aventeren ,

This seems feasible.

The converted files could be automatically stored in a newly created sub-directory. In such a case, I think that we do not need to rename the original files, since there is no risk of confusion any more. Ok?

What is the converter which works for you? "Big Mike request 2"?

I will try to have a look this week-end.

Nicolas

Excellent; you are correct that the file name would not have to be renamed if the newly converted files were dropped into a new sub directory.

I can't wait to see what you come up with.

Please let me know if you have any questions.

Thanks and all best,

Aventeren

Reply With Quote
  #17 (permalink)
 
Zwaen's Avatar
 Zwaen 
Netherlands, Blaricum
 
Experience: Intermediate
Platform: Excel, Python, R
Broker: IB
Trading: Options
Posts: 250 since Dec 2010
Thanks Given: 848
Thanks Received: 238

Hi Nicolas11,

great program, I was looking for this. Like BigMike posted in request 2, I was also very interested in a converter for Qcollector data to NT-format. This is a horror in excel, besides excel has not enough rows.

I wanted to check the output, and did what you said, put the Qcollector data and the Jar file in the same directory, and try to run the Jar file. Somehow, nothing happens when I double click or run the Jar file:


Java is correctly installed:


Can you see what I do wrong?

Thanks!

One of my worst enemies are my own false assumptions
Reply With Quote
  #18 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769

@Zwaen ,

I do not know where this problem comes from. I have tested on my side just now, and the output file is produced.

Maybe you can try...

Trick 1: right-click on the jar and execute it with administrator privileges

Trick 2: launch the jar from the DOS command line (after having used cd to move the right directory)



Nicolas

Visit my NexusFi Trade Journal Started this thread Reply With Quote
Thanked by:
  #19 (permalink)
 
Zwaen's Avatar
 Zwaen 
Netherlands, Blaricum
 
Experience: Intermediate
Platform: Excel, Python, R
Broker: IB
Trading: Options
Posts: 250 since Dec 2010
Thanks Given: 848
Thanks Received: 238

So the problem is just my computer. I do not see the option to open the file as administrator, but then I will figure it out, or just try the same on another computer /laptop.

Thanks for your help!


Edit: computersavvy as I am , I renamed the Qcollector file "input.txt" as it already was a .txt file. Renaming the file in "input" did the job.

One of my worst enemies are my own false assumptions
Reply With Quote
  #20 (permalink)
 
Nicolas11's Avatar
 Nicolas11 
near Paris, France
 
Experience: Beginner
Platform: -
Trading: -
Posts: 1,071 since Aug 2011
Thanks Given: 2,232
Thanks Received: 1,769


@aventeren ,

Could you test the following?

I propose to use a Windows batch file to apply the jar to the various .txt files of the directory.

1. In the directory of your CL .txt files, make sure that there is :
- no "input.txt"
- no "output.txt"
- no ConvertedFiles sub-directory

2. In the directory of your CL .txt files, copy the jar file

3. Copy/paste the code below into NotePad (or similar) and save it in the CL .txt folder under the name "convert.bat" (no .txt suffix)

4. At this stage, your directory should like:


5. Launch the batch console from the Window "Accessories" Menu.
Navigate to the above-mentioned directory with "cd"
Type "convert"
It should launch the process, ending with something like:


The ConvertedFiles sub-directory should contain the output files.

Nicolas

 
Code
@echo OFF

rem | Batch Conversion
rem | v1
rem | September 5th, 2013
rem | by Nicolas11 @ nexusfi.com (formerly BMT)

echo Batch conversion (v1)
echo ---------------------

if exist ConvertedFiles (
	echo ERROR: Directory ConvertedFiles already exist
	goto End
)

if exist "input.txt" (
	echo ERROR: input.txt already exits
	goto End
)

if exist "output.txt" (
	echo ERROR: output.txt already exits
	goto End
)

echo Creation of ConvertFiles directory
mkdir ConvertedFiles

for /f "delims=" %%a in ('dir /b *.txt') do (
	echo Conversion of %%a
	rename "%%a" "input.txt"
	DataConverter_for_BigMike_req02_v01.jar
	move "output.txt" "ConvertedFiles\%%a" > del123.tmp
	rename "input.txt" "%%a"
)

del del123.tmp

:End
echo End.

Visit my NexusFi Trade Journal Started this thread Reply With Quote
Thanked by:




Last Updated on March 30, 2015


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts