How to create a database for backtesting? - futures io
futures io futures trading



How to create a database for backtesting?


Discussion in Traders Hideout

Updated by schematics12
      Top Posters
    1. looks_one rleplae with 3 posts (0 thanks)
    2. looks_two Big Mike with 3 posts (1 thanks)
    3. looks_3 GoldenRatio with 2 posts (0 thanks)
    4. looks_4 baywolf with 1 posts (0 thanks)
    1. trending_up 5,969 views
    2. thumb_up 6 thanks given
    3. group 7 followers
    1. forum 12 replies
    2. attach_file 1 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

How to create a database for backtesting?

(login for full post details)
  #11 (permalink)
Site Administrator
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,050 since Jun 2009
Thanks: 31,658 given, 94,919 received


rleplae View Post
I am able to put something on a chart...

I am not able to create a chart from zero (like a basic series/instrument)
Let say i create synthetic data and want that charted in ninjatrader...

What would you recommend ?

Better to let someone that uses NinjaTrader answer.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my Facebook Visit my futures io Trade Journal Reply With Quote

Can you help answer these questions
from other members on futures io?
Stacked Imbalances.
Sierra Chart
GBP/ARS Pair
Traders Hideout
EQIX - Digital REITs - any experiences?
Stocks and ETFs
help to convert from thinkscript to ninjascript
NinjaTrader
Help finding a web based chart with tick charts and publishing of ideas
Platforms and Indicators
 
 
(login for full post details)
  #12 (permalink)
Gits (Hooglede) Belgium
 
Experience: Master
Platform: NinjaTrader, Proprietary,
Broker: Ninjabrokerage/IQfeed + Synthetic datafeed
Trading: 6A, 6B, 6C, 6E, 6J, 6S, ES, NQ, YM, AEX, CL, NG, ZB, ZN, ZC, ZS, GC
 
rleplae's Avatar
 
Posts: 2,990 since Sep 2013
Thanks: 2,437 given, 5,794 received


Big Mike View Post
Better to let someone that uses NinjaTrader answer.

Mike

A datafeed connector is a solution....

Follow me on Twitter Visit my Facebook Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #13 (permalink)
Los Angeles, CA
 
 
Posts: 14 since Jan 2016
Thanks: 0 given, 5 received


Flat text file-based storage with some clever file naming convention and maybe a directory hierarchy to split contracts by markets and types (i.e. futures from stocks from options, etc...) is a more scalable solution than a traditional DB. Most of the time when you work with historical data (time series) - you don't need to query data, or update /delete records, you just load it up as a sequence of data points, and reading files line-by-line works great. Most languages have support for streaming, so you don't need to load all data into memory at once..

So there is often no need for a SQL-based (or even a non-SQL based) database. Market price data, once obtained and cleaned, is static. You can zip it to save more space, and it's easy to archive. In addition, no DB means one less system to setup, maintain, backup, worry about fail-over, etc... Flat files do not consume CPU power or memory until they are used, unlike a DB engine. You can also easily load comma- or tab- separated files in Excel, or any other tool of your choice.


By the way found a really great tool for getting stocks/futures/options/Forex historical data as csv text files from Interactive Brokers, highly recommend it. You can forget about pacing violation or request/response size limits , it splits large data requests into many smaller once automatically behind the scenes:

Interactive Brokers Historical Data Downloader

Reply With Quote


futures io Trading Community Traders Hideout > How to create a database for backtesting?


March 21, 2016


Upcoming Webinars and Events
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts