Database setup recommendation - Tech Support | futures io social day trading
futures io futures trading


Database setup recommendation
Updated: Views / Replies:1,512 / 16
Created: by treydog999 Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Database setup recommendation

  #11 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,366 since Jun 2009
Thanks: 3,573 given, 3,982 received


treydog999 View Post
Optimally I would collect each depth of market change, so that i could estimate order cancellation and modification rates. If it do it after each trade, then it would basically just be a random snapshot.
...

You should start by doing some basic statistics and count how many changes/second you have for each tracked instrument.
It's easy using the Ninjatrader OnMarketDepth() for example, then plot the results.
I'm afraid that a single server won't be able to collect 500 symbols level 2 updates, and this is before writing the data somewhere.
Nanex do this, NinjaTrader too for their Market Replay, but not on a single server I think .

Success requires no deodorant! (Sun Tzu)
Reply With Quote
The following user says Thank You to sam028 for this post:
 
  #12 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,240 since Jun 2009
Thanks: 29,352 given, 83,229 received

I would never use HDD for this, only SSD

Sent from my phone

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #13 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,366 since Jun 2009
Thanks: 3,573 given, 3,982 received


I just checked very quickly on ES 03-16 15 minutes after the open today, about 2000 market depth updates/second maximum, the average seems to be around 300/second.
Okay, it's liquid instrument, but if @treydog999 wants to deal with 500 symbols this going to be tricky to manage this kind of frequency.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
 
  #14 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


sam028 View Post
I just checked very quickly on ES 03-16 15 minutes after the open today, about 2000 market depth updates/second maximum, the average seems to be around 300/second.
Okay, it's liquid instrument, but if @treydog999 wants to deal with 500 symbols this going to be tricky to manage this kind of frequency.

That does seem like a lot. I appreciate you taking a look at that for me. What if I did not need to collect the data myself? For example buying it from a tick warehouse / professional data provider. Instead just used this database to store the data after the close and run queries against. That may be a better alternative for me as I definitely will not be able to handle 500 instruments that way, maybe not even 100.

Reply With Quote
 
  #15 (permalink)
 Vendor: www.probabletrades.com 
OC, California, USA
 
Futures Experience: Advanced
Platform: IB/TWS, NinjaTrader, thinkorswim
Favorite Futures: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,859 since Jun 2009
Thanks: 480 given, 3,254 received

I think you should build your software solution first, then scale up as needed. Don't worry about hardware needs for now.

So, get data for, say, 10 symbols. Write your apps or queries that are going to read and write this data. Use it for a while, work out the bugs. Keep adding symbols until your exiting hardware starts to struggle, then that will give you a good idea on how high-end your hardware solution needs to be.

In general, my philosophies towards hardware are
- SSDs always, at least for anything where response time matters. You can use HDD for archive and backups. You can get a 2TB SSD for $600+ now.
- The more RAM the better
- RAID 10 is optimal, at least do RAID 0 for speed and backup transaction logs to a separate drive. You only need to upgrade to RAID 1+0 if you can't miss a single transaction ever in your data stream. If you can re-load it later from an outside source then you don't need RAID 1, just restore from a transaction log backup and reload data from your outside source anything after your last backup.

Initially, I'm guessing you want to play with and research this set of data and come up with some algos based on the historical data, so the always-on, real-time aspect of the server is not crucial if you can reload it later from an outside source. Once you move to trade live money with it then the real-time uptime will be crucial and you'll need to implement the highest degree of speed and availability.

And I don't think mongoDB or any other noSQL solution is right for you. Those are more appropriate for large, unstructured data structures (video, documents, etc.) but if you're dealing with a deterministic structure of numbers then SQL will be superior for you both storage and application-wise (query-ability). However, noSQL has superior horizontal scaling capabilities but poor support for ACID transactions. For more see this SQL vs. noSQL article.

Reply With Quote
The following 2 users say Thank You to shodson for this post:
 
  #16 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


shodson View Post
I think you should build your software solution first, then scale up as needed. Don't worry about hardware needs for now.

So, get data for, say, 10 symbols. Write your apps or queries that are going to read and write this data. Use it for a while, work out the bugs. Keep adding symbols until your exiting hardware starts to struggle, then that will give you a good idea on how high-end your hardware solution needs to be.

In general, my philosophies towards hardware are
- SSDs always, at least for anything where response time matters. You can use HDD for archive and backups. You can get a 2TB SSD for $600+ now.
- The more RAM the better
- RAID 10 is optimal, at least do RAID 0 for speed and backup transaction logs to a separate drive. You only need to upgrade to RAID 1+0 if you can't miss a single transaction ever in your data stream. If you can re-load it later from an outside source then you don't need RAID 1, just restore from a transaction log backup and reload data from your outside source anything after your last backup.

Initially, I'm guessing you want to play with and research this set of data and come up with some algos based on the historical data, so the always-on, real-time aspect of the server is not crucial if you can reload it later from an outside source. Once you move to trade live money with it then the real-time uptime will be crucial and you'll need to implement the highest degree of speed and availability.

And I don't think mongoDB or any other noSQL solution is right for you. Those are more appropriate for large, unstructured data structures (video, documents, etc.) but if you're dealing with a deterministic structure of numbers then SQL will be superior for you both storage and application-wise (query-ability). However, noSQL has superior horizontal scaling capabilities but poor support for ACID transactions. For more see this SQL vs. noSQL article.


Thanks for the comments. Have you heard of KERF? just curious as it is new for time series data. Not well known or battle tested but since you seem knowledgeable i thought i could get your opinion.

In regards to using this for development and research you are bang on. Our production servers use a totally different system that is outsourced solution. I am doing some basic research to see what can be brought in house.

The negative to that is most of the tick data suppliers for large quantities of what i need usually do a 1 time data dump and then FTP uploads daily for updates. I would not be able to get that back by going back to my original data source. For example Bloomberg uses this methodology and that initial data dump is not cheap. So losing that at all would be disastrous, 100x multiple of the cost of my test server. So raid 1+0 is probably the way to go for me. Thoughts on raid 5?

Reply With Quote
 
  #17 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received


treydog999 View Post
The negative to that is most of the tick data suppliers for large quantities of what i need usually do a 1 time data dump and then FTP uploads daily for updates. I would not be able to get that back by going back to my original data source. For example Bloomberg uses this methodology and that initial data dump is not cheap. So losing that at all would be disastrous, 100x multiple of the cost of my test server. So raid 1+0 is probably the way to go for me. Thoughts on raid 5?

Store both the raw files from the FTP dump and the database derived from the raw files. They can just be in two different directories in the same file system.

If you want a cheap and quick backup of your raw files from the FTP dump on top of the copy that resides on your database server, since these are static, you can probably do it for real cheap with Glacier, or worse, a plug-and-play NAS or an external HDD.

Reply With Quote
The following user says Thank You to artemiso for this post:

Reply



futures io > > > > Database setup recommendation

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Broker/data recommendation Jmccoyiii Reviews of Brokers and Data Feeds 4 May 28th, 2015 03:25 AM
intraday trading home study course / setup recommendation contrails The Elite Circle 1 September 25th, 2013 10:10 PM
Trading School Recommendation Request plethora Trading Reviews and Vendors 37 September 4th, 2012 01:02 PM
JPMorgan CEO should not be chairman: recommendation kbit News and Current Events 0 April 30th, 2012 07:35 PM
EL Book Recommendation/s? TonyB EasyLanguage Programming 4 March 24th, 2012 05:19 AM


All times are GMT -4. The time now is 08:52 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-13 in 0.13 seconds with 19 queries on phoenix via your IP 54.160.245.121