Petabyte-scale storage server - Tech Support | futures io social day trading
futures io futures trading


Petabyte-scale storage server
Updated: Views / Replies:735 / 6
Created: by artemiso Attachments:1

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Petabyte-scale storage server

  #1 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

Petabyte-scale storage server

I noticed several members here have plenty of IT experience. Is there any particular hardware that you all might recommend for petabyte-scale storage? Or any existing build lists that we can get ideas out of? We're thinking of building everything from scratch instead of using a turn-key expansion enclosure.

Reply With Quote
 
  #2 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received


artemiso View Post
I noticed several members here have plenty of IT experience. Is there any particular hardware that you all might recommend for petabyte-scale storage? Or any existing build lists that we can get ideas out of? We're thinking of building everything from scratch instead of using a turn-key expansion enclosure.

My last job before trading was VP of a high-end storage company.

It is really impossible to answer your question without knowing a whole lot more about the purpose.

What level of redundancy is needed? Active/active, active/passive, none?
Is it primarily random or sequential IO?
What level of throughput/iops in both scenarios above is required?
How are you going to access the data (FC SAN, iSCSI, etc)?
How many hosts are going to connect to it at once?
What type of file system requirements do you have? One huge file system? 100 small file systems?
How do you want to handle file sharing and LUN provisioning?

In the end, you can just build your own bare 4U chassis loaded with as many 3TB spindles as can fit, and get dozens of SAS cards and attach them to hosts running a Linux server. That is the poor man solution that will also require the most experience to handle.

If you are looking at Tier 1 storage and a seven figure budget, then the above would be laughed out of the park.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received


BTW, I haven't even talked about disaster recovery, snapshots, etc. Depending on your requirements, you may be able to do it at the software level, or may need a hardware solution to do it.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #4 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

Thanks @Big Mike, excellent hardware advice as always. I'll answer to my best ability, but I have to confess this is not an area of my expertise.

In the end, you can just build your own bare 4U chassis loaded with as many 3TB spindles as can fit, and get dozens of SAS cards and attach them to hosts running a Linux server. That is the poor man solution that will also require the most experience to handle.

If you are looking at Tier 1 storage and a seven figure budget, then the above would be laughed out of the park.


Right. Currently our data storage is completely outsourced. There's a submarine cable that goes directly across the river to a data center in Boston. As one might guess, this is beginning to be expensive and we're looking to mitigate costs. The current arrangement will still be kept as it works well for some kind of operational redundancy (disaster recovery as you put it), but the plan is to set up a storage server inhouse. We've thought of hiring a dedicated database engineer to solve the problem - I'm surprised how far we've gone without one.

Our current, first-impression plan was indeed as you've described: poor man's solution, bare 4U chassis with plenty of 2 TB spindles, several stacks of these, probably nets the highest capacity/cost ratio and is very achievable with a 6-digit budget. What are the drawbacks?

A reason I'm asking for general ideas is not that I have to be doing most of the assembly, but I'm ultimately responsible for allocating the budget and it would be good to have an informed opinion before I green-light a build. Is there any reading material that you'd recommend in this area?

What level of redundancy is needed? Active/active, active/passive, none? How do you want to handle file sharing and LUN provisioning?
Active/active. Never thought too far. Opinions?

Is it primarily random or sequential IO?
Random.

What level of throughput/iops in both scenarios above is required? How are you going to access the data (FC SAN, iSCSI, etc)?
Haven't decided on the former, revamping a lot of our software layer lately. 16 Gb FC SAN. Possibly IB-based SAN.

How many hosts are going to connect to it at once?
<=15.

What type of file system requirements do you have? One huge file system? 100 small file systems?
One huge.

Reply With Quote
 
  #5 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,214 received


artemiso View Post
Thanks @Big Mike, excellent hardware advice as always. I'll answer to my best ability, but I have to confess this is not an area of my expertise.

In the end, you can just build your own bare 4U chassis loaded with as many 3TB spindles as can fit, and get dozens of SAS cards and attach them to hosts running a Linux server. That is the poor man solution that will also require the most experience to handle.

If you are looking at Tier 1 storage and a seven figure budget, then the above would be laughed out of the park.


Right. Currently our data storage is completely outsourced. There's a submarine cable that goes directly across the river to a data center in Boston. As one might guess, this is beginning to be expensive and we're looking to mitigate costs. The current arrangement will still be kept as it works well for some kind of operational redundancy (disaster recovery as you put it), but the plan is to set up a storage server inhouse. We've thought of hiring a dedicated database engineer to solve the problem - I'm surprised how far we've gone without one.

Our current, first-impression plan was indeed as you've described: poor man's solution, bare 4U chassis with plenty of 2 TB spindles, several stacks of these, probably nets the highest capacity/cost ratio and is very achievable with a 6-digit budget. What are the drawbacks?

A reason I'm asking for general ideas is not that I have to be doing most of the assembly, but I'm ultimately responsible for allocating the budget and it would be good to have an informed opinion before I green-light a build. Is there any reading material that you'd recommend in this area?

What level of redundancy is needed? Active/active, active/passive, none? How do you want to handle file sharing and LUN provisioning?
Active/active. Never thought too far. Opinions?

Is it primarily random or sequential IO?
Random.

What level of throughput/iops in both scenarios above is required? How are you going to access the data (FC SAN, iSCSI, etc)?
Haven't decided on the former, revamping a lot of our software layer lately. 16 Gb FC SAN. Possibly IB-based SAN.

How many hosts are going to connect to it at once?
<=15.

What type of file system requirements do you have? One huge file system? 100 small file systems?
One huge.

Well keep in mind your on a trading site, and I left the industry 6 years ago.

That said, the first trouble area is wanting a single enormous file system. Technically you can do it but there are risks.

Random IO is also a bit of an issue, but you didn't give an IOPS number so its unclear how demanding the application is. If it's not that demanding, just enormous, then you can save a lot of money.

Basically, demanding random IO (high IOPS) means you need lots of SSD's. Otherwise you could get away with 3TB spindles for the most part, and then just a few (say 10%) SSD's if you use the right kind of file system, like ZFS, where it can cache to SSD for near line and push the other stuff to the slower spindles.

If you want full redundancy, it really means you should invest in at least a hardware raid controller head unit. For this level of capacity, it means you would need several of those. Each head unit can manage 100-200TB of capacity, depending on the level of performance you need. So each head unit would have say (100) 3TB spindles, and (10) 512GB SSD spindles hanging off it.

At this point you have the option of whether you want to present the spindles raw to the host, which might be preferred in some situations with ZFS, or if you want it to manage the RAID directly. Maybe (5) RAID 10 arrays, each made up of 20 spindles, for example on the HDD side, and (1) RAID 10 array on the SSD side. RAID 10 is best for high random IO but at high cost (more spindles) vs RAID 5 or RAID 6. Each box would give you roughly 50*2.6TB[usable] so 130TB usable capacity, plus the SSD cache for ZFS of roughly 2.2TB usable.

The hardware raid head units have two controllers per head unit, and each drive bay should be dual-port so there is two paths to the drive. This provides redundancy. Each of these head units would connect to your FC fabric.

You might check names like Infortrend and Promise for some example 1U active/active hardware raid head units, and they'll just use SFP to connect from the head units to the SAS JBOD chassis.

The next level up from there would be to buy a tier 1 named solution, at many times the price, but also guaranteed service levels and such. Probably 5x the cost of just buying the head units and JBOD chassis from Infortrend or etc, and buying your own drives, and managing all of it yourself.

As for where to turn for help, sorry but I am out of the industry and the company I was the VP of went out of business shortly after I quit. I would start with Infortrend or Promise website, see if any of their solutions make sense, and find a reseller. You can then issue some RFQ's.

You should also check hardforum.com and ask for advice there.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 3 users say Thank You to Big Mike for this post:
 
  #6 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

@Big Mike

Great tips. We're starting work on this project. More complicated than initially expected and we might have to expand our office space because of the servers humming away. Will let you know how it goes.

Reply With Quote
 
  #7 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

Confirmed! Breaking the contract on our current office lease to move. New office needs to be rewired for the considerable power delivery, three-phase etc. I wonder how other people are coping with their servers. Hope you're enjoying your holiday.

Reply With Quote

Reply



futures io > > > > Petabyte-scale storage server

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Tick Database Storage MXASJ Platforms and Indicators 40 July 20th, 2012 01:30 PM
Quantify an indicator scale with Price scale deercreek NinjaTrader 3 April 30th, 2012 04:53 PM
Scale In or Scale Out - The Latest Thoughts tderrick Psychology and Money Management 12 March 15th, 2012 08:49 AM
U.S. Strategic Reserve Oil May Be Put in Storage by Traders kbit News and Current Events 1 July 6th, 2011 08:28 PM
Tick Data Storage timmyb The Elite Circle 9 June 16th, 2010 05:44 AM


All times are GMT -4. The time now is 11:42 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-11 in 0.13 seconds with 19 queries on phoenix via your IP 54.82.81.154