Buying / Building an Enterprise Server for Trading Research - Tech Support | futures io social day trading
futures io futures trading


Buying / Building an Enterprise Server for Trading Research
Updated: Views / Replies:737 / 7
Created: by treydog999 Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Buying / Building an Enterprise Server for Trading Research

  #1 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received

Buying / Building an Enterprise Server for Trading Research

I am fairly well versed in desktops and laptops. But when it comes to Enterprise Rack Servers I am completely lost. I have been looking at Dell, Lenovo, and building it myself from enterprise grade parts from Newegg. I am looking at 2 servers 1 as a data base/ storage unit and 1 as a research / number crunching monster. The type of data used will be raw tick data, this includes all exchange messages not just trades, as well as changes to level 2 either 5 deep or 10 deep each side depending on the instrument. I don't think i can roll these into 2 huge 2U or 4U server to solve both purposes. My budget for each server is about 15k. Less is better though :P

Any recommendations on set ups for the storage server? Raid configs i am between 10 and 5? Either pre built or build your own?

Any recommendations for the number cruncher? Dual processors and large memory capacity is probably required.

Thanks for the help.

Reply With Quote
 
  #2 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,366 since Jun 2009
Thanks: 3,573 given, 3,982 received

How many usable GB/TB will you need for your storage server? How many IOPS is required to crunch your data? Can you afford to lose some data if you lose a disk?
That is required to answer how many axes are required, what kind of disks (sata/sas/ssd) and the raid card type (cache, bbu, ...). It's possible to mix the disk types, with cheap and big sata disk to backup sas/ssd disks which contains the real data and could be with redundancy (Raid 0).
I usually avoir Raid 5, Raid 50 is a better option, but I only use Raid 10 myself.

Is the number cruncher will be highly multithreaded and scalable? As your budget is large enough an high end Intel E5-26xx will do the job (E5-2660 V2 and higher). Avoid AMD processors, more cores/socket but so slow...

I would recommend to not build it from individual parts. It's a bit cheaper but you may have a compatibility issues and it's at the end a waste of time (so a waste of $$).
In buying a Dell/IBM/HP you'll get less troubles and are sure that everything is working well together. You'll also be sure to find replacement parts years later.
I've tested and worked with Dell/IBM/HP and few others like Fujitsu/Sun/..., HP 1U/2U servers are the best.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
 
  #3 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received



sam028 View Post
How many usable GB/TB will you need for your storage server? How many IOPS is required to crunch your data? Can you afford to lose some data if you lose a disk?
That is required to answer how many axes are required, what kind of disks (sata/sas/ssd) and the raid card type (cache, bbu, ...). It's possible to mix the disk types, with cheap and big sata disk to backup sas/ssd disks which contains the real data and could be with redundancy (Raid 0).
I usually avoir Raid 5, Raid 50 is a better option, but I only use Raid 10 myself.

Is the number cruncher will be highly multithreaded and scalable? As your budget is large enough an high end Intel E5-26xx will do the job (E5-2660 V2 and higher). Avoid AMD processors, more cores/socket but so slow...

I would recommend to not build it from individual parts. It's a bit cheaper but you may have a compatibility issues and it's at the end a waste of time (so a waste of $$).
In buying a Dell/IBM/HP you'll get less troubles and are sure that everything is working well together. You'll also be sure to find replacement parts years later.
I've tested and worked with Dell/IBM/HP and few others like Fujitsu/Sun/..., HP 1U/2U servers are the best.

Hi sam thanks for reply, I will answer as best I can since this is really out of my depth.

10-20 Usable TB would be great IOPS can be fairly low ~150 for the storage machine. I think raid 10 is the safest and most reliable for critical data, so I will probably be going with that. Even though raid 5 would give me better performance and more usable TBs. It will not be recording the raw tick data live, but just store it after it has already been buffered. Losing data should be avoided but is not entirely catastrophic.

For the number cruncher, multi threaded and scalable would be a huge plus. So that as the processes grow the infrastructure can grow as well. Those AMDs seem like the better choice as the core and speed ratio is much better number wise. But I think intel for valuable systems has always been my choice.

Reply With Quote
 
  #4 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

@treydog999

Where are you going to store your servers? I'm guessing it's much cheaper in S.K. but in the U.S., I'd set aside about $1-1.5k/month for the cabinet space and power, another $200-500/month for the cross-connects to your bandwidth provider, and another $400-4000/month for your bandwidth provider(s). Maybe cut those prices in 1/3 since you just need to host 2 servers. If you're working at some co-working office space, I suppose you can find space for much cheaper.

Based on what you've said, I highly recommend that you set aside a bit of your budget and pick up 1-2 Westmere spec'ed servers from eBay for $150-$250 each and 4-6 cheap disks with matching trays (for $20+ each) and test out your software setup (OS, RAID settings, data samples etc.) and infrastructure (cooling, power, network management etc.) before you spend $15k on each server.


sam028 View Post
I've tested and worked with Dell/IBM/HP and few others like Fujitsu/Sun/..., HP 1U/2U servers are the best.

Hm, I'd be interested to hear what made you come to that conclusion.

I've always felt IBM components were more robust, documented and replaceable than HP parts; I'd give Lenovo a try the next time I'm looking for a ready-built server.

Someone I know just launched a startup that required a cluster of 30+ servers in a data center deployment, not many but still enough that Dell, HP, Supermicro and Tyan sent him samples of their most highly spec'ed servers and he ultimately decided that Dell was the easiest to maintain and service. I guess your mileage will vary.



sam028 View Post
I would recommend to not build it from individual parts. It's a bit cheaper but you may have a compatibility issues and it's at the end a waste of time (so a waste of $$). In buying a Dell/IBM/HP you'll get less troubles and are sure that everything is working well together. You'll also be sure to find replacement parts years later.

That's true. It's sometimes difficult to get the different permutations of heat sinks, PCIe riser cards, extra power cables, server case IO slots, motherboard form factors, additional fans, SAS fanout cables etc. correct. 1-2 server specs is probably manageable but it becomes a dependency hell when you reach 5-6 different kinds of specs.



treydog999 View Post
It will not be recording the raw tick data live, but just store it after it has already been buffered. Losing data should be avoided but is not entirely catastrophic.

It sounds to me that you need a high performance workstation and a good NAS or an Amazon EC2 cluster and not a server. The objectives of a server are generally:

1. Very low tolerance for downtime
2. Density of compute resources vs power consumption ratio (cost efficiency of compute cycles)
3. Optional: Real-time application (for example, Apple's Siri has a backend of about 420Us+ of server hardware in each region)

It doesn't seem like you need to meet any of these objectives. If you spec out a server closet in your home just for 2 servers, then you're probably violating objective 2. If you don't have dual PDUs with separate mains/secondary and server UPSes, proper cooling, and loss of data is not catastrophic, then you're probably violating objective 1 since the source of downtime is dominated by your support infrastructure and not the servers themselves.

Reply With Quote
 
  #5 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,366 since Jun 2009
Thanks: 3,573 given, 3,982 received


artemiso View Post
...
Hm, I'd be interested to hear what made you come to that conclusion.
...

For DELL it's mainly the fact that if you buy the same server (same part number/configuration) 3 months after the first one you can have different component (network and Raid card chipset for example). So if you have tuned some home made drivers for the first servers you'll have to do the job again.
But the main point is the IPMI/remote console stuff, which were not very reliable with IBM and DELL compared to HP. If you can't have access to your remote console quickly in the middle of the night, or can't have access to it at all and have to go physically near the server to plug a physical screen/keyboard, then you really start to hate the hardware vendor .
I had some funny stories with HP, some Bios bugs who let the fans run full power all the time (extremely noisy...), but I don't remember having unable to access to the remote console, since the DL3xx G1 generation (2001?). The management tools are very good, if an hardware is failing you usually know exactly what.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
The following user says Thank You to sam028 for this post:
 
  #6 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


artemiso View Post
@treydog999

Where are you going to store your servers? I'm guessing it's much cheaper in S.K. but in the U.S., I'd set aside about $1-1.5k/month for the cabinet space and power, another $200-500/month for the cross-connects to your bandwidth provider, and another $400-4000/month for your bandwidth provider(s). Maybe cut those prices in 1/3 since you just need to host 2 servers. If you're working at some co-working office space, I suppose you can find space for much cheaper.

Based on what you've said, I highly recommend that you set aside a bit of your budget and pick up 1-2 Westmere spec'ed servers from eBay for $150-$250 each and 4-6 cheap disks with matching trays (for $20+ each) and test out your software setup (OS, RAID settings, data samples etc.) and infrastructure (cooling, power, network management etc.) before you spend $15k on each server.



Hm, I'd be interested to hear what made you come to that conclusion.

I've always felt IBM components were more robust, documented and replaceable than HP parts; I'd give Lenovo a try the next time I'm looking for a ready-built server.

Someone I know just launched a startup that required a cluster of 30+ servers in a data center deployment, not many but still enough that Dell, HP, Supermicro and Tyan sent him samples of their most highly spec'ed servers and he ultimately decided that Dell was the easiest to maintain and service. I guess your mileage will vary.




That's true. It's sometimes difficult to get the different permutations of heat sinks, PCIe riser cards, extra power cables, server case IO slots, motherboard form factors, additional fans, SAS fanout cables etc. correct. 1-2 server specs is probably manageable but it becomes a dependency hell when you reach 5-6 different kinds of specs.




It sounds to me that you need a high performance workstation and a good NAS or an Amazon EC2 cluster and not a server. The objectives of a server are generally:

1. Very low tolerance for downtime
2. Density of compute resources vs power consumption ratio (cost efficiency of compute cycles)
3. Optional: Real-time application (for example, Apple's Siri has a backend of about 420Us+ of server hardware in each region)

It doesn't seem like you need to meet any of these objectives. If you spec out a server closet in your home just for 2 servers, then you're probably violating objective 2. If you don't have dual PDUs with separate mains/secondary and server UPSes, proper cooling, and loss of data is not catastrophic, then you're probably violating objective 1 since the source of downtime is dominated by your support infrastructure and not the servers themselves.

I will be in an office building that already houses trading firms, rather than inside my house. I believe the infrastructure for this is already there. But still building out a server closest for just 2-4u of servers is still kind of a pain in the A** and expensive. If i do get anything its going to be prebuilt from a company, i do not think i want to spend that much time and resources especially if this is going to be scalable for what i may save self buidling.

Actually I was looking into some workstations last night, so I am going to continue doing research on that road. As for the EC2 cluster are you refering to that for storage purposes? But maybe you were refering to using that for computing power, I am not sure.

I am thinking that doing a NAS or file server only with a set of high end workstations might work better. I do not understand the difference fully between a NAS system and say file server. For example the file server will have low CPU requirements but lots of HDDs in raid 10.

Reply With Quote
 
  #7 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received


treydog999 View Post
I will be in an office building that already houses trading firms, rather than inside my house. I believe the infrastructure for this is already there. But still building out a server closest for just 2-4u of servers is still kind of a pain in the A** and expensive. If i do get anything its going to be prebuilt from a company, i do not think i want to spend that much time and resources especially if this is going to be scalable for what i may save self buidling.

I see, if that building has a server room or two, then you should be OK; do make sure that they have the right IT personnel to set up a private network for you.


treydog999 View Post
Actually I was looking into some workstations last night, so I am going to continue doing research on that road. As for the EC2 cluster are you refering to that for storage purposes? But maybe you were refering to using that for computing power, I am not sure.

I am thinking that doing a NAS or file server only with a set of high end workstations might work better. I do not understand the difference fully between a NAS system and say file server. For example the file server will have low CPU requirements but lots of HDDs in raid 10.

Yes, I was referring to both computing and storage purposes. EC2 refers to the cloud computing platform; EBS refers to the cloud storage platform, the two tend to be referred to with assumption that the other is involved.

A NAS appliance is a specific implementation of a "file server". It provides data access on a file-level. Check the rackmount NAS's that Synology/QNAP sell and you should get it. It basically comes with all the functionality you're looking for except more: they usually provide additional management software in a GUI or console application to make the features (e.g. RAID setup, monitoring, IEEE LACP) more accessible for you. Scaling a Rackmount NAS appliance to several hundred TB is usually as simple as a matter of plug-and-play, and these appliances often come with 4 LAN failover off-the-shelf.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
  #8 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received


sam028 View Post
For DELL it's mainly the fact that if you buy the same server (same part number/configuration) 3 months after the first one you can have different component (network and Raid card chipset for example). So if you have tuned some home made drivers for the first servers you'll have to do the job again.
But the main point is the IPMI/remote console stuff, which were not very reliable with IBM and DELL compared to HP. If you can't have access to your remote console quickly in the middle of the night, or can't have access to it at all and have to go physically near the server to plug a physical screen/keyboard, then you really start to hate the hardware vendor .
I had some funny stories with HP, some Bios bugs who let the fans run full power all the time (extremely noisy...), but I don't remember having unable to access to the remote console, since the DL3xx G1 generation (2001?). The management tools are very good, if an hardware is failing you usually know exactly what.

I see, thanks for the color.

Reply With Quote

Reply



futures io > > > > Buying / Building an Enterprise Server for Trading Research

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Research: trading pullbacks in CL Georgii Trading Journals 113 October 19th, 2014 10:38 PM
Thinking About Buying a Used Gaming PC; Need Advice on Home Server aventeren Tech Support 0 October 8th, 2014 09:06 PM
Stupidest government enterprise: $406k on coffee enemas kbit News and Current Events 0 December 27th, 2011 06:33 PM


All times are GMT -4. The time now is 02:52 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-16 in 0.19 seconds with 19 queries on phoenix via your IP 54.163.209.109