How many usable GB/TB will you need for your storage server? How many IOPS is required to crunch your data? Can you afford to lose some data if you lose a disk?
That is required to answer how many axes are required, what kind of disks (sata/sas/ssd) and the raid card type (cache, bbu, ...). It's possible to mix the disk types, with cheap and big sata disk to backup sas/ssd disks which contains the real data and could be with redundancy (Raid 0).
I usually avoir Raid 5, Raid 50 is a better option, but I only use Raid 10 myself.

Is the number cruncher will be highly multithreaded and scalable? As your budget is large enough an high end Intel E5-26xx will do the job (E5-2660 V2 and higher). Avoid AMD processors, more cores/socket but so slow...

I would recommend to not build it from individual parts. It's a bit cheaper but you may have a compatibility issues and it's at the end a waste of time (so a waste of $$).
In buying a Dell/IBM/HP you'll get less troubles and are sure that everything is working well together. You'll also be sure to find replacement parts years later.
I've tested and worked with Dell/IBM/HP and few others like Fujitsu/Sun/..., HP 1U/2U servers are the best.

I've tested and worked with Dell/IBM/HP and few others like Fujitsu/Sun/..., HP 1U/2U servers are the best.

I would recommend to not build it from individual parts. It's a bit cheaper but you may have a compatibility issues and it's at the end a waste of time (so a waste of $$). In buying a Dell/IBM/HP you'll get less troubles and are sure that everything is working well together. You'll also be sure to find replacement parts years later.

It will not be recording the raw tick data live, but just store it after it has already been buffered. Losing data should be avoided but is not entirely catastrophic.

...
Hm, I'd be interested to hear what made you come to that conclusion.
...

@treydog999

Where are you going to store your servers? I'm guessing it's much cheaper in S.K. but in the U.S., I'd set aside about $1-1.5k/month for the cabinet space and power, another $200-500/month for the cross-connects to your bandwidth provider, and another $400-4000/month for your bandwidth provider(s). Maybe cut those prices in 1/3 since you just need to host 2 servers. If you're working at some co-working office space, I suppose you can find space for much cheaper.

Based on what you've said, I highly recommend that you set aside a bit of your budget and pick up 1-2 Westmere spec'ed servers from eBay for $150-$250 each and 4-6 cheap disks with matching trays (for $20+ each) and test out your software setup (OS, RAID settings, data samples etc.) and infrastructure (cooling, power, network management etc.) before you spend $15k on each server.



Hm, I'd be interested to hear what made you come to that conclusion.

I've always felt IBM components were more robust, documented and replaceable than HP parts; I'd give Lenovo a try the next time I'm looking for a ready-built server.

Someone I know just launched a startup that required a cluster of 30+ servers in a data center deployment, not many but still enough that Dell, HP, Supermicro and Tyan sent him samples of their most highly spec'ed servers and he ultimately decided that Dell was the easiest to maintain and service. I guess your mileage will vary.




That's true. It's sometimes difficult to get the different permutations of heat sinks, PCIe riser cards, extra power cables, server case IO slots, motherboard form factors, additional fans, SAS fanout cables etc. correct. 1-2 server specs is probably manageable but it becomes a dependency hell when you reach 5-6 different kinds of specs.




It sounds to me that you need a high performance workstation and a good NAS or an Amazon EC2 cluster and not a server. The objectives of a server are generally:

1. Very low tolerance for downtime
2. Density of compute resources vs power consumption ratio (cost efficiency of compute cycles)
3. Optional: Real-time application (for example, Apple's Siri has a backend of about 420Us+ of server hardware in each region)

It doesn't seem like you need to meet any of these objectives. If you spec out a server closet in your home just for 2 servers, then you're probably violating objective 2. If you don't have dual PDUs with separate mains/secondary and server UPSes, proper cooling, and loss of data is not catastrophic, then you're probably violating objective 1 since the source of downtime is dominated by your support infrastructure and not the servers themselves.

I will be in an office building that already houses trading firms, rather than inside my house. I believe the infrastructure for this is already there. But still building out a server closest for just 2-4u of servers is still kind of a pain in the A** and expensive. If i do get anything its going to be prebuilt from a company, i do not think i want to spend that much time and resources especially if this is going to be scalable for what i may save self buidling.

Actually I was looking into some workstations last night, so I am going to continue doing research on that road. As for the EC2 cluster are you refering to that for storage purposes? But maybe you were refering to using that for computing power, I am not sure.

I am thinking that doing a NAS or file server only with a set of high end workstations might work better. I do not understand the difference fully between a NAS system and say file server. For example the file server will have low CPU requirements but lots of HDDs in raid 10.

For DELL it's mainly the fact that if you buy the same server (same part number/configuration) 3 months after the first one you can have different component (network and Raid card chipset for example). So if you have tuned some home made drivers for the first servers you'll have to do the job again.
But the main point is the IPMI/remote console stuff, which were not very reliable with IBM and DELL compared to HP. If you can't have access to your remote console quickly in the middle of the night, or can't have access to it at all and have to go physically near the server to plug a physical screen/keyboard, then you really start to hate the hardware vendor .
I had some funny stories with HP, some Bios bugs who let the fans run full power all the time (extremely noisy...), but I don't remember having unable to access to the remote console, since the DL3xx G1 generation (2001?). The management tools are very good, if an hardware is failing you usually know exactly what.