Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Institutional Grade Massive Data Dump for Quant Research Analaysis Work
You guys are right. That original tweet has got deleted, because the data was too precious and someone might have complained regarding it being shared openly free of cost.
I really hope, a few members here might have been able to download this for sure.
Although I myself failed to download it, because of the HUGE Size of the dataset. It was worth around 3 TB+ in total. So I was doing some searching regarding which particular cloud data storage etc. might be best for downloading and keeping such huge data for future use. I was not aware that the link would go down within a few days itself. If someone here has downloaded, then please do send me a private message.
And if such an opportunity becomes available in the future again, then I want to keep a cloud storage account prepared beforehand.
I am looking for an ideal Cloud Storage Solution which could be used for storing and sharing such huge amounts of Stock Market Data in the most efficient and cost effective manner.
The basic requirements for the cloud storage are as follows -
It should ideally have unlimited cloud space or at least a few TB otherwise.
No charge for the BANDWIDTH, so that I can upload and download without worries, as otherwise the normal vendors like Amazon AWS S3 Buckets etc charge extra money for the bandwidth.
Good data transfer speeds for uploads as well as downloads.
I can even share my cloud account details with some members here, so that we can use it for storing the stock market data backups, economically.
I intend to use this cloud storage for storing all the data that I have and I will keep on adding more data for each new trading session, going forward.
The normal Google Drive Account does not really suit this because I have noticed that if the number of files are less, "even if they are of Big Size of few GB per file", then Google Drive works efficiently. But if you use it for transferring huge numbers of files, like thousands of CSV data files "even if they are of Small Size of few KB or MB per file", then its performance goes down drastically and it becomes very slow.
Could someone here please help and provide some guidance in this regards?
I took a look at it and honestly the only dataset on there that I felt was valuable (and most likely a contractual violation to distribute) was the Ravenpack data, which was about 60~ GB. Others were kind of junk to keep around unless you could continue forward filling it, had quality assurance from the original source, and had means to interpret and/or manipulate the data model from the original source.
That said, the cheapest cloud providers I could think of after Google/Dropbox/Box free tier would be Cloudflare/Wasabi/Backblaze.