NexusFi: Find Your Edge


Home Menu

 





Institutional Grade Massive Data Dump for Quant Research Analaysis Work


Discussion in Traders Hideout

Updated
      Top Posters
    1. looks_one ab456 with 4 posts (9 thanks)
    2. looks_two artemiso with 1 posts (2 thanks)
    3. looks_3 xplorer with 1 posts (0 thanks)
    4. looks_4 SMCJB with 1 posts (0 thanks)
    1. trending_up 1,655 views
    2. thumb_up 11 thanks given
    3. group 5 followers
    1. forum 6 posts
    2. attach_file 1 attachments




 
Search this Thread

Institutional Grade Massive Data Dump for Quant Research Analaysis Work

  #1 (permalink)
 ab456 
New Delhi, India
 
Experience: Intermediate
Platform: SierraChart NinjaTrader,
Trading: ES,Stocks, Futures
Posts: 170 since Sep 2011
Thanks Given: 734
Thanks Received: 111



All credit goes to https://twitter.com/quant_arb for sharing this.

According to him, it contains decades of data, which was available only to the Hedge Funds.


You may download what looks useful to you.
The complete data set might be many TB in size.

Best Regards

Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Deepmoney LLM
Elite Quantitative GenAI/LLM
ZombieSqueeze
Platforms and Indicators
Ninja Mobile Trader VPS (ninjamobiletrader.com)
Trading Reviews and Vendors
Futures True Range Report
The Elite Circle
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
59 thanks
Funded Trader platforms
37 thanks
GFIs1 1 DAX trade per day journal
22 thanks
NexusFi site changelog and issues/problem reporting
22 thanks
The Program
20 thanks
  #2 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

Maybe I'm missing it, but download from where? You linked to a Twitter profile not a specific post/tweet.

Reply With Quote
  #3 (permalink)
 
xplorer's Avatar
 xplorer 
London UK
Site Moderator
 
Experience: Beginner
Platform: CQG
Broker: S5
Trading: Futures
Posts: 5,944 since Sep 2015
Thanks Given: 15,447
Thanks Received: 15,291



SMCJB View Post
Maybe I'm missing it, but download from where? You linked to a Twitter profile not a specific post/tweet.

You may have same issue as me, in that I don't see embedded tweets on FIO


If you quote the original post you will see the link OP is talking about.



EDIT: It looks like that specific tweet has been deleted anyway.

Reply With Quote
  #4 (permalink)
 ab456 
New Delhi, India
 
Experience: Intermediate
Platform: SierraChart NinjaTrader,
Trading: ES,Stocks, Futures
Posts: 170 since Sep 2011
Thanks Given: 734
Thanks Received: 111


SMCJB View Post
Maybe I'm missing it, but download from where? You linked to a Twitter profile not a specific post/tweet.


xplorer View Post
You may have same issue as me, in that I don't see embedded tweets on FIO


If you quote the original post you will see the link OP is talking about.



EDIT: It looks like that specific tweet has been deleted anyway.


You guys are right. That original tweet has got deleted, because the data was too precious and someone might have complained regarding it being shared openly free of cost.

I really hope, a few members here might have been able to download this for sure.

Although I myself failed to download it, because of the HUGE Size of the dataset. It was worth around 3 TB+ in total. So I was doing some searching regarding which particular cloud data storage etc. might be best for downloading and keeping such huge data for future use. I was not aware that the link would go down within a few days itself. If someone here has downloaded, then please do send me a private message.

And if such an opportunity becomes available in the future again, then I want to keep a cloud storage account prepared beforehand.



I am looking for an ideal Cloud Storage Solution which could be used for storing and sharing such huge amounts of Stock Market Data in the most efficient and cost effective manner.

The basic requirements for the cloud storage are as follows -

It should ideally have unlimited cloud space or at least a few TB otherwise.

No charge for the BANDWIDTH, so that I can upload and download without worries, as otherwise the normal vendors like Amazon AWS S3 Buckets etc charge extra money for the bandwidth.

Good data transfer speeds for uploads as well as downloads.


I can even share my cloud account details with some members here, so that we can use it for storing the stock market data backups, economically.

I intend to use this cloud storage for storing all the data that I have and I will keep on adding more data for each new trading session, going forward.

The normal Google Drive Account does not really suit this because I have noticed that if the number of files are less, "even if they are of Big Size of few GB per file", then Google Drive works efficiently. But if you use it for transferring huge numbers of files, like thousands of CSV data files "even if they are of Small Size of few KB or MB per file", then its performance goes down drastically and it becomes very slow.

Could someone here please help and provide some guidance in this regards?

Any ideas are welcome.

Thanks a lot.

Best Regards

Started this thread Reply With Quote
Thanked by:
  #5 (permalink)
 artemiso 
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
Posts: 1,152 since Jul 2012
Thanks Given: 784
Thanks Received: 2,685

I took a look at it and honestly the only dataset on there that I felt was valuable (and most likely a contractual violation to distribute) was the Ravenpack data, which was about 60~ GB. Others were kind of junk to keep around unless you could continue forward filling it, had quality assurance from the original source, and had means to interpret and/or manipulate the data model from the original source.

That said, the cheapest cloud providers I could think of after Google/Dropbox/Box free tier would be Cloudflare/Wasabi/Backblaze.

Reply With Quote
Thanked by:
  #6 (permalink)
 ab456 
New Delhi, India
 
Experience: Intermediate
Platform: SierraChart NinjaTrader,
Trading: ES,Stocks, Futures
Posts: 170 since Sep 2011
Thanks Given: 734
Thanks Received: 111


artemiso View Post
I took a look at it and honestly the only dataset on there that I felt was valuable (and most likely a contractual violation to distribute) was the Ravenpack data, which was about 60~ GB. Others were kind of junk to keep around unless you could continue forward filling it, had quality assurance from the original source, and had means to interpret and/or manipulate the data model from the original source.

That said, the cheapest cloud providers I could think of after Google/Dropbox/Box free tier would be Cloudflare/Wasabi/Backblaze.

Thanks for the suggestions @artemiso

btw, if anyone is curious, I found the snapshot of the Original Post of this thread, which shows the actual tweet, which has got delete now.


Snapshot



Regards

Started this thread Reply With Quote
Thanked by:
  #7 (permalink)
 ab456 
New Delhi, India
 
Experience: Intermediate
Platform: SierraChart NinjaTrader,
Trading: ES,Stocks, Futures
Posts: 170 since Sep 2011
Thanks Given: 734
Thanks Received: 111

After doing further analysis, I found that this was the original download link posted in the tweet -
https://t.co/Z7euOfPgb0
https://b.scsi.to/

And it is showing the following message now -

"After serving almost 100 TB of downloads, I received a DMCA notice.
Data will be offline until further notice.

"

Seems like many people were able to download the complete datasets from that link.

Started this thread Reply With Quote
Thanked by:




Last Updated on February 7, 2023


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts