Market analysis with SQL - futures io
futures io futures trading



Market analysis with SQL


Discussion in Platforms and Indicators

Updated
      Top Posters
    1. looks_one shodson with 4 posts (11 thanks)
    2. looks_two anny with 3 posts (0 thanks)
    3. looks_3 Quick Summary with 1 posts (0 thanks)
    4. looks_4 Big Mike with 1 posts (2 thanks)
    1. trending_up 4,356 views
    2. thumb_up 15 thanks given
    3. group 3 followers
    1. forum 8 posts
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Market analysis with SQL

(login for full post details)
  #1 (permalink)
OC, California, USA
 
Experience: Advanced
Platform: IB/TWS, NinjaTrader, ToS
Broker: IB, ToS, Kinetick
Trading: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,958 since Jun 2009
Thanks: 528 given, 3,648 received

I know I should probably learn MATLIB or SPSS or some other statistical tool, but I've never learned those tools and it turns out I'm really good with SQL Server, so I'm looking at building a data mine of market data and metrics that I am interested in researching.

Currently, I am just doing this on the freely available SQL Express. If/when I need more BI capabilities and SSIS I will migrate to a full developer version of SQL Server.

So, what I've done so far is export a few years of historical data out of NT7 continuous contract data, imported them into SQL, and am now working on creating daily measurements (bars) of the market via stored procedures into fact tables.

For example, here's my script that processes the daily OHLCV data and stores various SMAs that might be of interest.



I have a lot more to do, and can provide updates of my progress, but I was just wondering if anybody else likes analyzing the markets with SQL.

Follow me on Twitter Visit my futures io Trade Journal Started this thread Reply With Quote
The following 7 users say Thank You to shodson for this post:

Journal Challenge February 2021 results (so far):
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(154 thanks from 29 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(111 thanks from 32 posts)
looks_3Wisdom is Emptinessby Mtype
(68 thanks from 25 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(31 thanks from 16 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 799 received


Interesting.

If you subscribe to the idea that a trading platform is really just a front end to a database with some market data and brokerage connectors rolled in, it gives you a new perspective.

I posted some C# console apps I'm using to help my own SQL development and understanding of using distributed SQL instances on a network here: . It is in the Elite section but I'd be happy to post the code here if you think it will help this tread develop.

Have you looked at LINQ? Once you have the data you need in SQL you can use external apps to manipulate it and report. LINQ/C#, R to SQL, Matlab to SQL, etc are all at your disposal in way not easily done with the NT database itself.

If you like free stuff R might be worth a look before Matlab.

Reply With Quote
The following 2 users say Thank You to MXASJ for this post:
 
(login for full post details)
  #4 (permalink)
OC, California, USA
 
Experience: Advanced
Platform: IB/TWS, NinjaTrader, ToS
Broker: IB, ToS, Kinetick
Trading: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,958 since Jun 2009
Thanks: 528 given, 3,648 received


MXASJ View Post
Interesting.

If you subscribe to the idea that a trading platform is really just a front end to a database with some market data and brokerage connectors rolled in, it gives you a new perspective.

I posted some C# console apps I'm using to help my own SQL development and understanding of using distributed SQL instances on a network here: . It is in the Elite section but I'd be happy to post the code here if you think it will help this tread develop.

Have you looked at LINQ? Once you have the data you need in SQL you can use external apps to manipulate it and report. LINQ/C#, R to SQL, Matlab to SQL, etc are all at your disposal in way not easily done with the NT database itself.

If you like free stuff R might be worth a look before Matlab.

I've played with LINQ but, like most other object-data mapping attempts created by Microsoft, I hit a wall that prompted me to throw it away. For object-relational mapping (ORM) I have created my own code generator that I've used for years, based on similar approaches used by Ruby on Rails and other frameworks inspired by enterprise design patterns (Fowler). Microsoft's "entity framework" looks perhaps maybe they're getting closer, but if you don't have your own ORM framework for .NET I'd recommend Subsonic.

Follow me on Twitter Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to shodson for this post:
 
(login for full post details)
  #5 (permalink)
Chandigarh India
 
Experience: Intermediate
Platform: TradeStation, Multicharts
Trading: Stocks
 
Posts: 20 since Jun 2013
Thanks: 26 given, 7 received


shodson View Post
I know I should probably learn MATLIB or SPSS or some other statistical tool, but I've never learned those tools and it turns out I'm really good with SQL Server, so I'm looking at building a data mine of market data and metrics that I am interested in researching.

Currently, I am just doing this on the freely available SQL Express. If/when I need more BI capabilities and SSIS I will migrate to a full developer version of SQL Server.

So, what I've done so far is export a few years of historical data out of NT7 continuous contract data, imported them into SQL, and am now working on creating daily measurements (bars) of the market via stored procedures into fact tables.

For example, here's my script that processes the daily OHLCV data and stores various SMAs that might be of interest.



I have a lot more to do, and can provide updates of my progress, but I was just wondering if anybody else likes analyzing the markets with SQL.

@shodson thanks a lot for this thread. I know it is a very old thread, but could you please post an update. Did you make any progress in doing analysis with SQL ? Or you found it to be very limiting and therefor dumped it midway ?

I am also interested in doing some analysis within SQL Server for the EOD database that I have. I think that a lot of work could be done within such databases, which would be difficult to do in the normal platforms like ninja etc. Since you did this years ago, therefor I am very interested in knowing about your overall practical experience in this regards.

Thanks and regards

Reply With Quote
 
(login for full post details)
  #6 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,744 since Jun 2009
Thanks: 32,292 given, 97,488 received

Be sure to also check:



And



I use MySQL extensively in my own platform (second thread).

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Big Mike for this post:
 
(login for full post details)
  #7 (permalink)
Chandigarh India
 
Experience: Intermediate
Platform: TradeStation, Multicharts
Trading: Stocks
 
Posts: 20 since Jun 2013
Thanks: 26 given, 7 received


Big Mike View Post
Be sure to also check:



And



I use MySQL extensively in my own platform (second thread).

Mike

Thanks Mike, I will check them out.

Reply With Quote
 
(login for full post details)
  #8 (permalink)
OC, California, USA
 
Experience: Advanced
Platform: IB/TWS, NinjaTrader, ToS
Broker: IB, ToS, Kinetick
Trading: stocks, options, futures, VIX
 
shodson's Avatar
 
Posts: 1,958 since Jun 2009
Thanks: 528 given, 3,648 received


anny View Post
@shodson thanks a lot for this thread. I know it is a very old thread, but could you please post an update. Did you make any progress in doing analysis with SQL ? Or you found it to be very limiting and therefor dumped it midway ?

I am also interested in doing some analysis within SQL Server for the EOD database that I have. I think that a lot of work could be done within such databases, which would be difficult to do in the normal platforms like ninja etc. Since you did this years ago, therefor I am very interested in knowing about your overall practical experience in this regards.

Thanks and regards

Yes I made progress and it's a core part of my trading today. Instead of finding it limiting, I find it very expansive and additive to my trading toolbelt instead of just relying on vendor-provided software. If you have the skills, time, and passion for statistics and quantitative methodologies I highly recommend something along these lines.

Follow me on Twitter Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to shodson for this post:
 
(login for full post details)
  #9 (permalink)
Chandigarh India
 
Experience: Intermediate
Platform: TradeStation, Multicharts
Trading: Stocks
 
Posts: 20 since Jun 2013
Thanks: 26 given, 7 received


shodson View Post
Yes I made progress and it's a core part of my trading today. Instead of finding it limiting, I find it very expansive and additive to my trading toolbelt instead of just relying on vendor-provided software. If you have the skills, time, and passion for statistics and quantitative methodologies I highly recommend something along these lines.

Thanks for the update shodson.

That means that SQL Server has got good potential in this direction. I will definitely spend more time on this.

Regards

Reply With Quote


futures io Trading Community Platforms and Indicators > Market analysis with SQL


Last Updated on March 27, 2015


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1500 prizes from Topstep!

February
 

Identifying Setups & Targets Using Profile Charts w/Trevor & Tradovate

Feb 25
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts