NexusFi: Find Your Edge


Home Menu

 





help with basic programming: database, excel , web data collection


Discussion in Traders Hideout

Updated
      Top Posters
    1. looks_one GridKing with 6 posts (0 thanks)
    2. looks_two furytrader with 4 posts (3 thanks)
    3. looks_3 bukkan with 3 posts (3 thanks)
    4. looks_4 Quick Summary with 1 posts (0 thanks)
    1. trending_up 8,110 views
    2. thumb_up 6 thanks given
    3. group 2 followers
    1. forum 14 posts
    2. attach_file 0 attachments




 
Search this Thread

help with basic programming: database, excel , web data collection

  #1 (permalink)
 
GridKing's Avatar
 GridKing 
San Diego, CA USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: ES
Posts: 3,630 since May 2011
Thanks Given: 584
Thanks Received: 881

If someone can please point me in the right direction on some basic programming and data collection

my skills are non existent but I can figure it out if I know what I am looking for... I want to build a database that will get data from web pages daily and store it, also download files and store them and tie it all together so I can spit out a daily summary

I know that in excel I can use "data from web" but many of the pages I use it doesn't work, so should i just load the whole page into excel and have it delete the stuff I don't need or is there a better way ? Here is an example of a page I would like to get the table data in the table and download the csv file in the link at the end of each line http://www.patterntrapper.com/!Data_Futures.shtml

I have Access , Excel , Visual Studio not sure what is the most efficient and easiest to setup and begin to build a historical database that will become automated without spending years to learn how to do it? Thanks a bunch

Visit my NexusFi Trade Journal Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
NexusFi Journal Challenge - April 2024
Feedback and Announcements
Futures True Range Report
The Elite Circle
Exit Strategy
NinjaTrader
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
 
  #3 (permalink)
 
furytrader's Avatar
 furytrader 
Lake Forest, IL USA
 
Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker: Advantage Futures, IQFeed.net
Trading: YM, ES, EU, US, S
Posts: 153 since Jun 2011
Thanks Given: 109
Thanks Received: 147


I would probably use Visual Studio and write a simple application that first retrieves all of the .csv files and then parses the data that you want.

VB.Net, for example, has a lot of built-in routines for retrieving files off the web and parsing text. The nice thing about that particular webpage that you referenced in your post is that the .csv files have the same name everyday. You can just create a list of their addresses and have the program cycle through, retrieve each file at a designated time of day and then parse each one.

You indicated in your post that you want to automate the collection of this data everyday. Do you simply want to create a databse with this data or do you want to run some additional analysis on it?

Reply With Quote
  #4 (permalink)
 
GridKing's Avatar
 GridKing 
San Diego, CA USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: ES
Posts: 3,630 since May 2011
Thanks Given: 584
Thanks Received: 881


furytrader View Post
I would probably use Visual Studio and write a simple application that first retrieves all of the .csv files and then parses the data that you want.

VB.Net, for example, has a lot of built-in routines for retrieving files off the web and parsing text. The nice thing about that particular webpage that you referenced in your post is that the .csv files have the same name everyday. You can just create a list of their addresses and have the program cycle through, retrieve each file at a designated time of day and then parse each one.

You indicated in your post that you want to automate the collection of this data everyday. Do you simply want to create a databse with this data or do you want to run some additional analysis on it?

Thanks, and yes to be able to run additional analysis and collect data from other places and have it all together in one place for analysis, i guess , thanks again

Visit my NexusFi Trade Journal Started this thread Reply With Quote
  #5 (permalink)
 
furytrader's Avatar
 furytrader 
Lake Forest, IL USA
 
Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker: Advantage Futures, IQFeed.net
Trading: YM, ES, EU, US, S
Posts: 153 since Jun 2011
Thanks Given: 109
Thanks Received: 147

What kind of other data would you look to be pulling down? Is that data as user-friendly as the .csv files you referenced? Also, what kind of output do you want? Charts? Tables? Where things get a bit laborious is when you have to write individual routines to scrape information off of particular websites. Also, for example, if you wanted to retrieve historical data and then chart it -- well, there are lots of programs already available that do charting well, so no point in reinventing the wheel.

Reply With Quote
  #6 (permalink)
 
GridKing's Avatar
 GridKing 
San Diego, CA USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: ES
Posts: 3,630 since May 2011
Thanks Given: 584
Thanks Received: 881


furytrader View Post
What kind of other data would you look to be pulling down? Is that data as user-friendly as the .csv files you referenced? Also, what kind of output do you want? Charts? Tables? Where things get a bit laborious is when you have to write individual routines to scrape information off of particular websites. Also, for example, if you wanted to retrieve historical data and then chart it -- well, there are lots of programs already available that do charting well, so no point in reinventing the wheel.

most of the other stuff will work with excel basic web query , so what to use for actual database ? access or visual studio?

The data analysis will just be to get an overview of the market as a whole on one page , doesn't have to be with charts just perhaps visually pleasing

ideally I guess I would like to have it presented within it's parameters meaning for example last price in the context of pivots moving averages, and other levels

sort of like this but not exactly ESY00 | Futures Trader's Cheat Sheet for E-Mini S&P 500 Cash

Thanks again

Visit my NexusFi Trade Journal Started this thread Reply With Quote
  #7 (permalink)
 
furytrader's Avatar
 furytrader 
Lake Forest, IL USA
 
Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker: Advantage Futures, IQFeed.net
Trading: YM, ES, EU, US, S
Posts: 153 since Jun 2011
Thanks Given: 109
Thanks Received: 147

With Visual Studio, you can easily connect to an Access database (there's a lot of pre-written code for doing that) or you can use an SQL-Lite database (I believe Visual Studio ships with SQL-Lite). If you already own the Microsoft Access program, it may make setting up the database a bit easier.

What I would recommend is writing out a list of all of the statistics you want to generate, and for which markets. You could also do a mock-up of how you want it to look. This information in turn will drive (a) where you get the data, (b) how much data you need to save and (c) what kind of formulas you need to be able to write.

Reply With Quote
Thanked by:
  #8 (permalink)
 
GridKing's Avatar
 GridKing 
San Diego, CA USA
 
Experience: Intermediate
Platform: NinjaTrader
Trading: ES
Posts: 3,630 since May 2011
Thanks Given: 584
Thanks Received: 881

there are so many languages and programs and just want to learn it and get it done with some flexibility , thanks for pointing me in the right direction , much appreciated

Visit my NexusFi Trade Journal Started this thread Reply With Quote
  #9 (permalink)
 
furytrader's Avatar
 furytrader 
Lake Forest, IL USA
 
Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker: Advantage Futures, IQFeed.net
Trading: YM, ES, EU, US, S
Posts: 153 since Jun 2011
Thanks Given: 109
Thanks Received: 147

Especially in relation to VB.net (which is one language I know quite well), there are LOTS of code examples available for free on the Internet.

So, for example, if you want to see how to use VB.net to retrieve a file off the Internet, just look here:

How to download file from the internet to my computer (VB.net Windows aplication)

Reply With Quote
Thanked by:
  #10 (permalink)
 bukkan 
Calcutta, India
 
Experience: Intermediate
Platform: ArthaChitra
Posts: 278 since Jun 2009
Thanks Given: 161
Thanks Received: 271


i agree with furytrader. go for visual studios. since you are using NinjaTrader, go for C# to be precise.

for this job, google httpwebrequest and regex. that will enable you to scrap the data to csv files. if you want something in excel then do explore Gummy Gone my tryst with programming began with the excel macro from there only.

Reply With Quote
Thanked by:




Last Updated on July 7, 2011


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts