help with basic programming: database, excel , web data collection - Traders Hideout | futures io social day trading
futures io futures trading


help with basic programming: database, excel , web data collection
Updated: Views / Replies:2,709 / 14
Created: by GridKing Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

help with basic programming: database, excel , web data collection

  #1 (permalink)
Elite Member
San Diego, CA USA
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: ES
 
GridKing's Avatar
 
Posts: 3,650 since May 2011
Thanks: 588 given, 877 received

help with basic programming: database, excel , web data collection

If someone can please point me in the right direction on some basic programming and data collection

my skills are non existent but I can figure it out if I know what I am looking for... I want to build a database that will get data from web pages daily and store it, also download files and store them and tie it all together so I can spit out a daily summary

I know that in excel I can use "data from web" but many of the pages I use it doesn't work, so should i just load the whole page into excel and have it delete the stuff I don't need or is there a better way ? Here is an example of a page I would like to get the table data in the table and download the csv file in the link at the end of each line http://www.patterntrapper.com/!Data_Futures.shtml

I have Access , Excel , Visual Studio not sure what is the most efficient and easiest to setup and begin to build a historical database that will become automated without spending years to learn how to do it? Thanks a bunch

Reply With Quote
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
Chicago, IL USA
 
Futures Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker/Data: Advantage Futures, IQFeed.net
Favorite Futures: YM, ES, EU, US, S
 
furytrader's Avatar
 
Posts: 153 since Jun 2011
Thanks: 109 given, 144 received

Programming Project

Answer
This post has been selected as an answer to the original posters question Answer


I would probably use Visual Studio and write a simple application that first retrieves all of the .csv files and then parses the data that you want.

VB.Net, for example, has a lot of built-in routines for retrieving files off the web and parsing text. The nice thing about that particular webpage that you referenced in your post is that the .csv files have the same name everyday. You can just create a list of their addresses and have the program cycle through, retrieve each file at a designated time of day and then parse each one.

You indicated in your post that you want to automate the collection of this data everyday. Do you simply want to create a databse with this data or do you want to run some additional analysis on it?

Reply With Quote
 
  #4 (permalink)
Elite Member
San Diego, CA USA
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: ES
 
GridKing's Avatar
 
Posts: 3,650 since May 2011
Thanks: 588 given, 877 received

Thank You! and yes


furytrader View Post
I would probably use Visual Studio and write a simple application that first retrieves all of the .csv files and then parses the data that you want.

VB.Net, for example, has a lot of built-in routines for retrieving files off the web and parsing text. The nice thing about that particular webpage that you referenced in your post is that the .csv files have the same name everyday. You can just create a list of their addresses and have the program cycle through, retrieve each file at a designated time of day and then parse each one.

You indicated in your post that you want to automate the collection of this data everyday. Do you simply want to create a databse with this data or do you want to run some additional analysis on it?

Thanks, and yes to be able to run additional analysis and collect data from other places and have it all together in one place for analysis, i guess , thanks again

Reply With Quote
 
  #5 (permalink)
Elite Member
Chicago, IL USA
 
Futures Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker/Data: Advantage Futures, IQFeed.net
Favorite Futures: YM, ES, EU, US, S
 
furytrader's Avatar
 
Posts: 153 since Jun 2011
Thanks: 109 given, 144 received

What kind ...

What kind of other data would you look to be pulling down? Is that data as user-friendly as the .csv files you referenced? Also, what kind of output do you want? Charts? Tables? Where things get a bit laborious is when you have to write individual routines to scrape information off of particular websites. Also, for example, if you wanted to retrieve historical data and then chart it -- well, there are lots of programs already available that do charting well, so no point in reinventing the wheel.

Reply With Quote
 
  #6 (permalink)
Elite Member
San Diego, CA USA
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: ES
 
GridKing's Avatar
 
Posts: 3,650 since May 2011
Thanks: 588 given, 877 received

yeah , same just data , text and numbers


furytrader View Post
What kind of other data would you look to be pulling down? Is that data as user-friendly as the .csv files you referenced? Also, what kind of output do you want? Charts? Tables? Where things get a bit laborious is when you have to write individual routines to scrape information off of particular websites. Also, for example, if you wanted to retrieve historical data and then chart it -- well, there are lots of programs already available that do charting well, so no point in reinventing the wheel.

most of the other stuff will work with excel basic web query , so what to use for actual database ? access or visual studio?

The data analysis will just be to get an overview of the market as a whole on one page , doesn't have to be with charts just perhaps visually pleasing

ideally I guess I would like to have it presented within it's parameters meaning for example last price in the context of pivots moving averages, and other levels

sort of like this but not exactly ESY00 | Futures Trader's Cheat Sheet for E-Mini S&P 500 Cash

Thanks again

Reply With Quote
 
  #7 (permalink)
Elite Member
Chicago, IL USA
 
Futures Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker/Data: Advantage Futures, IQFeed.net
Favorite Futures: YM, ES, EU, US, S
 
furytrader's Avatar
 
Posts: 153 since Jun 2011
Thanks: 109 given, 144 received

Database

With Visual Studio, you can easily connect to an Access database (there's a lot of pre-written code for doing that) or you can use an SQL-Lite database (I believe Visual Studio ships with SQL-Lite). If you already own the Microsoft Access program, it may make setting up the database a bit easier.

What I would recommend is writing out a list of all of the statistics you want to generate, and for which markets. You could also do a mock-up of how you want it to look. This information in turn will drive (a) where you get the data, (b) how much data you need to save and (c) what kind of formulas you need to be able to write.

Reply With Quote
The following user says Thank You to furytrader for this post:
 
  #8 (permalink)
Elite Member
San Diego, CA USA
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: ES
 
GridKing's Avatar
 
Posts: 3,650 since May 2011
Thanks: 588 given, 877 received

Thanks a bunch

there are so many languages and programs and just want to learn it and get it done with some flexibility , thanks for pointing me in the right direction , much appreciated

Reply With Quote
 
  #9 (permalink)
Elite Member
Chicago, IL USA
 
Futures Experience: Intermediate
Platform: MultiCharts + CTS T4
Broker/Data: Advantage Futures, IQFeed.net
Favorite Futures: YM, ES, EU, US, S
 
furytrader's Avatar
 
Posts: 153 since Jun 2011
Thanks: 109 given, 144 received

One last tip ...

Especially in relation to VB.net (which is one language I know quite well), there are LOTS of code examples available for free on the Internet.

So, for example, if you want to see how to use VB.net to retrieve a file off the Internet, just look here:

How to download file from the internet to my computer (VB.net Windows aplication)

Reply With Quote
The following 2 users say Thank You to furytrader for this post:
 
  #10 (permalink)
Elite Member
Calcutta, India
 
Futures Experience: Intermediate
Platform: ArthaChitra
 
Posts: 278 since Jun 2009
Thanks: 161 given, 259 received

Answer
This post has been selected as an answer to the original posters question Answer


i agree with furytrader. go for visual studios. since you are using NinjaTrader, go for C# to be precise.

for this job, google httpwebrequest and regex. that will enable you to scrap the data to csv files. if you want something in excel then do explore Gummy Gone my tryst with programming began with the excel macro from there only.

Reply With Quote
The following 2 users say Thank You to bukkan for this post:

Reply



futures io > > > help with basic programming: database, excel , web data collection

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Screenshot tools collection max-td Beginners and Introductions 26 November 4th, 2012 02:41 AM
MT4 - Excel streaming data capture Madtrader MetaTrader 2 August 7th, 2011 11:26 AM
PLT_NTDataCollect store bid ask tick data to its local database? rcabri Trading Reviews and Vendors 3 December 27th, 2010 11:00 AM
Exporting Volume Profile data from NT6.5 to Excel. How is it jungian Beginners and Introductions 4 May 20th, 2010 11:07 AM
OHLC Data for downloading to Excel.csv zeller4 NinjaTrader Programming 15 December 6th, 2009 07:11 PM


All times are GMT -4. The time now is 02:55 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-12 in 0.14 seconds with 37 queries on phoenix via your IP 54.90.207.75