forexfactory calendar downloader - Matlab, R project and Python | futures io social day trading
futures io futures trading


forexfactory calendar downloader
Updated: Views / Replies:6,080 / 9
Created: by enjoyaol Attachments:3

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

forexfactory calendar downloader

  #1 (permalink)
Elite Member
Paris, France
 
Futures Experience: Intermediate
Platform: MT4, Amibroker, Custom
Favorite Futures: EUR/USD
 
Posts: 44 since Jan 2012
Thanks: 5 given, 24 received

forexfactory calendar downloader

hi,

Here is my forex factory calendar downloader. It creates a CSV file containing historical events from forexfactory.
It's in python and uses lxml, it's a good start for those who never made web scrapping before. Code is quite clean, but hasn't any real error management yet.

Also, it creates a 'raw' CSV view of what is available on the website. It's not filling out @NA data, doesn't try to be smart about the data. I intend to add some 'smart' behaviour during the import inside the SQL database.

Have fun.

https://www.dropbox.com/s/mmcjejumucq1mli/ff.py

 
Code
from __future__ import unicode_literals
import codecs
import pprint
import lxml.html
import mechanize
import cookielib

#some utils
pp = pprint.PrettyPrinter()


#########################
#variables
#########################
START_YEAR = 2008
END_YEAR = 2013
URL = r"http://www.forexfactory.com/calendar.php?month="
OUTFILE = r"events.csv"
#########################


#our month list for the URL
monthslist = ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]

#sets up the browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

#set correct timezone
br.open("http://www.forexfactory.com/timezone.php")
formindex = 0
for form in br.forms():
    if "timezone.php" in form.action:
        form["timezoneoffset"] = ["0"]
        break
    formindex += 1

br.select_form(nr=formindex)
br.submit()


def getData(html, outfile):
    """
    Gets data from one page of events
    """
    root = lxml.html.fromstring(html)
    lines = root.find_class("calendar_row")
    curWeekDay = None
    curMonthDay = None
    for event in lines:
        date = event.xpath("td[@class='date']")[0]

        #get the day of the month
        weekDay = date.xpath("span")
        monthDay = date.xpath("span/span")
        if len(weekDay) > 0:
            curWeekDay = weekDay[0].text
            curMonthDay = monthDay[0].text

        #get the time
        time = event.xpath("td[@class='time']")[0].text if (len(event.xpath("td[@class='time']")) > 0) else ""

        #get currency
        currency = event.xpath("td[@class='currency']")[0].text if len(event.xpath("td[@class='currency']")) else ""

        #get impact
        impact = event.xpath("td[@class='impact']/span/@title")[0]\
            if len(event.xpath("td[@class='impact']/span/@title")) else ""

        #get name of event
        nevent = event.xpath("td[@class='event']/span")[0].text if len(event.xpath("td[@class='event']/span")) > 0 else ""

        #get actual
        actual = event.xpath("td[@class='actual']")[0].text if len(event.xpath("td[@class='actual']")) else ""
        #retry if actual is in a span (can happen if they colorize it)
        if actual is None or len(actual.strip()) == 0:
            actual = event.xpath("td[@class='actual']/span")[0].text if len(event.xpath("td[@class='actual']/span")) else ""
        actual = actual.strip().replace("\n", " ") if actual is not None else ""

        #get forecast
        forecast = event.xpath("td[@class='forecast']")[0].text if len(event.xpath("td[@class='forecast']")) else ""
        #retry if forecast is in a span (can happen if they colorize it)
        if forecast is None or len(forecast.strip()) == 0:
            forecast = event.xpath("td[@class='forecast']/span")[0].text if len(event.xpath("td[@class='forecast']/span")) else ""
        forecast = forecast.strip().replace("\n", " ") if forecast is not None else ""

        #get previous
        previous = event.xpath("td[@class='previous']")[0].text if len(event.xpath("td[@class='previous']")) else ""
        #retry if previous is in a span (can happen if they colorize it)
        if previous is None or len(previous.strip()) == 0:
            previous = event.xpath("td[@class='previous']/span")[0].text if len(event.xpath("td[@class='previous']/span")) else ""

        previous = previous.strip().replace("\n", " ") if previous is not None else ""

        outfile.write("{};{};{};{};{};{};{};{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))


year = START_YEAR
outfile = open(OUTFILE, "w")
while year <= END_YEAR:
    for month in monthslist:
        url = "{}{}.{}".format(URL, month, year)
        print("Getting {} {} from {}".format(month, year, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
    year += 1
outfile.close()

Reply With Quote
The following 10 users say Thank You to enjoyaol for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Elite Member
Cary, NC
 
Futures Experience: Advanced
Platform: MC, NT, Python, R
Broker/Data: FXCM, IB, Oanda, IQFeed
Favorite Futures: 6E, CL, EUR/USD
 
Fu510n's Avatar
 
Posts: 94 since Oct 2009
Thanks: 726 given, 77 received

Updated


I tried running the original code but it appears that Forex Factory changed the HTML output that broke the Python XPath parsing so I tweaked the code (below) and it seems to be working "better" now. The timezone adjustment logic seemed to be skewing the times incorrectly so I simply disabled that for now.

Further tweaking may be required but thought I'd pass mine along,
-Guy

 
Code
#!/usr/bin/env python

from __future__ import unicode_literals
import sys
# import codecs
import pprint
import lxml.html
import mechanize
import cookielib

# some utils
pp = pprint.PrettyPrinter()
debug = 0


#########################
# variables
#########################
START_YEAR = 2015
END_YEAR = 2016
URL = r"http://www.forexfactory.com/calendar.php?month="
OUTFILE = r"events.csv"
#########################


# our month list for the URL
monthslist = ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]

# sets up the browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# set correct timezone
br.open("http://www.forexfactory.com/timezone.php")
formindex = 0
for form in br.forms():
    if "timezone.php" in form.action:
        form["timezoneoffset"] = ["0"]
        break
    formindex += 1

br.select_form(nr=formindex)
# br.submit()


def getData(html, outfile):
    """
    Gets data from one page of events
    """
    root = lxml.html.fromstring(html)
    lines = root.find_class("calendar__row calendar_row calendar__row--grey")
    # curWeekDay = None
    curMonthDay = None
    time = curTime = ""
    # pp.pprint(lines)
    for event in lines:
        # pp.pprint(event)
        if len(event.xpath("td[@class='calendar__cell calendar__date date']")) > 0:
            date = event.xpath("td[@class='calendar__cell calendar__date date']")[0]
        else:
            sys.exit("BOOM")

        # get the day of the month
        weekDay = date.xpath("span")
        monthDay = date.xpath("span/span")
        if len(weekDay) > 0:
            # curWeekDay = weekDay[0].text
            # print "curWeekDay=[" + curWeekDay + "]"
            curMonthDay = monthDay[0].text
            if debug:
                print "curMonthDay=[" + curMonthDay + "]"

        # get the time
        curTime = time
        time = event.xpath("td[contains(@class, 'calendar__time')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__time')]")) else ""
        if time == '' or time == None:
            time = curTime
        if debug:
            print "time=[" + str(time) + "]"

        # get currency
        currency = event.xpath("td[contains(@class, 'calendar__currency')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__currency')]")) else ""
        if debug:
            print "currency=[" + currency + "]"

        # get impact
        impact = event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")[0] if len(event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")) else ""
        if debug:
            print "impact=[" + impact + "]"

        # get name of event
        nevent = event.xpath("td[contains(@class, 'calendar__event')]/div/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__event')]/div/span")) else ""
        if debug:
            print "nevent=[" + nevent + "]"

        # get actual
        actual = event.xpath("td[contains(@class, 'calendar__actual')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__actual')]/span")) else ""

        # retry if actual is in a span (can happen if they colorize it)
        # if actual is None or len(actual.strip()) == 0:
        #     actual = event.xpath("td[@class='actual']/span")[0].text if len(event.xpath("td[@class='actual']/span")) else ""
        actual = actual.strip().replace("\n", " ") if actual is not None else ""
        if debug:
            print "actual=[" + actual + "]"

        # get forecast
        forecast = event.xpath("td[contains(@class, 'calendar__forecast')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__forecast')]")) else ""
        # retry if forecast is in a span (can happen if they colorize it)
        # if forecast is None or len(forecast.strip()) == 0:
        #    forecast = event.xpath("td[@class='forecast']/span")[0].text if len(event.xpath("td[@class='forecast']/span")) else ""
        forecast = forecast.strip().replace("\n", " ") if forecast is not None else ""
        if debug:
            print "forecast=[" + forecast + "]"

        # get previous
        previous = event.xpath("td[contains(@class, 'calendar__previous')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]")) else ""
        # retry if previous is in a span (can happen if they colorize it)
        if previous is None or len(previous.strip()) == 0:
            previous = event.xpath("td[contains(@class, 'calendar__previous')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]/span")) else ""
        previous = previous.strip().replace("\n", " ") if previous is not None else ""
        if debug:
            print "previous=[" + previous + "]\n"

        outfile.write("{};{};{};{};{};{};{};{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))


year = START_YEAR
outfile = open(OUTFILE, "w")
while year <= END_YEAR:
    for month in monthslist:
        url = "{}{}.{}".format(URL, month, year)
        print("Getting {} {} from {}".format(month, year, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
    year += 1
outfile.close()

Reply With Quote
The following user says Thank You to Fu510n for this post:
 
  #4 (permalink)
Trading Apprentice
London, UK
 
Futures Experience: Master
Platform: London hedge fund
Broker/Data: Goldman Sachs
Favorite Futures: Crude CL
 
wintergasp's Avatar
 
Posts: 29 since Sep 2016
Thanks: 1 given, 29 received

Yes, they change their HTML often, I tried to parse many things from investing.com and forex factory in the past and after a while it always break.

I ended up writing my own Calendar Events method.... when you look at it closely, there are maybe 10 public holiday in the US, 10 in the UK, and most macro events are like "first wednsday of the month" kind of schedule. For Fed talks and other government talks, most Fed websites and ECB website provide you with an XML feed of events that you can pull once a week or so.

Reply With Quote
 
  #5 (permalink)
Elite Member
Cary, NC
 
Futures Experience: Advanced
Platform: MC, NT, Python, R
Broker/Data: FXCM, IB, Oanda, IQFeed
Favorite Futures: 6E, CL, EUR/USD
 
Fu510n's Avatar
 
Posts: 94 since Oct 2009
Thanks: 726 given, 77 received

Re-updated with week/month options

# ffcal.py -?
ffcal.py <-h> <-f {filename}> <-w {this|next|mmmdd.yyyy}> <-m {this|next|mmm.yyyy}>

-h : display usage
-f {filename} : direct output to file instead of stdout (default)
-w {this|next|mmmdd.yyyy} : output specific week
-m {this|next|mmm.yyyy} : output specific month

If you don't specify a specific week/month, it will walk through all months for the current year.

# ffcal.py -w this
Getting this from Forex Calendar @ Forex Factory (stderr)
Jan 8;5:30pm;AUD;Low Impact Expected;AIG Construction Index;;;46.6
Jan 8;All Day;JPY;Non-Economic;Bank Holiday;;;
Jan 8;7:30pm;AUD;Medium Impact Expected;Building Approvals m/m;7.0%;4.6%;-11.8%
Jan 8;7:30pm;AUD;Low Impact Expected;ANZ Job Advertisements m/m;;;1.6%
...

 
Code
#!/usr/bin/env python

from __future__ import unicode_literals
import sys
import datetime
import getopt
# import codecs
import pprint
import lxml.html
import mechanize
import cookielib

# some utils
pp = pprint.PrettyPrinter()
debug = 0


#########################
# variables
#########################
START_YEAR = datetime.datetime.now().year
END_YEAR = START_YEAR
WEEKURL = r"http://www.forexfactory.com/calendar.php?week="
MONTHURL = r"http://www.forexfactory.com/calendar.php?month="
#OUTFILE = r"events.csv"
USAGE = "ffcal.py <-h> <-f {filename}> <-w {this|next|mmmdd.yyyy}> <-m {this|next|mmm.yyyy}>\n"
#########################


# our month list for the URL
monthslist = ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]

# sets up the browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# set correct timezone
br.open("http://www.forexfactory.com/timezone.php")
formindex = 0
for form in br.forms():
    if "timezone.php" in form.action:
        form["timezoneoffset"] = ["0"]
        break
    formindex += 1

br.select_form(nr=formindex)
# br.submit()


def getData(html, outfile):
    """
    Gets data from one page of events
    """
    root = lxml.html.fromstring(html)
    #lines = root.find_class("calendar__row calendar_row calendar__row--grey")
    #if not lines:
    lines = root.find_class("calendar__row calendar_row")

    # curWeekDay = None
    curMonthDay = None
    time = curTime = ""
    # pp.pprint(lines)
    for event in lines:
        # pp.pprint(event)
        if len(event.xpath("td[@class='calendar__cell calendar__date date']")) > 0:
            date = event.xpath("td[@class='calendar__cell calendar__date date']")[0]
        else:
            sys.exit("BOOM")

        # get the day of the month
        weekDay = date.xpath("span")
        monthDay = date.xpath("span/span")
        if len(weekDay) > 0:
            # curWeekDay = weekDay[0].text
            # print "curWeekDay=[" + curWeekDay + "]"
            curMonthDay = monthDay[0].text
            if debug:
                print "curMonthDay=[" + curMonthDay + "]"

        # get the time
        curTime = time
        time = event.xpath("td[contains(@class, 'calendar__time')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__time')]")) else ""
        if time == '' or time == None:
            time = curTime
        if debug:
            print "time=[" + str(time) + "]"

        # get currency
        currency = event.xpath("td[contains(@class, 'calendar__currency')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__currency')]")) else ""
        if currency == None:
            continue
        if debug:
            print "currency=[" + currency + "]"

        # get impact
        impact = event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")[0] if len(event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")) else ""
        if debug:
            print "impact=[" + impact + "]"

        # get name of event
        nevent = event.xpath("td[contains(@class, 'calendar__event')]/div/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__event')]/div/span")) else ""
        if debug:
            print "nevent=[" + nevent + "]"

        # get actual
        actual = event.xpath("td[contains(@class, 'calendar__actual')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__actual')]/span")) else ""

        # retry if actual is in a span (can happen if they colorize it)
        # if actual is None or len(actual.strip()) == 0:
        #     actual = event.xpath("td[@class='actual']/span")[0].text if len(event.xpath("td[@class='actual']/span")) else ""
        actual = actual.strip().replace("\n", " ") if actual is not None else ""
        if debug:
            print "actual=[" + actual + "]"

        # get forecast
        forecast = event.xpath("td[contains(@class, 'calendar__forecast')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__forecast')]")) else ""
        # retry if forecast is in a span (can happen if they colorize it)
        # if forecast is None or len(forecast.strip()) == 0:
        #    forecast = event.xpath("td[@class='forecast']/span")[0].text if len(event.xpath("td[@class='forecast']/span")) else ""
        forecast = forecast.strip().replace("\n", " ") if forecast is not None else ""
        if debug:
            print "forecast=[" + forecast + "]"

        # get previous
        previous = event.xpath("td[contains(@class, 'calendar__previous')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]")) else ""
        # retry if previous is in a span (can happen if they colorize it)
        if previous is None or len(previous.strip()) == 0:
            previous = event.xpath("td[contains(@class, 'calendar__previous')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]/span")) else ""
        previous = previous.strip().replace("\n", " ") if previous is not None else ""
        if debug:
            print "previous=[" + previous + "]\n"

        outfile.write("{};{};{};{};{};{};{};{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))


OUTFILE = ""

try:
    opts, args = getopt.getopt(sys.argv[1:], "f:hm:w:")
except getopt.GetoptError:
    sys.stderr.write(USAGE)
    sys.exit(2)

for opt, arg in opts:

    if opt == "-h":
        sys.stderr.write(USAGE)
        sys.exit()

    if opt == "-f":
        OUTFILE = arg
    elif opt == "-w" or opt == "-m":
        outfile = open(OUTFILE, "w") if OUTFILE != "" else sys.stdout
        if opt == "-w":
            url = "{}{}".format(WEEKURL, arg)
        else:
            url = "{}{}".format(MONTHURL, arg)
        sys.stderr.write("Getting {} from {}\n".format(arg, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
        if outfile is not sys.stdout:
            outfile.close()
        sys.exit()

year = START_YEAR
outfile = open(OUTFILE, "w") if OUTFILE != "" else sys.stdout
while year <= END_YEAR:
    for month in monthslist:
        url = "{}{}.{}".format(MONTHURL, month, year)
        sys.stderr.write("Getting {} {} from {}\n".format(month.title(), year, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
    year += 1
if outfile is not sys.stdout:
    outfile.close()

Reply With Quote
The following user says Thank You to Fu510n for this post:
 
  #6 (permalink)
Trading Apprentice
Antwerp + Antwerp/Belgium
 
Futures Experience: None
Platform: Ninjatrader
Favorite Futures: Stocks
 
Posts: 6 since Nov 2013
Thanks: 2 given, 0 received


Fu510n View Post
# ffcal.py -?
ffcal.py <-h> <-f {filename}> <-w {this|next|mmmdd.yyyy}> <-m {this|next|mmm.yyyy}>

-h : display usage
-f {filename} : direct output to file instead of stdout (default)
-w {this|next|mmmdd.yyyy} : output specific week
-m {this|next|mmm.yyyy} : output specific month

If you don't specify a specific week/month, it will walk through all months for the current year.

# ffcal.py -w this
Getting this from Forex Calendar @ Forex Factory (stderr)
Jan 8;5:30pm;AUD;Low Impact Expected;AIG Construction Index;;;46.6
Jan 8;All Day;JPY;Non-Economic;Bank Holiday;;;
Jan 8;7:30pm;AUD;Medium Impact Expected;Building Approvals m/m;7.0%;4.6%;-11.8%
Jan 8;7:30pm;AUD;Low Impact Expected;ANZ Job Advertisements m/m;;;1.6%
...

 
Code
#!/usr/bin/env python

from __future__ import unicode_literals
import sys
import datetime
import getopt
# import codecs
import pprint
import lxml.html
import mechanize
import cookielib

# some utils
pp = pprint.PrettyPrinter()
debug = 0


#########################
# variables
#########################
START_YEAR = datetime.datetime.now().year
END_YEAR = START_YEAR
WEEKURL = r"http://www.forexfactory.com/calendar.php?week="
MONTHURL = r"http://www.forexfactory.com/calendar.php?month="
#OUTFILE = r"events.csv"
USAGE = "ffcal.py <-h> <-f {filename}> <-w {this|next|mmmdd.yyyy}> <-m {this|next|mmm.yyyy}>\n"
#########################


# our month list for the URL
monthslist = ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]

# sets up the browser
br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# set correct timezone
br.open("http://www.forexfactory.com/timezone.php")
formindex = 0
for form in br.forms():
    if "timezone.php" in form.action:
        form["timezoneoffset"] = ["0"]
        break
    formindex += 1

br.select_form(nr=formindex)
# br.submit()


def getData(html, outfile):
    """
    Gets data from one page of events
    """
    root = lxml.html.fromstring(html)
    #lines = root.find_class("calendar__row calendar_row calendar__row--grey")
    #if not lines:
    lines = root.find_class("calendar__row calendar_row")

    # curWeekDay = None
    curMonthDay = None
    time = curTime = ""
    # pp.pprint(lines)
    for event in lines:
        # pp.pprint(event)
        if len(event.xpath("td[@class='calendar__cell calendar__date date']")) > 0:
            date = event.xpath("td[@class='calendar__cell calendar__date date']")[0]
        else:
            sys.exit("BOOM")

        # get the day of the month
        weekDay = date.xpath("span")
        monthDay = date.xpath("span/span")
        if len(weekDay) > 0:
            # curWeekDay = weekDay[0].text
            # print "curWeekDay=[" + curWeekDay + "]"
            curMonthDay = monthDay[0].text
            if debug:
                print "curMonthDay=[" + curMonthDay + "]"

        # get the time
        curTime = time
        time = event.xpath("td[contains(@class, 'calendar__time')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__time')]")) else ""
        if time == '' or time == None:
            time = curTime
        if debug:
            print "time=[" + str(time) + "]"

        # get currency
        currency = event.xpath("td[contains(@class, 'calendar__currency')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__currency')]")) else ""
        if currency == None:
            continue
        if debug:
            print "currency=[" + currency + "]"

        # get impact
        impact = event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")[0] if len(event.xpath("td[contains(@class, 'calendar__impact')]/div/span/@title")) else ""
        if debug:
            print "impact=[" + impact + "]"

        # get name of event
        nevent = event.xpath("td[contains(@class, 'calendar__event')]/div/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__event')]/div/span")) else ""
        if debug:
            print "nevent=[" + nevent + "]"

        # get actual
        actual = event.xpath("td[contains(@class, 'calendar__actual')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__actual')]/span")) else ""

        # retry if actual is in a span (can happen if they colorize it)
        # if actual is None or len(actual.strip()) == 0:
        #     actual = event.xpath("td[@class='actual']/span")[0].text if len(event.xpath("td[@class='actual']/span")) else ""
        actual = actual.strip().replace("\n", " ") if actual is not None else ""
        if debug:
            print "actual=[" + actual + "]"

        # get forecast
        forecast = event.xpath("td[contains(@class, 'calendar__forecast')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__forecast')]")) else ""
        # retry if forecast is in a span (can happen if they colorize it)
        # if forecast is None or len(forecast.strip()) == 0:
        #    forecast = event.xpath("td[@class='forecast']/span")[0].text if len(event.xpath("td[@class='forecast']/span")) else ""
        forecast = forecast.strip().replace("\n", " ") if forecast is not None else ""
        if debug:
            print "forecast=[" + forecast + "]"

        # get previous
        previous = event.xpath("td[contains(@class, 'calendar__previous')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]")) else ""
        # retry if previous is in a span (can happen if they colorize it)
        if previous is None or len(previous.strip()) == 0:
            previous = event.xpath("td[contains(@class, 'calendar__previous')]/span")[0].text if len(event.xpath("td[contains(@class, 'calendar__previous')]/span")) else ""
        previous = previous.strip().replace("\n", " ") if previous is not None else ""
        if debug:
            print "previous=[" + previous + "]\n"

        outfile.write("{};{};{};{};{};{};{};{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))


OUTFILE = ""

try:
    opts, args = getopt.getopt(sys.argv[1:], "f:hm:w:")
except getopt.GetoptError:
    sys.stderr.write(USAGE)
    sys.exit(2)

for opt, arg in opts:

    if opt == "-h":
        sys.stderr.write(USAGE)
        sys.exit()

    if opt == "-f":
        OUTFILE = arg
    elif opt == "-w" or opt == "-m":
        outfile = open(OUTFILE, "w") if OUTFILE != "" else sys.stdout
        if opt == "-w":
            url = "{}{}".format(WEEKURL, arg)
        else:
            url = "{}{}".format(MONTHURL, arg)
        sys.stderr.write("Getting {} from {}\n".format(arg, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
        if outfile is not sys.stdout:
            outfile.close()
        sys.exit()

year = START_YEAR
outfile = open(OUTFILE, "w") if OUTFILE != "" else sys.stdout
while year <= END_YEAR:
    for month in monthslist:
        url = "{}{}.{}".format(MONTHURL, month, year)
        sys.stderr.write("Getting {} {} from {}\n".format(month.title(), year, url))
        br.open(url)
        html = br.response().read()
        getData(html, outfile)
    year += 1
if outfile is not sys.stdout:
    outfile.close()

hi,

do you have an alternative for python 3? I see that mechanize is only available for python2. it'd be awesome if you could change it for python3 as I'm not that good of a python programmer yet!

thanks


Reply With Quote
 
  #7 (permalink)
Trading Apprentice
London, UK
 
Futures Experience: Intermediate
Platform: IG
Favorite Futures: Forex
 
Posts: 12 since Mar 2014
Thanks: 25 given, 10 received

What do people typically use the calendar data for, is it for some sort of research for automated trading?

I have written a program in AutoIT to download and display the CSV calendar from the FXStreet website, but I'm a bit wary of releasing it.

If the program got modified and ends up downloading far to frequently due to abuse or a bug then the download facility may get withdrawn. It basically works OK at the moment, if there's some good interest I'll look into tidying it up and adapting it so it can use the data from FX Factory also, but I'm really short on time these days so it may take a few weeks.

The user interface is pretty minimal but has some highlighting and filtering features, it can also give a 5 minute and due reminder for news events (depending on highlighting), and download an updated calendar 1 minute after events are due (again depending on the highlighting.)

Attached Thumbnails
forexfactory calendar downloader-eventcal1.png   forexfactory calendar downloader-eventcal2.png  
Reply With Quote
The following user says Thank You to smileypete for this post:
 
  #8 (permalink)
Trading Apprentice
Montréal
 
Futures Experience: None
Platform: td
Favorite Futures: stocks
 
Posts: 1 since Apr 2017
Thanks: 0 given, 0 received

The python code is incorrect

Something is not working with the scraping.


1-Jan 6:30pm AUD Low Impact Expected AIG Manufacturing Index 54.4


No actual data is scraped, but on the website you can see :

6:30pm AUD AIG Manufacturing Index 52.4 54.4


52.4 !

which is the actual data.

Reply With Quote
 
  #9 (permalink)
Trading Apprentice
London, UK
 
Futures Experience: Intermediate
Platform: IG
Favorite Futures: Forex
 
Posts: 12 since Mar 2014
Thanks: 25 given, 10 received

For historical data it's probably better to download CSV data from the FX Street website.

Seems to go back as far as Jan 2007, I've downloaded from then to end of March this year, with each year in a single .CSV file, and put them all in a zip file, hope that helps.

Attached Files
Register to download File Type: zip FxStreetEconomicCalendar2007ToMar2017.zip (863.0 KB, 18 views)
Reply With Quote
The following 2 users say Thank You to smileypete for this post:
 
  #10 (permalink)
Trading Apprentice
Auckland, New Zealand
 
Futures Experience: Intermediate
Platform: MetaTrader
Favorite Futures: Forex
 
Posts: 2 since May 2017
Thanks: 4 given, 2 received



leocazanova View Post
Something is not working with the scraping.


1-Jan 6:30pm AUD Low Impact Expected AIG Manufacturing Index 54.4


No actual data is scraped, but on the website you can see :

6:30pm AUD AIG Manufacturing Index 52.4 54.4


52.4 !

which is the actual data.

Hi after some tweaking, I finally made it to work. However, since I'm just new to this forum I cannot upload the fully working code because of some restrictions. Anyway, here are the modifications I did:

uncomment this line by removing "#": # if actual is None or len(actual.strip()) == 0:
replace this line: # actual = event.xpath("td[@class='actual']/span")[0].text if len(event.xpath("td[@class='actual']/span")) else ""

with: actual = event.xpath("td[contains(@class, 'calendar__actual')]")[0].text if len(event.xpath("td[contains(@class, 'calendar__actual')]")) else ""

I also changed: outfile.write("{};{};{};{};{};{};{};{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))

to: outfile.write("{},{},{},{},{},{},{},{}\n".format(curMonthDay, time, currency, impact, nevent, actual, forecast, previous))

so that it can be open easily with excel.

Reply With Quote
The following user says Thank You to datascientist for this post:

Reply



futures io > > > > forexfactory calendar downloader

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

July Journal Challenge w/$1100 in prizes from TopstepTrader

July
 

An Afternoon with FIO member Softsoap

Elite only

Battlestations: Show us your trading desks and win $750 in prizes

August
 

Extended Ask Me Anything w/Brett Steenbarger

Elite only

Prototyping Python Strategies (part 3) w/Ran Aroussi

Elite only

Brannigan Barrett w/Axia Futures

Elite only

Introduction to Phillip Capital futures brokerage services

Aug TBD

How Monte Carlo Analysis Can Help Your Trading w/Kevin Davey

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Economic Events Downloader (news) Nicolas11 Platforms and Indicators 25 March 16th, 2017 03:17 AM
chard downloader NT7 snusnufreak NinjaTrader 3 December 10th, 2013 04:02 PM
Market Analyzer as a Historical Downloader? LostTrader NinjaTrader 2 February 17th, 2013 09:49 PM
NT7 excel downloader tinkerz NinjaTrader 0 February 16th, 2013 04:11 PM
Data Downloader StockJock AmiBroker 22 April 8th, 2012 01:44 PM


All times are GMT -4. The time now is 07:02 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-07-22 in 0.14 seconds with 20 queries on phoenix via your IP 54.224.233.92