Zipline backtesting with local csv files - Matlab, R project and Python | futures io social day trading
futures io futures trading


Zipline backtesting with local csv files
Updated: Views / Replies:4,370 / 10
Created: by DarkPoolTrading Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Zipline backtesting with local csv files

  #1 (permalink)
 Vendor: diversifyportfolio.com 
PTA, Gauteng
 
Futures Experience: Advanced
Platform: Self built + Sierra + TWS
Favorite Futures: Stocks and Options
 
DarkPoolTrading's Avatar
 
Posts: 1,036 since May 2012
Thanks: 1,244 given, 1,304 received

Zipline backtesting with local csv files

Can someone please advise how to use zipline with locally stored csv files. Using yahoo finance or any other online sources is simple enough, but I can't seem to get it working with my own csv files.

From what I can tell there are two options:
- load the csv directly into a pandas dataframe using read_csv
- use fetch_csv to read in the locally stored file://

I haven't had any luck with either option. Im not sure if my problem lies in the python script, or am I using the incorrect flags when calling run_algo.py

I thought this would work (just as a basic example):

To keep it simple I downloaded some AAPL daily data directly from yahoo finance and added a symbol column (which I think is required?)

csv file:
 
Code
symbol, Date, Open, High, Low, Close, Volume, AdjClose
Script:

 
Code
from zipline.api import order, record, symbol
import pandas as pd

data = pd.read_csv('AAPL.csv')

def initialize(context):
    pass

def handle_data(context, data):
    order(data[symbol('AAPL')], 10)
    record(AAPL=data[symbol('AAPL')].price)
Then run it using:
 
Code
run_algo.py -f E:\..\Main.py --start 2010-1-1 --end 2015-1-1 --symbols AAPL -o AAPL_out.pickle

But no luck. Please can someone point me in the right direction. There's probably something really simple im doing wrong.

I hope you don't mind me tagging a few likely suspects who could probably help:
@ericbrown @Nicolas11 @treydog999 @traderwerks @Big Mike

Diversification is the only free lunch

Last edited by DarkPoolTrading; March 27th, 2015 at 12:33 PM.
Reply With Quote
The following user says Thank You to DarkPoolTrading for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
 Vendor: sentimentrader.com 
Tulsa, OK
 
Futures Experience: Advanced
Platform: Tradestation, TOS, Python
Broker/Data: IQFeed, Tradestation, TOS
Favorite Futures: ES, SPY, Options
 
ericbrown's Avatar
 
Posts: 201 since Jan 2011
Thanks: 339 given, 234 received


Your example is correct on reading the data in...I think your error is how you are using the data. Here's a sample strat using csv file to run a bollinger band strat. I haven't run this against data lately but it worked for me about a year ago....that said...if there are any issues with the code let me know and i can help explain or troubleshoot. Note: I saw your stackoverflow question and posted the same there.

 
Code
import pandas as pd
import locale
import matplotlib.pyplot as plt
from zipline.algorithm import TradingAlgorithm
import zipline.finance.trading as trading
from zipline.utils.factory import load_from_yahoo
from datetime import datetime, date
from pytz import timezone
import pytz
from dateutil.tz import tzutc
from dateutil.parser import parse
from zipline.finance.slippage import FixedSlippage
from zipline.finance.commission import PerShare, PerTrade


central = timezone('US/Central')
HOLDTIME = 5
locale.setlocale( locale.LC_ALL, 'en_US.UTF-8')
COMMISSION=0.005

def date_utc(s):
    return parse(s, tzinfos=tzutc)


class BollingerBands(TradingAlgorithm):   
   
    def initialize(self):
        self.invested       = False
       
        self.trade_size=1000
        self.long = False
        self.short = False
        self.closed = False
        self.trading_day_counter = 0  
        self.pct_stop = 0.025
        self.long_stop_price =  0.0
        self.short_stop_price=0.0
        self.target=0.05
        commission_cost = self.trade_size * COMMISSION
        self.set_slippage(FixedSlippage(spread=0.10))  
        self.set_commission(PerTrade(cost=commission_cost))  

    def handle_data(self, data):

       
        if data['Close'].price >= data['upper'].price and not self.invested:
            self.order('Close', self.trade_size)
            self.long_stop_price =  data['Open'].price - data['Open'].price * float(self.pct_stop) 
            self.short_stop_price =  data['Open'].price + data['Open'].price * float(self.target)
            self.long = True
            self.closed=False
            self.invested = True
            self.trading_day_counter = 0  
        if data['Close'].price <= data['lower'].price and not self.invested:
            self.short_stop_price =  data['Open'].price + data['Open'].price * float(self.pct_stop)
            self.long_stop_price =  data['Open'].price - data['Open'].price * float(self.target) 
            self.order('Close', -self.trade_size)
            self.short = True
            self.closed=False
            self.invested = True
            self.trading_day_counter = 0  
        if self.invested and (data['Close'].price <=  self.long_stop_price or data['Close'].price >=  self.short_stop_price): #or self.trading_day_counter == HOLDTIME):
  
            if self.long: 
                self.order('Close', -self.trade_size)
             if self.short:
                self.order('Close', self.trade_size)
 
            self.closed=True
            self.long=False
            self.short=False
            self.invested=False
           

        self.trading_day_counter =  self.trading_day_counter +1
        self.record(Close=data['Close'].price,
                    upper=data['upper'].price,
                    lower=data['lower'].price,
                    long=self.long,
                    short=self.short,
                    holdtime=self.trading_day_counter,
                    closed_position=self.closed,
                    shares=self.trade_size)


if __name__ == '__main__':
    df=pd.read_csv('SPY.csv') # contains Date, Open, High, Low, Close, Volume


    df['avg'] = pd.rolling_mean(df['Close'], 21)
    df['std'] = pd.rolling_std(df['Close'], 21)
    df['upper'] = df['avg'] + 2 * df['std']
    df['lower'] = df['avg'] - 2 * df['std']
    df = df.dropna()

    # # # # init Strat Class
    Strategy = BollingerBands()
    # #print df

    # # # # # # Run Strategy
    results = Strategy.run(df)
    results['algorithm_returns'] = (1 + results.returns).cumprod()

 
    results.to_csv('output.csv')
    print results['algorithm_returns'].tail(1)[0]*100

Reply With Quote
The following 2 users say Thank You to ericbrown for this post:
 
  #4 (permalink)
 Vendor: diversifyportfolio.com 
PTA, Gauteng
 
Futures Experience: Advanced
Platform: Self built + Sierra + TWS
Favorite Futures: Stocks and Options
 
DarkPoolTrading's Avatar
 
Posts: 1,036 since May 2012
Thanks: 1,244 given, 1,304 received

Thanks Eric,

I've gotten the below error. Im busy trying to figure it out but im such a complete beginner at this im not having any luck so far. I don't see assert used anywhere in the code.

I initially got an unsupported locale setting error. So I just commented out that line since I presume it's not necessary.

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py

[2015-03-27 14:18:28.519033] WARNING: Loader: No benchmark data found for date range.
start_date=2015-03-27 00:00:00+00:00, end_date=2015-03-27 14:18:27.818993, url=http://ichart.finance.yahoo.com/table.csv?d=2&e=27&f=2015&g=d&a=2&b=27&
c=2015&s=%5EGSPC

Traceback (most recent call last):
  File "Main.py", line 101, in <module>
    results = Strategy.run(df)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 372, in run
    source = DataFrameSource(source)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\sources\data_frame_source.py", line 42, in __init__
    assert isinstance(data.index, pd.tseries.index.DatetimeIndex)
AssertionError

Diversification is the only free lunch
Reply With Quote
 
  #5 (permalink)
 Vendor: sentimentrader.com 
Tulsa, OK
 
Futures Experience: Advanced
Platform: Tradestation, TOS, Python
Broker/Data: IQFeed, Tradestation, TOS
Favorite Futures: ES, SPY, Options
 
ericbrown's Avatar
 
Posts: 201 since Jan 2011
Thanks: 339 given, 234 received


DarkPoolTrading View Post
Thanks Eric,

I've gotten the below error. Im busy trying to figure it out but im such a complete beginner at this im not having any luck so far. I don't see assert used anywhere in the code.

I initially got an unsupported locale setting error. So I just commented out that line since I presume it's not necessary.

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py

[2015-03-27 14:18:28.519033] WARNING: Loader: No benchmark data found for date range.
start_date=2015-03-27 00:00:00+00:00, end_date=2015-03-27 14:18:27.818993, url=http://ichart.finance.yahoo.com/table.csv?d=2&e=27&f=2015&g=d&a=2&b=27&
c=2015&s=%5EGSPC

Traceback (most recent call last):
  File "Main.py", line 101, in <module>
    results = Strategy.run(df)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 372, in run
    source = DataFrameSource(source)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\sources\data_frame_source.py", line 42, in __init__
    assert isinstance(data.index, pd.tseries.index.DatetimeIndex)
AssertionError

The error is a zipline error around pulling their benchmark data (it always downloads the data from yahoo). I'll take a look over the weekend at this.

Reply With Quote
The following 2 users say Thank You to ericbrown for this post:
 
  #6 (permalink)
 Vendor: diversifyportfolio.com 
PTA, Gauteng
 
Futures Experience: Advanced
Platform: Self built + Sierra + TWS
Favorite Futures: Stocks and Options
 
DarkPoolTrading's Avatar
 
Posts: 1,036 since May 2012
Thanks: 1,244 given, 1,304 received

Ok, thanks Eric.

The first error I got was actually the following:

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 18, in <module>
    locale.setlocale( locale.LC_ALL, 'en_US.UTF-8')
  File "E:\Python_Projects\Zipline_Testing\venv\lib\locale.py", line 573, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
Following that I commented out the line where you set the locale, which resulted in the error I posted in the previous reply. I think the real problem is probably something to do with the locale. I'll try and sort that out.

Sorry for the issues. Im sure this code probably works fine, im just such a beginner at this it takes me a while to debug things.

Diversification is the only free lunch
Reply With Quote
 
  #7 (permalink)
 Vendor: sentimentrader.com 
Tulsa, OK
 
Futures Experience: Advanced
Platform: Tradestation, TOS, Python
Broker/Data: IQFeed, Tradestation, TOS
Favorite Futures: ES, SPY, Options
 
ericbrown's Avatar
 
Posts: 201 since Jan 2011
Thanks: 339 given, 234 received


DarkPoolTrading View Post
Ok, thanks Eric.

The first error I got was actually the following:

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 18, in <module>
    locale.setlocale( locale.LC_ALL, 'en_US.UTF-8')
  File "E:\Python_Projects\Zipline_Testing\venv\lib\locale.py", line 573, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
Following that I commented out the line where you set the locale, which resulted in the error I posted in the previous reply. I think the real problem is probably something to do with the locale. I'll try and sort that out.

Sorry for the issues. Im sure this code probably works fine, im just such a beginner at this it takes me a while to debug things.

No worries. Let me know if you need any help...I can take a look over the weekend.

Reply With Quote
The following user says Thank You to ericbrown for this post:
 
  #8 (permalink)
 Vendor: diversifyportfolio.com 
PTA, Gauteng
 
Futures Experience: Advanced
Platform: Self built + Sierra + TWS
Favorite Futures: Stocks and Options
 
DarkPoolTrading's Avatar
 
Posts: 1,036 since May 2012
Thanks: 1,244 given, 1,304 received


ericbrown View Post
No worries. Let me know if you need any help...I can take a look over the weekend.

Im pulling my hair out with this. Continuing to get the same error and just can't figure it out:

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 101, in <module>
    results = Strategy.run(df)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 372, in run
    source = DataFrameSource(source)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\sources\data_frame_source.py", line 42, in __init__
    assert isinstance(data.index, pd.tseries.index.DatetimeIndex)
AssertionError

(venv) E:\Python_Projects\Zipline_Testing>

Just a question, should this be run using the command line run_algo.py or just the regular python filename.py.
When running it using python main.py I get the above error.

When running it using :
run_algo.py -f E:\Python_Projects\Zipline_Testing\Main.py -o buyapple_out.pickle

I get the following:

 
Code
Traceback (most recent call last):
  File "E:\Python_Projects\Zipline_Testing\venv\Scripts\run_algo.py", line 24, in <module>
    run_pipeline(print_algo=True, **parsed)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\utils\cli.py", line 190, in run_pipeline
    capital_base=float(kwargs['capital_base']))
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 183, in __init__
    raise ValueError('You must define a handle_data function.')
ValueError: You must define a handle_data function.

I've researched the AssertionError to no avail.
Im running this in a Python 3.3 virtualenv if that makes any difference.

I find it really strange that zipline doesn't have an example of how to use your own csv files. They seem to be more concerned with Quantopian than with those wanting to use zipline directly.

Diversification is the only free lunch

Last edited by DarkPoolTrading; March 28th, 2015 at 08:07 AM.
Reply With Quote
 
  #9 (permalink)
 Vendor: diversifyportfolio.com 
PTA, Gauteng
 
Futures Experience: Advanced
Platform: Self built + Sierra + TWS
Favorite Futures: Stocks and Options
 
DarkPoolTrading's Avatar
 
Posts: 1,036 since May 2012
Thanks: 1,244 given, 1,304 received

@ericbrown Cool im making some progress. The csv needed to be read in using:
 
Code
df=pd.read_csv('AAPL.csv', parse_dates=True, index_col=0)

Now i've got a new error but at least this makes a bit more sense because I was initially getting an error with the locale setting, so I just commented it out. It looks like I just need to figure out how to set the locale properly and then I should hopefully be good to go.

Current error (when the locale setting section is commented out):

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 101, in <module>
    results = Strategy.run(df)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 391, in run
    self.sim_params._update_internal()
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\finance\trading.py", line 374, in _update_internal
    assert self.period_start <= environment.last_trading_day, \
  File "pandas\tslib.pyx", line 826, in pandas.tslib._Timestamp.__richcmp__ (pandas\tslib.c:15597)
  File "pandas\tslib.pyx", line 855, in pandas.tslib._Timestamp._assert_tzawareness_compat (pandas\tslib.c:15921)
TypeError: Cannot compare tz-naive and tz-aware timestamps

(venv) E:\Python_Projects\Zipline_Testing>

If I uncomment the section that sets the locale, I get the following:

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 18, in <module>
    locale.setlocale( locale.LC_ALL, 'en_US.UTF-8')
  File "E:\Python_Projects\Zipline_Testing\venv\lib\locale.py", line 573, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

(venv) E:\Python_Projects\Zipline_Testing>

I think im getting there slowly.

Diversification is the only free lunch
Reply With Quote
The following user says Thank You to DarkPoolTrading for this post:
 
  #10 (permalink)
Trading Apprentice
Melbourne, Land of Oz
 
Futures Experience: Advanced
Platform: NT, IB TWS
Favorite Futures: HSI
 
Posts: 1 since Jun 2011
Thanks: 0 given, 0 received



DarkPoolTrading View Post
@ericbrown Cool im making some progress. The csv needed to be read in using:
 
Code
df=pd.read_csv('AAPL.csv', parse_dates=True, index_col=0)

Now i've got a new error but at least this makes a bit more sense because I was initially getting an error with the locale setting, so I just commented it out. It looks like I just need to figure out how to set the locale properly and then I should hopefully be good to go.

Current error (when the locale setting section is commented out):

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 101, in <module>
    results = Strategy.run(df)
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\algorithm.py", line 391, in run
    self.sim_params._update_internal()
  File "E:\Python_Projects\Zipline_Testing\venv\lib\site-packages\zipline\finance\trading.py", line 374, in _update_internal
    assert self.period_start <= environment.last_trading_day, \
  File "pandas\tslib.pyx", line 826, in pandas.tslib._Timestamp.__richcmp__ (pandas\tslib.c:15597)
  File "pandas\tslib.pyx", line 855, in pandas.tslib._Timestamp._assert_tzawareness_compat (pandas\tslib.c:15921)
TypeError: Cannot compare tz-naive and tz-aware timestamps

(venv) E:\Python_Projects\Zipline_Testing>

If I uncomment the section that sets the locale, I get the following:

 
Code
(venv) E:\Python_Projects\Zipline_Testing>python Main.py
Traceback (most recent call last):
  File "Main.py", line 18, in <module>
    locale.setlocale( locale.LC_ALL, 'en_US.UTF-8')
  File "E:\Python_Projects\Zipline_Testing\venv\lib\locale.py", line 573, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

(venv) E:\Python_Projects\Zipline_Testing>

I think im getting there slowly.

How did you go with this? Did you end up having any success?

Reply With Quote

Reply



futures io > > > > Zipline backtesting with local csv files

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
ETF .txt or .csv List Prtester Stocks and ETFs Trading 3 May 7th, 2014 09:41 PM
multitimeframe live CSV export rirhasiu NinjaTrader 5 May 16th, 2013 03:15 PM
The Electronic Local method cunparis Trading Reviews and Vendors 105 August 18th, 2011 07:05 AM
Best platform for Backtesting own CSV files FAUGH45568 Platforms and Indicators 26 July 21st, 2011 07:20 AM
Local vs Exchange time Mondrakete TradeStation 2 May 8th, 2011 06:40 PM


All times are GMT -4. The time now is 12:13 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-15 in 0.13 seconds with 19 queries on phoenix via your IP 54.226.34.209