What should be the next step in my backtesting?? - Emini Index Futures Trading | futures io social day trading
futures io futures trading


What should be the next step in my backtesting??
Updated: Views / Replies:6,584 / 76
Created: by budfox Attachments:4

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 4  
 
Thread Tools Search this Thread
 

What should be the next step in my backtesting??

  #1 (permalink)
Membership Temporarily Revoked
Toronto
 
Futures Experience: Beginner
Platform: Sierra
Broker/Data: MB
Favorite Futures: ES
 
Posts: 313 since Jun 2013
Thanks: 182 given, 58 received

What should be the next step in my backtesting??

Hi,

I am currently backtesting a new model in excel, where I tested 47 trades (I am doing this manually; don't have any backtesting software, thats why I currently only tested 47 trades) here are my statistics:
  • Win ratio 55%
  • Average Profit: 1.4 points
  • Average Loss: 0.9 points

So I was reading a thread that Mike posted and downloaded this Monte Carlo simulation model (which is attached), I ran numerous simulations and it reported positive profit after 47 trades (ranging from $400 to $2000)

I am not sure if this is too good to be true or not, or how valid/relevant the model Mike posted is.

I am hoping backtesting experts such as @kevinkdog @swz168 @Fat Tails @DNFX @treydog999 @sixtyseven @LightWeight @SMCJB @deaddog @josh @ could provide some guidance.

This is for the ES contract of course.

My next likely steps in my plan are to :
  1. trade this in sim over the next three months (incubation)
  2. compare how my equity stacks up against the MC model

This is my first model I am building and really don't feel confident.

Thanks to those that provide constructive replies.

Attached Files
Register to download File Type: xls montecarloanalysis.xls (213.5 KB, 22 views)

Last edited by budfox; September 3rd, 2014 at 04:36 PM.
Reply With Quote
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Membership Temporarily Revoked
Toronto
 
Futures Experience: Beginner
Platform: Sierra
Broker/Data: MB
Favorite Futures: ES
 
Posts: 313 since Jun 2013
Thanks: 182 given, 58 received


Here is my equity curve:
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

Reply With Quote
 
  #4 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received

Here is just a quick rundown of what I saw and how I interpret my own backtests. so take this with a grain of salt. I am not judging manual vs automated testing or any of that at all, just purely at the results given.

So, the first thing i notice is low sample size 47 trades. Yes statisticians say 30+ is "significant" for a sample however more is always better. A quick plug into the old formula you get 14.43% standard error with this sample set alone. So any monte carlo based on that is going to have a built in 14% error either way. Monte carlos are only as good as the data set they can take from.

Looking at the trade stats you NET .5(pt) per trade and win 55% of the time. Is this with or without commissions? If not then they are going to take a sizable portion of your edge and make this system very difficult to trade. If it is with commissions then go on right ahead to incubation mode.

Another thing you did not mention is how many rules you have or filters. When you have a sample size this small it really can effect and skew both results and monte carlos. If you have 500 + trades and have 5 filters your ok, but if you 50 trades and 5 filters thats not ok. Statistically speaking, although i am a huge proponent of less filters, and less rules are more robust.

Also mention the date range that your using to test. Is this over 1 day, 1 month, 1 year? Both insample and out of sample to cover atleast 1 full market cycle. Though optimal is not very realistic, however more time is better. I look to have at least 3-5 years segregated for out of sample.

Hope to hear how this all works out.

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
  #5 (permalink)
Membership Temporarily Revoked
Toronto
 
Futures Experience: Beginner
Platform: Sierra
Broker/Data: MB
Favorite Futures: ES
 
Posts: 313 since Jun 2013
Thanks: 182 given, 58 received

Unsure.

First, I have to thank you for taking the time to help me out. I really appreciate it . (sitting here at home trying to figure this all out on my laptop).

Please realize that this is my first model ever I am attempting to implement using the @kevinkdog style. I have no experience whatsoever in backtesting (but did read his 'Taking it Live' thread).

My challenge is being able to backtest 200 samples without using any software.




treydog999 View Post

So, the first thing i notice is low sample size 47 trades. Yes statisticians say 30+ is "significant" for a sample however more is always better. A quick plug into the old formula you get 14.43% standard error with this sample set alone. So any monte carlo based on that is going to have a built in 14% error either way. Monte carlos are only as good as the data set they can take from.

Yes I realize N is small, this is due to the fact I am manually backtesting this (ie looking at the trade and manually recording the entry/exit, profit/loss details into excel.

Currently N= 50 Ave win = 1.38point Ave loss = 0.87

I currently havent figured out how to use backtesting software, (most likely will get Ninja Strategy Analyzer)


Quoting 
A quick plug into the old formula you get 14.43% standard error with this sample set alone.

Sorry, which formula is this? ( I still have a lot to learn).


Quoting 
Looking at the trade stats you NET .5(pt) per trade and win 55% of the time. Is this with or without commissions? If not then they are going to take a sizable portion of your edge and make this system very difficult to trade. If it is with commissions then go on right ahead to incubation mode.

No I forgot to mention commissions, they are 4.7, but I did factor them into the monte carlo analysis.


Quoting 
Another thing you did not mention is how many rules you have or filters. When you have a sample size this small it really can effect and skew both results and monte carlos. If you have 500 + trades and have 5 filters your ok, but if you 50 trades and 5 filters thats not ok. Statistically speaking, although i am a huge proponent of less filters, and less rules are more robust.

I have 5 rules. So should I eliminate some of them and rebacktest?


Quoting 
Also mention the date range that your using to test. Is this over 1 day, 1 month, 1 year? Both insample and out of sample to cover atleast 1 full market cycle.

The sample ranges from 14 April to 2 September.

How do you determine a market cycle?

Btw how much experience do you have backtesting? and what software you use to do it?

Thanks a lot Trey....I really need your help

Have a great day

Reply With Quote
 
  #6 (permalink)
Membership Temporarily Revoked
Toronto
 
Futures Experience: Beginner
Platform: Sierra
Broker/Data: MB
Favorite Futures: ES
 
Posts: 313 since Jun 2013
Thanks: 182 given, 58 received

use this one

Discard the MC file I earlier attached (its flawed), use this one instead

It would be great if someone could explain to me how to use the second sheet (Monte Carlo).



Software would be much easier since I wouldn't have to manualy change all the trades when I need to remove/add filters.

TY

Attached Files
Register to download File Type: rar bettermontecarlo.rar (670.6 KB, 14 views)

Last edited by budfox; September 4th, 2014 at 04:28 PM. Reason: add file
Reply With Quote
 
  #7 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


budfox View Post
First, I have to thank you for taking the time to help me out. I really appreciate it . (sitting here at home trying to figure this all out on my laptop).

Please realize that this is my first model ever I am attempting to implement using the @kevinkdog style. I have no experience whatsoever in backtesting (but did read his 'Taking it Live' thread).

My challenge is being able to backtest 200 samples without using any software.





Yes I realize N is small, this is due to the fact I am manually backtesting this (ie looking at the trade and manually recording the entry/exit, profit/loss details into excel.

Currently N= 50 Ave win = 1.38point Ave loss = 0.87

I currently havent figured out how to use backtesting software, (most likely will get Ninja Strategy Analyzer)

Yes actually having software, is really important as it speeds up your testing. Every software has its pros and cons its up to you to figure out what kind of quirks each system has. I know Ninja is well known to have errors or give erroneous answers in certain situations but have never used it myself. I personally have used MultiCharts and have reported every bug i detected, and i use R (quantstrat).

Sorry, which formula is this? ( I still have a lot to learn).
standard error of a mean is technically StdDev / sqrt(N). I shortcut it and make it 1/sqrt(Trades). I think i did Trades+1 in the previous example cause I was not sure how many degrees of freedom you have. But its a quick and dirty way of getting some error benchmark.


No I forgot to mention commissions, they are 4.7, but I did factor them into the monte carlo analysis.
Where they included in the original trade results as well? If they were included in the monte carlo thats fine also but its good to include them at every stage of the game.


I have 5 rules. So should I eliminate some of them and rebacktest?
Hard to say, as every system is different. Another general rule is managing degrees of freedom. Your degrees of freedom are your trades (N) - 1. So you have 50-1 = 49 degrees of freedom. You do not want more than 1% degrees of freedom so you want as a MAX .49 rules/filters/ anything using a >, < ,and, or, in code terms. So in your example your going to want to have only 1 rule. Again this is because of a small data set, it forces you to use less rules. Usually i stick around the .5% degrees of freedom. Any more then that and you are using rules to force the data to give you the results you want. Any statistician can beat the data enough to show you what you want.


The sample ranges from 14 April to 2 September.
Not nearly long enough, you need at least 2-3 years. Manually that can take you a very long time.
How do you determine a market cycle?
Well you can do a seasonal analysis and determine what its cycle is. I did some seasonal stuff in my development thread a long time ago. Or you can visually inspect to see if you can identify long term bull/bear markets. They key is you want to encapsulate every type of market, trending, ranging, bull bear into your training and testing sets to make it robust. if you only test on bull markets, then a long strategy is always going to work. and vice versa
Btw how much experience do you have backtesting? and what software you use to do it?
I have been backtesting for several years now. I have taken courses in Financial engineering as well, also done some analysis and backtesting for some small/mid sized trading firms. Software for the "Retailer" I believe multicharts is the best. I have tested and debugged it myself and gotten most of the bugs fixed by the MC team. I trust the results since my live vs backtesting has been within 1-2 ticks of entry and exit when coded properly. Also it supports easylanguage or C# depending on your level of coding ability.

Flip side for more detailed work and for something I have to submit to others. I almost always use R, in a combination of packages including, quantstrat, TTR, quantmod, and redis (for redistributable computing). Along with some custom coded functions and trading reports I developed myself. R simulates an Orderbook rather than trades, so you can see the entire market timeline and how orders became trades and then how trades were worked out. This is better for cross checking, as well as you can add specific delays to see how that affects you. As opposed to ninja trader or multicharts where you just put (next bar). I can set exact latency times for specific situations.

Thanks a lot Trey....I really need your help

Have a great day

Hope that cleared some stuff up.

Reply With Quote
The following 3 users say Thank You to treydog999 for this post:
 
  #8 (permalink)
Membership Temporarily Revoked
Toronto
 
Futures Experience: Beginner
Platform: Sierra
Broker/Data: MB
Favorite Futures: ES
 
Posts: 313 since Jun 2013
Thanks: 182 given, 58 received

Where do you get your data from? how much do you pay for it?



I'm playing around with the Strategy Analyzer in NT now since its free, but at a later stage in my career I will look at some of the quant software.

I am trying to figure out what is a good way to learn all this.

Thanks.

Reply With Quote
 
  #9 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


budfox View Post
Where do you get your data from? how much do you pay for it?



I'm playing around with the Strategy Analyzer in NT now since its free, but at a later stage in my career I will look at some of the quant software.

I am trying to figure out what is a good way to learn all this.

Thanks.

I have CQG and DTN iq feed. You can search their websites to see which service fits your needs. i would say DTN is best bang for the buck. However the best free resource is quandl, you can download a lot of free good data there.

Reply With Quote
The following 3 users say Thank You to treydog999 for this post:
 
  #10 (permalink)
Elite Member
Singapore
 
Futures Experience: None
Platform: -
Favorite Futures: Diversified
 
Ming80's Avatar
 
Posts: 41 since Jan 2012
Thanks: 25 given, 49 received


Hi BudFox,

Glad you've built your first model. Just like you, when I started my first model from excel I had tons of questions and wanted to share any some info that might be of use to you.

1. On Backtesting Software - Treydog has given excellent points. Apart from the speed and scale which you can backtest many markets/asset classes, there is also the huge benefit of automated execution. I found my usage of time much more productive in research instead of monitoring trades. One can now trade up to 20 or more positions simulatneously (long term or short term). The caveat of course is that the model developed is accurate and robust and monitoring systems from time to time is still a prerequisite.

2. On Monte Carlo - It is an interesting measure but wouldn't put too much faith in it. Actual drawdowns are a function of a specific market events/environment in relation a series of bad trades produced. Trade scrambling removes the serial correlation which happens during this period and how the drawdown was actually produced. It would be useful to look into the periods which drawdowns occured and understand what caused them. This is crucial to deduce if risk management approaches are needed (stopping the trading model, develop non-correlated models etc.) ... All these decisions have 'costs' to the long term outcome of the eventual pnl. Alternatively, one could accept that this is a function of markets where drawdowns are inevitable from time to time and just continue to stick to the plan.

3. Number of trades - Perhaps instead of measures of statistical significance, one could take a qualitative approach and see what the system was designed to capture. Warren buffett as well as many long term trend followers although might fail statistical tests by their number of trades, but their performance is certainly more than random. One needs to take into account the holding frequency and frequency of conditions present.

4. Clean Data - Good models need to be built on clean and more importantly adjusted data. Even accredited data providers do not provide properly adjusted data for backtests and will lead to incorrect trading conclusions. Thus, the need for backtesting software to adjust data to generate a proper backtest.

Lastly, I personally found the most crucial parts in building good trading models was in thinking and capturing the stationarity of the market. I got this general idea from visiting the local library and reading books by clifford j sherry. The 2nd part was asking myself how long this 'stationarity' would tend to persist. The better which these 2 questions could be answered, the better the models have performed going forward. I hope some of my personal experiences would be of use to you going forward.

Reply With Quote
The following 2 users say Thank You to Ming80 for this post:

Reply



futures io > > > > What should be the next step in my backtesting??

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Step MA crazybears Sierra Chart 2 July 15th, 2014 06:25 AM
Top Step Trader website down? birdoggg Platforms and Indicators 2 November 7th, 2012 10:07 AM
The Next Step Towards The End Of The Euro Quick Summary News and Current Events 0 November 16th, 2011 12:50 AM


All times are GMT -4. The time now is 01:02 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-18 in 0.19 seconds with 20 queries on phoenix via your IP 54.196.201.241