Strategy Fitness Test - NinjaTrader Programming | futures io social day trading
futures io futures trading


Strategy Fitness Test
Updated: Views / Replies:3,421 / 17
Created: by Zach b Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Strategy Fitness Test

  #1 (permalink)
Elite Member
NV
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: fx
 
Posts: 21 since Mar 2010
Thanks: 18 given, 11 received

Strategy Fitness Test

This is for anyone who automates their strategies. I was wondering at what point you consider a strategy good enough to go into sim/live trading. What kind of sample size do you look for, do you have a profit VS drawdown you look for. Do you have a PF requirement? Risk of ruin etc? Any insight would be greatly appreciated as I feel that I don't really know at what point I can consider a strategy (that has not been over-optimized) ready to go. Thanks in advance.

Reply With Quote
The following user says Thank You to Zach b for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received


Hi Zach,

That's a very, very open-ended and highly subjective question

I'm currently reading the 500+ page book "Evidence Based Technical Analysis", and even this book, as excellent as it is, cannot answer your post in a definitive fashion so I don't think you can really ever expect an answer.

But, in loose generalized outline form, I look for:
- At least 1 year of tick level data
- At least 1,000 trades in sample
- Always do a out of sample/walk forward analysis, make sure the results are at least comparable to the in sample results
- Always include commission costs
- Always include slippage costs
- Always sim test for 1-2 weeks before going live, and check every order carefully
- Don't use Renko, Point and Figure, Kagi or Line Break charts
- Incorporate daily stop and targets that you can live with* (more on this below)

* This is a big one. If your automated strategy says it will make a million bucks, but you lose faith in it when on cash simply because it is in a drawdown period, it does no good for anyone. Put another way, if you can't stomach a $ 1,000 daily draw down for two weeks straight, then make sure your strategy can't either. If you can't stomach having a $ 1,000 profitable trade turn into $ 0, or even a loss, then make sure your strategy can't either. In other words, make sure that you will not be tempted to override the strategy or stop trading the strategy, etc etc, or otherwise you end up negating all the backtesting.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 3 users say Thank You to Big Mike for this post:
 
  #4 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received

A few more:

- Don't have separate parameters for Longs vs. Shorts, to do so introduces a hindsight bias and curve fitting bias, not unless you can predict the future

- Always aim for a positive expectancy

- Don't optimize based on net profit

- Don't choose the best performer from the in sample data, instead make careful decisions primarily based on the out of sample data

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #5 (permalink)
Elite Member
Near the BEuTiFULL Horse Shoe
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: MBTrading Dukascopy ZenFire
Favorite Futures: $EURUSD when it is trending
 
Trader.Jon's Avatar
 
Posts: 500 since Jul 2009
Thanks: 401 given, 176 received

"""" - Don't use Renko, Point and Figure, Kagi or Line Break charts """"

Is this a general rule on all markets?
Are there specific reasons for each of these chart types, or is this a general subjective &/or objective determination?

TJ

Reply With Quote
 
  #6 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received


Trader.Jon View Post
"""" - Don't use Renko, Point and Figure, Kagi or Line Break charts """"

Is this a general rule on all markets?
Are there specific reasons for each of these chart types, or is this a general subjective &/or objective determination?

TJ

Yes, it's a general rule, all markets, related to backtesting problems only.

Basically, if you don't know about the problems generated by backtesting these chart types, then do not use them. If you do know, then you know how to overcome them and all the extra code it will take, and you can disregard.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #7 (permalink)
Elite Member
NV
 
Futures Experience: Intermediate
Platform: NinjaTrader
Favorite Futures: fx
 
Posts: 21 since Mar 2010
Thanks: 18 given, 11 received


Big Mike View Post
Hi Zach,

That's a very, very open-ended and highly subjective question

I'm currently reading the 500+ page book "Evidence Based Technical Analysis", and even this book, as excellent as it is, cannot answer your post in a definitive fashion so I don't think you can really ever expect an answer.

But, in loose generalized outline form, I look for:
- At least 1 year of tick level data
- At least 1,000 trades in sample
- Always do a out of sample/walk forward analysis, make sure the results are at least comparable to the in sample results
- Always include commission costs
- Always include slippage costs
- Always sim test for 1-2 weeks before going live, and check every order carefully
- Don't use Renko, Point and Figure, Kagi or Line Break charts
- Incorporate daily stop and targets that you can live with* (more on this below)

* This is a big one. If your automated strategy says it will make a million bucks, but you lose faith in it when on cash simply because it is in a drawdown period, it does no good for anyone. Put another way, if you can't stomach a $ 1,000 daily draw down for two weeks straight, then make sure your strategy can't either. If you can't stomach having a $ 1,000 profitable trade turn into $ 0, or even a loss, then make sure your strategy can't either. In other words, make sure that you will not be tempted to override the strategy or stop trading the strategy, etc etc, or otherwise you end up negating all the backtesting.

Mike

Thanks for the advise. I have been in the game enough to understand what it means to accept risk and stick with a plan. I just have a hard time determining at what point these automated plans are good enough to go live with. If I can ask you this, once you have your sample set and you have your historical draw down defined, how do you determine your point of system failure? Do you just say that if it draws down X*previousDrawdown that you will pull the plug?

Reply With Quote
 
  #8 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received


Zach b View Post
Thanks for the advise. I have been in the game enough to understand what it means to accept risk and stick with a plan. I just have a hard time determining at what point these automated plans are good enough to go live with. If I can ask you this, once you have your sample set and you have your historical draw down defined, how do you determine your point of system failure? Do you just say that if it draws down X*previousDrawdown that you will pull the plug?

Well, we know that a backtest is data mining and curve fitting. The likelihood that a future result will outperform a backtest, given sufficient sample size, is not very high.

So, if you accept that the backtest results were "optimal", you can accept that all of your performance measuring metrics were also optimal, and that going forward you should allow for wiggle room. For instance, if over 1,000 trades your win percentage was 50%, and your win/loss dollar ratio was 1.5, and your maximum close-to-close drawdown was $5,000, then you could reasonably define 'failure' by using these figures as guidelines and adding the 'wiggle room', such as a 45% win rate, a 1.2 wld ratio, or a drawdown exceeding $ 7,500.

There are more scientific ways of doing this such as using a standard deviation measurement. MultiCharts actually has a lot of these types of metrics built-in, whereas NinjaTrader does not.

If you still have doubts then I suggest you post the performance data for one of your strategies in question. Let's look at the sample size and the IS and OOS results and then take a look at a two-week period of live (sim) trading and see what we have.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #9 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received

Also you really can't place enough emphasis on using out of sample data. The real problem is that once out of sample data is used just once, it really is no longer virgin out of sample data.

For instance, if you have Strategy A and you run a backtest on 360 days of in-sample data, and 180 days of out-of-sample data. Let's say you didn't much like the results of the OOS data.

You can't just make changes to Strategy A and then re-run your backtest on the same IS and OOS data. You've effectively turned the OOS data into IS data, at least in your own mind, because you have the hindsight of know if your changes to the strategy have "improved" the result. Naturally, it's not OOS data if you can make a change to it and measure if the change is better or worse.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #10 (permalink)
Member
California
 
Futures Experience: Beginner
Platform: NinjaTrader
 
Posts: 87 since Jun 2009
Thanks: 5 given, 119 received


it's my opinion (for what it's worth) that it's as hard to develop a good metric with which to evaluate an automated strategy as it is to develop the strategy itself. it's also much more important to have a good metric than to have a good strategy - without a good metric you cannot know whether or not the strategy you have is any good.

the problem is that you have to find some function that takes into account all the various characteristics of your strategy's performance:
- expectancy
- draw-down
- cumulative profit
- cumulative/contiguous risk
- resilience to changes in market conditions
- etc..

... and it has to represent all of that all in a single number. but how do you weigh the various values with respect to each other?

i'd love to see some more discussion on what people consider to be a good measure of an automated strategy. and, no, for me the sharpe ratio doesn't cut it...

Reply With Quote
The following 3 users say Thank You to piersh for this post:

Reply



futures io > > > > > Strategy Fitness Test

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
System Optimization Fitness Functions fluxsmith NinjaTrader Programming 27 April 29th, 2011 07:52 PM
Expectancy custom fitness sam028 EasyLanguage Programming 13 September 29th, 2010 10:23 PM


All times are GMT -4. The time now is 12:52 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-11 in 0.15 seconds with 19 queries on phoenix via your IP 107.20.115.174