Backtesting: What is a good sample size? - futures io
futures io futures trading



Backtesting: What is a good sample size?


Discussion in Commodities

Updated
      Top Posters
    1. looks_one ellinas with 2 posts (0 thanks)
    2. looks_two tpredictor with 2 posts (1 thanks)
    3. looks_3 ClearTrades with 2 posts (1 thanks)
    4. looks_4 Bermudan Option with 1 posts (3 thanks)
      Best Posters
    1. looks_one rleplae with 4 thanks per post
    2. looks_two Bermudan Option with 3 thanks per post
    3. looks_3 sixtyseven with 1 thanks per post
    4. looks_4 ClearTrades with 0.5 thanks per post
    1. trending_up 4,126 views
    2. thumb_up 10 thanks given
    3. group 9 followers
    1. forum 12 posts
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Backtesting: What is a good sample size?

(login for full post details)
  #1 (permalink)
Yakima WA
 
 
Posts: 28 since Mar 2019
Thanks: 67 given, 20 received

When you are backtesting a strategy, how many trades is a good sample size? How many trades is considered enough to rely on? So far I have backtested 20 trades on a new strategy I am developing. I don't have any fancy backtesting software, I am just manually filling out an excel spreadsheet based on analysis of the chart.
So far, after 20 trades into backtesting, my strategy is profitable, and I am now temped to trade it live with real money, starting small, with just 1 contract in case of failure. My backtesting is complete with stop loss placement so I feel like its ready to go with risk mitigation, but want some input from the community here.

I am planning to keep backtesting my strategy another 10 trades at least but how far back should I go? Is 30 trades enough? Backtesting this way is very time consuming. What do you think?

Reply With Quote

Journal Challenge February 2021 results (so far):
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(151 thanks from 28 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(110 thanks from 31 posts)
looks_3Wisdom is Emptinessby Mtype
(66 thanks from 24 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(31 thanks from 15 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
Gits (Hooglede) Belgium
 
Experience: Master
Platform: NinjaTrader, Proprietary,
Broker: Ninjabrokerage/IQfeed + Synthetic datafeed
Trading: 6A, 6B, 6C, 6E, 6J, 6S, ES, NQ, YM, AEX, CL, NG, ZB, ZN, ZC, ZS, GC
 
rleplae's Avatar
 
Posts: 2,991 since Sep 2013
Thanks: 2,437 given, 5,801 received


what period of time did you test ?

market have various regimes/periods
- calm versus volatile
- trending versus ranging

depending of the type of stratergy and the type of market it best
performs in, you need more or less back testing. you want to be
sure that favorable and less favorable periods were tested. it's
ok, not to make money if the type of market is not good, but you
don't want to go bust during that period, that is what you want
to test..

if your approach is very systematic, then a automation is recommended
i did that for several members and quite often it turns out that a
strategy that looks manually (quite good on excel on paper or screen)
does not do well, when doing systematically...

if it's simple, automate it and run it through a year of data..
it well learn you a lot

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following 4 users say Thank You to rleplae for this post:
 
(login for full post details)
  #4 (permalink)
North Carolina
 
Experience: Beginner
Platform: NinjaTrader, Tradestation
Trading: es
 
Posts: 644 since Nov 2011

This is not the way to backtest. I recommend you get Tradestation. You can also paper trade your idea in Ninjatrader.

There is no hard and fast answer as to how many trades. It will depend on how big the edge is (bigger edge, less trades), how many variables you have (more variables more trades), and considerations around your beliefs or Bayesian reasons as to why a system works. The number 30 I believe comes from statistical studies around confidence interval and if I recall the Chi square distribution.

I would typically look for at least 30 trades per year and 10 years of performance data. But, this may not be the way that every system would be evaluated. With less trades, you would need to go back further in history and if you have more trades you might need less history or if you have Bayesian beliefs then that might influence how far back in history you go, as well.




ClearTrades View Post
When you are backtesting a strategy, how many trades is a good sample size? How many trades is considered enough to rely on? So far I have backtested 20 trades on a new strategy I am developing. I don't have any fancy backtesting software, I am just manually filling out an excel spreadsheet based on analysis of the chart.
So far, after 20 trades into backtesting, my strategy is profitable, and I am now temped to trade it live with real money, starting small, with just 1 contract in case of failure. My backtesting is complete with stop loss placement so I feel like its ready to go with risk mitigation, but want some input from the community here.

I am planning to keep backtesting my strategy another 10 trades at least but how far back should I go? Is 30 trades enough? Backtesting this way is very time consuming. What do you think?


Reply With Quote
 
(login for full post details)
  #5 (permalink)
Yakima WA
 
 
Posts: 28 since Mar 2019
Thanks: 67 given, 20 received

Thanks for the reply.
I am looking at only the 30 minute chart, on CL crude oil.
1 round trip trade per day based and analizing the last 48 hours, with analysis of the same timeframe each time. Trade execution goal is to complete the round-trip trade within about a 12 hour trading window.
So far I have tested the last 20 trading days.
I don't really know if it is possible to automate because I am using subjective/relative analysis of trend and momentum of the last 48 hours to give me my entry signal. I have my criteria but it is not numerical, it is basically current momentum and trend relative to historical momentum and trend.

I guess since oil has been uptrending lately I should test it back when it was downtrending huh?

Reply With Quote
The following user says Thank You to ClearTrades for this post:
 
(login for full post details)
  #6 (permalink)
North Carolina
 
Experience: Beginner
Platform: NinjaTrader, Tradestation
Trading: es
 
Posts: 644 since Nov 2011

@ClearTrades I think, in that case, trading it with Ninjatrader market replay makes sense and paper trading it, i.e. real-time sim account, will be most helpful.

Reply With Quote
 
(login for full post details)
  #7 (permalink)
south africa
 
 
Posts: 169 since Dec 2018

The heuristic of using 30 samples comes from confidence levels between the normal and student t distribution when sampling.
It has absolutely nothing to do with trading or most other real world problems.

IMO you need to think in more of a Bayesian way than in using the tools for sampling from known distributions with all kinds of assumptions on the variable.

If you take 1 sample, then another, then another each new sample is giving you information about the system.

10 samples with 7 winners of course does not tell you this is a "winning" system but it also isn't meaningless.

The main problem though is like rleplae mentioned. If we know anything for certain with trading it is that the distributions will change.

Don't get too hung up on sampling from stale data and last year's distributions. Like right now I am not even sure how much value samples have from a few months ago when fed fund futures had hugely different distributions than today.

Reply With Quote
 
(login for full post details)
  #8 (permalink)
Golden Bay, New Zealand
 
Experience: Beginner
Platform: Sierra Chart
Trading: ES, NQ
 
Posts: 186 since May 2012
Thanks: 851 given, 336 received


ClearTrades View Post
Is 30 trades enough? Backtesting this way is very time consuming. What do you think?

l'll say this nicely.... Don't be lazy.

30 trades is not even in the ballpark. You want well over 100. If you can't be bothered to even do that before throwing money at it, then you will have some very quick and hard lessons with this trading game.

Keep testing for some days. You'll be glad you did, I'll bet this strategy will revert to being no better than random. Strategies that work are not easy to come by. They take a lot of time to find. You will spend a lot of time, which will seem like wasted time as you churn through idea after idea that doesn't work out.

Save your money and your mental capital for those strategies that you have throughly tested out.

Reply With Quote
The following user says Thank You to sixtyseven for this post:
 
(login for full post details)
  #9 (permalink)
Newport Beach
 
Experience: Master
Platform: TradingBlox
Broker: Wisdom Trading
Trading: Global Futures
 
Posts: 3 since Mar 2019
Thanks: 0 given, 0 received

I would recommend thousands of trades in your test sample. Not just one market and not just one time frame.

Reply With Quote
 
(login for full post details)
  #10 (permalink)
Cyprus, Nico
 
 
Posts: 30 since Mar 2019
Thanks: 9 given, 8 received



ClearTrades View Post
When you are backtesting a strategy, how many trades is a good sample size? How many trades is considered enough to rely on? So far I have backtested 20 trades on a new strategy I am developing. I don't have any fancy backtesting software, I am just manually filling out an excel spreadsheet based on analysis of the chart.
So far, after 20 trades into backtesting, my strategy is profitable, and I am now temped to trade it live with real money, starting small, with just 1 contract in case of failure. My backtesting is complete with stop loss placement so I feel like its ready to go with risk mitigation, but want some input from the community here.

I am planning to keep backtesting my strategy another 10 trades at least but how far back should I go? Is 30 trades enough? Backtesting this way is very time consuming. What do you think?

Use only MR(Market replay) method for backtesting for at least 12-months back. This is the only reliable backtest method. Anything else is fake and fails.
Thanks-

Reply With Quote
 
(login for full post details)
  #11 (permalink)
Berlin, Germany
 
 
Posts: 5 since Nov 2019
Thanks: 0 given, 2 received

I backtest for much longer and try to always incorporate High Volatility time periods. There are some excellent platforms for this that use simple programming languages for backtesting rules likeTradestation, Multicharts.. I use TradeNavigator.

Reply With Quote
 
(login for full post details)
  #12 (permalink)
Cyprus, Nico
 
 
Posts: 30 since Mar 2019
Thanks: 9 given, 8 received


Hello1 View Post
I backtest for much longer and try to always incorporate High Volatility time periods. There are some excellent platforms for this that use simple programming languages for backtesting rules likeTradestation, Multicharts.. I use TradeNavigator.

Correct/precise backtesting is crucial when testing/creating a strategy. As I said before, market replay is the best backtesting method (regardless of your COBC is true or false). However, if you work on NT7 (or maybe on other platforms) and you dont have as an Add-on the market replay s/w (or if u dont want to buy it) then you can use the existing NT7's backtesting s/w based on 2 assumptions : a) your COBC should be true, and b) the type of Bars you use MUST show the exact market's move. The latter means that e.g sipmle Renko bars are not suirable for backtesting. You have to use "wicked" type Bars or Range Bars or other special type Bars. For example, I attach herewith the backtest results using the common NT7's existing s/w of one of my strategies using special type "wicked" Bars (with my COBC = true on my code, coded by me). These results are almost the same when compared with testing results of the same strategy when using market replay s/w. This proves the reliability of these results. Once again, this is only a real example. I hope this helps. Thank you,-

https://futures.io/attachment.php?attachmentid=286873&stc=1&d=1581587010

Reply With Quote
 
(login for full post details)
  #13 (permalink)
Chicago, Illinois
 
Experience: Intermediate
Platform: Tradingview
Broker: ThinkOrSwim
Trading: Forex, Stock & Options
 
Bermudan Option's Avatar
 
Posts: 554 since May 2011
Thanks: 612 given, 343 received

I stumbled across this post recently and I think it might help

https://www.dara.trade/blog/2019/10/14/how-to-build-a-profitable-trading-system-part-1-confidence-in-numbers

Visit my futures io Trade Journal Reply With Quote
The following 3 users say Thank You to Bermudan Option for this post:


futures io Trading Community Traders Hideout Commodities > Backtesting: What is a good sample size?


Last Updated on December 5, 2020


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1500 prizes from Topstep!

February
 

Identifying Setups & Targets Using Profile Charts w/Trevor & Tradovate

Feb 25
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts