How can I tell I my system will work in the Future - futures io
futures io futures trading



How can I tell I my system will work in the Future


Discussion in NinjaTrader

Updated
      Top Posters
    1. looks_one wallsteetking with 5 posts (0 thanks)
    2. looks_two ThatManFromTexas with 4 posts (1 thanks)
    3. looks_3 RM99 with 4 posts (2 thanks)
    4. looks_4 mainstream with 3 posts (0 thanks)
      Best Posters
    1. looks_one Fat Tails with 2 thanks per post
    2. looks_two Eric j with 2 thanks per post
    3. looks_3 fluxsmith with 1.5 thanks per post
    4. looks_4 RM99 with 0.5 thanks per post
    1. trending_up 5,320 views
    2. thumb_up 16 thanks given
    3. group 11 followers
    1. forum 32 posts
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

How can I tell I my system will work in the Future

(login for full post details)
  #1 (permalink)
MN
 
 
Posts: 25 since Feb 2011
Thanks: 4 given, 0 received

I have developed a trading system that did very well from 2007-2011. total of 700 trades taken. however, I feel that this does not tell me the whole story. I feel that I need at least then years back data for my trading system to see if it really works. However, because its an intra day strategy, I only got 4 years back data. Is there a way to tell if this strategy will work in the future or was this just luck.

Reply With Quote

Journal Challenge February 2021 results (so far):
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(151 thanks from 28 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(110 thanks from 31 posts)
looks_3Wisdom is Emptinessby Mtype
(66 thanks from 25 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(31 thanks from 15 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
Santa Maria
 
Experience: Advanced
Platform: NinjaTrader, ThinkOrSwim
Broker: Mirus/Zen-Fire
Trading: ES
 
Posts: 295 since May 2010
Thanks: 97 given, 321 received



wallsteetking View Post
I have developed a trading system that did very well from 2007-2011. total of 700 trades taken. however, I feel that this does not tell me the whole story. I feel that I need at least then years back data for my trading system to see if it really works. However, because its an intra day strategy, I only got 4 years back data. Is there a way to tell if this strategy will work in the future or was this just luck.

The only way you can tell if it will work in the future is to trade it in the future. But, since that's a fairly useless answer, there are characteristics which give us clues:

1. Profit factor
2. Sharpe ratio
3. SQN
4. Percent wins
5. Expectancy
6. Max drawdown
7. Average trade, winner, looser
8. Is the equity curve smooth or jagged?

All of the above have been thoroughly discussed here on Big Mike's. There is one more I like to see which is not discussed much. If the system takes both longs and shorts, then:

9. Is long and short profitability fairly evenly balanced?

For (9) I use this formula, 1 is a perfect score:
min(long net, short net) / (long net + short net) / 0.5

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to fluxsmith for this post:
 
(login for full post details)
  #4 (permalink)
NY
 
Experience: Advanced
Platform: Esignal, MBT navigator
Trading: Currencies
 
Posts: 2,526 since Jun 2009
Thanks: 1,396 given, 2,576 received

Forward testing your strategy through enough weeks to trade in at least one each of a trending , ranging and congested market is the only place to start . Backtests are fraught with inconsistencies that could take more (valuable) time to unravel than to forward test in a live market on a simulator .

I and most others here have had great tests that flopped hard in the live market . Dunno exactly why but its true , so invest your time in testing it in hand to hand combat long enough to shake out the bugs .

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #5 (permalink)
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,855 since Mar 2010
Thanks: 4,238 given, 26,726 received

If you develop a trading system it is always biased towards past data, and there is no guarantee that it will work in future.

To avoid curve fitting, you should only use a part of the past data to select and optimize the parameters. You can then use the remaining data for a walk forward analysis and Monte Carlo simulation. This will at least tell you whether your system is curve fitted or not. It does not tell you of course, whether the market conditions in the future changes. Is this what you did? Then it should give you some comfort.

2007-2011 is not a bad base for testing a system, because it includes both a bull and a bear market, so it should not be biased towards long or short trades.

Reply With Quote
The following 2 users say Thank You to Fat Tails for this post:
 
(login for full post details)
  #6 (permalink)
MN
 
 
Posts: 25 since Feb 2011
Thanks: 4 given, 0 received


Reply With Quote
 
(login for full post details)
  #7 (permalink)
MN
 
 
Posts: 25 since Feb 2011
Thanks: 4 given, 0 received

Hi how long should I test it on live data before starting with real money???

Reply With Quote
 
(login for full post details)
  #8 (permalink)
Houston,Tx
 
Experience: Advanced
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: TF
 
ThatManFromTexas's Avatar
 
Posts: 2,302 since Feb 2010
Thanks: 1,206 given, 4,332 received

I have never found a correlation between backtesting and actual results.

I'm just a simple man trading a simple plan.

My daddy always said, "Every day above ground is a good day!"
Reply With Quote
The following user says Thank You to ThatManFromTexas for this post:
 
(login for full post details)
  #9 (permalink)
Chicago, IL
 
Experience: Master
Platform: Kinetick Ninja Trader <7>
Broker: Ninja & IB
Trading: YM & Equities & Options & Mutual Funds
 
Posts: 175 since Jun 2010
Thanks: 56 given, 87 received

If you have good back tested results, turn the system on live and watch the system trade. Is it following your rules, on entries and exits?

When you're comfortable take it live... with a finger on the quick close in the event your stop criteria doesn't execute as planned.

Once you gotten through that stage set it and forget it... you might also choose a broker that will allow for a max daily loss restriction for double fail safe.

My question to you is how many instruments does the system work on?

Reply With Quote
 
(login for full post details)
  #10 (permalink)
Santa Maria
 
Experience: Advanced
Platform: NinjaTrader, ThinkOrSwim
Broker: Mirus/Zen-Fire
Trading: ES
 
Posts: 295 since May 2010
Thanks: 97 given, 321 received



ThatManFromTexas View Post
I have never found a correlation between backtesting and actual results.

Some version of this is a frequent theme here. I think you could be saying any combination of 3 things:

1. Historical data feeds don't match real-time data feeds.
This is true, but they are certainly very highly correlated

2. The past is not predictive of the future
This may be true, but if it had no predictive value at all then no trading system would work

3. The systems you've tried execute differently over historical data vs real-time data
This is true for NT by default, but it can be largely overcome

So... I find backtesting to be highly correlated with actual results. In fact, for each system I trade, I check daily after the close that the backtesting engine gives the same performance I experienced in real-time. If there is any difference at all which is not explainable by slippage, then I treat it as a programming error and seek to correct it.

Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to fluxsmith for this post:
 
(login for full post details)
  #11 (permalink)
Marietta, GA U.S.A
 
 
Posts: 9 since Mar 2011
Thanks: 0 given, 4 received

Your report doesn't appear to include slippage and commissions. I have no idea what (or how much of it) you are trading, but that can quickly turn a strategy negative.

The only way to get and idea is to do in sample and out of sample testing (forward testing) as was talked about above. For example, take 6 months from 2010 to optimize/test you strategy, then see if it worked up until now. That's replicating what you are trying to do. Write something in the past and have it work in the future.

I would then see if it worked in the data you have prior to that period. If not, then you probably need to check your code to make sure there aren't any constants in it, or at least determine if mitigating factor were an issue (the crash, etc).

I've read where people sim test for a year, but I only do so for a week measuring the expected buy and sell points. That's long enough for me to get confortable that the code works exactly like I want. In my opinion, It'll either work or it won't work. If I sim test for a month and its negative, does that invalidate my testing or was it a bad month? I just turn it on, monitor it constantly, and make sure that the percentages and averages I see in the backtest closely mirror the real time testing.

Also, I figure this is your first strategy so here are some things to look out for. I trade in TradeStation, but I know there are issues with limit orders (you may not get filled) and advanced bar types (buy on close instead of next bar). I'm not sure if they exist in Ninja, but before going live, make sure you research and understand any problems that may cause.

Reply With Quote
The following user says Thank You to shadrock55 for this post:
 
(login for full post details)
  #12 (permalink)
MN
 
 
Posts: 25 since Feb 2011
Thanks: 4 given, 0 received


mainstream View Post
If you have good back tested results, turn the system on live and watch the system trade. Is it following your rules, on entries and exits?

When you're comfortable take it live... with a finger on the quick close in the event your stop criteria doesn't execute as planned.

Once you gotten through that stage set it and forget it... you might also choose a broker that will allow for a max daily loss restriction for double fail safe.

My question to you is how many instruments does the system work on?


It only works on Crude Oil. but if I trade it on Euro my profit would of been 6% for years

Reply With Quote
 
(login for full post details)
  #13 (permalink)
Austin, TX
 
Experience: Advanced
Platform: TradeStation
Trading: Futures
 
Posts: 882 since Mar 2011
Thanks: 124 given, 701 received

A) There's nothing that can "tell" if your system will work in the future....it's only relative and a sliding scale, some things give more confidence than others.

B) Forward testing will not indicate if your system will fair well in the future either as there logic challenges with simulation as well (fill orders, slippage, adverse market effects, etc).

C) Obviously simulation will help to iron out bugs with algo guys....but another good reason to sim or forward test is so that you can do a comparison/normalization to your backtest.

I do a forward sim test and then compare that result with the same time period doing a backtest. This will give me an indication of how the back and forward tests compare.

If the backtest is promising and the forward test is promising and the comparison between the 2 is very close, then that's as good as you can do going forward before taking it live. Market conditions can and do change.

Some strategies fair better in trending markets, some in sideways markets, so extended periods of either can make your strategy results seem better or worse.

D) Using an appropriate amount of data to backtest is just as important as the backtest itself. You can actually use too much data/time rearward. You have to carefully breakdown the results to ensure that if the market changed last year (against you) but the backtest was conducted over a 4 year period and seems profitable, then you might be given false confidence taking it live.

E) There are simply some things that become so complicated, you have to employ a "safety factor." I do this for just about everything. Before I put something live in play, I multiply the worst drawdown by 2. I cut the return by 4. I then look at the sheer number of trades and if it's a nibbler (lots of small trades) I try to envison what would happen if it traded half as often (dry spells for entries)....I then apply a very CONSERVATIVE estimate for slippage and I apply the maximum for commission (even though I know my commissions shrink with increased trade volume). IF, and only if it still makes money forward and backward, do I feel confident enough to play it live.

For example....when you backtest CL, you have to use assumption based data like @CL, and anyone trading futures will tell you, there's always a transition volume at the turn of the contract, it's very difficult to replicate how you will approach that....will you split your trades (between May and June), will you transition at the same time each month? Will you employ a rule to switch once the newer contract has more volume than the current? Those are the sort of things that are just very difficult to wrap your arms around and continuous backtest data can't account for the effect without some very sophisticated coding. THAT is why I overengineer my strategies, because I'd rather shoot for the stars, that way, if I fall short, at least I land on the moon.

Reply With Quote
 
(login for full post details)
  #14 (permalink)
Chicago, IL
 
 
Posts: 29 since Mar 2011
Thanks: 7 given, 7 received

700 trades spread over 4 years? Get a statistician to compare your results with 700 random trades and check the difference. If your system is good there will be a difference.

Reply With Quote
 
(login for full post details)
  #15 (permalink)
Chicago, IL
 
Experience: Master
Platform: Kinetick Ninja Trader <7>
Broker: Ninja & IB
Trading: YM & Equities & Options & Mutual Funds
 
Posts: 175 since Jun 2010
Thanks: 56 given, 87 received

Some great points! I will be integrating them!


I am definitely scared of CL systems... the fill is just so iffy. Especially when that market starts moving, your slippage could be ten ticks.

Reply With Quote
 
(login for full post details)
  #16 (permalink)
Houston,Tx
 
Experience: Advanced
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: TF
 
ThatManFromTexas's Avatar
 
Posts: 2,302 since Feb 2010
Thanks: 1,206 given, 4,332 received


fluxsmith View Post
Some version of this is a frequent theme here. I think you could be saying any combination of 3 things:

1. Historical data feeds don't match real-time data feeds.
This is true, but they are certainly very highly correlated


2. The past is not predictive of the future
This may be true, but if it had no predictive value at all then no trading system would work

A system will work or not regardless of predictions

3. The systems you've tried execute differently over historical data vs real-time data
This is true for NT by default, but it can be largely overcome

So... I find backtesting to be highly correlated with actual results. In fact, for each system I trade, I check daily after the close that the backtesting engine gives the same performance I experienced in real-time. If there is any difference at all which is not explainable by slippage, then I treat it as a programming error and seek to correct it.

Actually I was saying one thing; I have never found a correlation between a system's back test results and it's actual results.

I'm just a simple man trading a simple plan.

My daddy always said, "Every day above ground is a good day!"
Reply With Quote
 
(login for full post details)
  #17 (permalink)
Malaysia
 
Experience: Beginner
Platform: Multichart DT
Broker: MB Trading
Trading: Fx
 
torroray's Avatar
 
Posts: 273 since Nov 2009
Thanks: 52 given, 61 received


Eric j View Post
Forward testing your strategy through enough weeks to trade in at least one each of a trending , ranging and congested market is the only place to start . Backtests are fraught with inconsistencies that could take more (valuable) time to unravel than to forward test in a live market on a simulator .

I and most others here have had great tests that flopped hard in the live market . Dunno exactly why but its true , so invest your time in testing it in hand to hand combat long enough to shake out the bugs .

I used to go thru almost every forum to find strategy and test them. I found that same conclusion that what was claimed to be great, fall short in live market. But after years of looking at many stategy I learned something from years of system evaluation. At a glance I can tell that such system work on certain market condition (trending, ranging, congestion) but not all at the same time.

Its like a racing car that have to be adjusted to the race track ( not the Nascar oval). The problem with trading system, the market is the unknown factor so adjusting to is difficult. Some tried and call them trading system but they are more prediction or what if trading system.

Reply With Quote
The following user says Thank You to torroray for this post:
 
(login for full post details)
  #18 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,742 since Jun 2009
Thanks: 32,288 given, 97,473 received


wallsteetking View Post
I have developed a trading system that did very well from 2007-2011. total of 700 trades taken. however, I feel that this does not tell me the whole story. I feel that I need at least then years back data for my trading system to see if it really works. However, because its an intra day strategy, I only got 4 years back data. Is there a way to tell if this strategy will work in the future or was this just luck.

Did you disclose the bar type and period? Exotic bars are not reliable for backtesting, and large bars (ie: 60 min) will produce inaccurate results due to NT backtest engine OHLC ordering.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #19 (permalink)
Stockholm
 
Experience: Intermediate
Platform: Nanotrader/ tradestaion
Broker: WHS / TS
Trading: EC / SI / ES / BP
 
Posts: 19 since Mar 2011
Thanks: 8 given, 5 received


wallsteetking View Post
I have developed a trading system that did very well from 2007-2011. total of 700 trades taken. however, I feel that this does not tell me the whole story. I feel that I need at least then years back data for my trading system to see if it really works. However, because its an intra day strategy, I only got 4 years back data. Is there a way to tell if this strategy will work in the future or was this just luck.

Always test your strategy with real money before you can say it works or not!
I use a autoprogram on Silver that gives me a minimum profit of 10k a month,
but it has cost 25k for the developper before it was really working as it should!
backtesting or testing in simulated trading did'nt give the real results!

Lamboo

Reply With Quote
 
(login for full post details)
  #20 (permalink)
Chicago, IL
 
Experience: Master
Platform: Kinetick Ninja Trader <7>
Broker: Ninja & IB
Trading: YM & Equities & Options & Mutual Funds
 
Posts: 175 since Jun 2010
Thanks: 56 given, 87 received

I think exotic instruments are the problem....


If the bar isn't full of ticks then you're going to have room for error. An instrument that is very liquid and doesn't jump 5 or six ticks in a second, because there is a deep depth of market is going to be more accurate.

Reply With Quote
 
(login for full post details)
  #21 (permalink)
Houston,Tx
 
Experience: Advanced
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: TF
 
ThatManFromTexas's Avatar
 
Posts: 2,302 since Feb 2010
Thanks: 1,206 given, 4,332 received


ThatManFromTexas View Post
I have never found a correlation between backtesting and actual results.


fluxsmith View Post
Some version of this is a frequent theme here. I think you could be saying any combination of 3 things:

1. Historical data feeds don't match real-time data feeds.
This is true, but they are certainly very highly correlated

2. The past is not predictive of the future
This may be true, but if it had no predictive value at all then no trading system would work

3. The systems you've tried execute differently over historical data vs real-time data
This is true for NT by default, but it can be largely overcome

So... I find backtesting to be highly correlated with actual results. In fact, for each system I trade, I check daily after the close that the backtesting engine gives the same performance I experienced in real-time. If there is any difference at all which is not explainable by slippage, then I treat it as a programming error and seek to correct it.


ThatManFromTexas View Post
Actually I was saying one thing; I have never found a correlation between a system's back test results and it's actual results.


Lamboo View Post
Always test your strategy with real money before you can say it works or not!
I use a autoprogram on Silver that gives me a minimum profit of 10k a month,
but it has cost 25k for the developper before it was really working as it should!
backtesting or testing in simulated trading did'nt give the real results!

Lamboo

My point exactly.

I'm just a simple man trading a simple plan.

My daddy always said, "Every day above ground is a good day!"
Reply With Quote
 
(login for full post details)
  #22 (permalink)
MN
 
 
Posts: 25 since Feb 2011
Thanks: 4 given, 0 received


Big Mike View Post
Did you disclose the bar type and period? Exotic bars are not reliable for backtesting, and large bars (ie: 60 min) will produce inaccurate results due to NT backtest engine OHLC ordering.

Mike

I just used Candle charts with 180 minute data
because this is a large bar how would I overcome the issue of ninja backtesting

Reply With Quote
 
(login for full post details)
  #23 (permalink)
Houston,Tx
 
Experience: Advanced
Platform: NinjaTrader
Broker: Mirus Futures/Zen-Fire
Trading: TF
 
ThatManFromTexas's Avatar
 
Posts: 2,302 since Feb 2010
Thanks: 1,206 given, 4,332 received


wallsteetking View Post
I just used Candle charts with 180 minute data
because this is a large bar how would I overcome the issue of ninja backtesting

To sleep, perchance to dream – ay, there's the rub:
For in that sleep of death what dreams may come, ... Hamlet Act 3 Scene 1

I'm just a simple man trading a simple plan.

My daddy always said, "Every day above ground is a good day!"
Reply With Quote
 
(login for full post details)
  #24 (permalink)
Market Wizard
Berlin, Europe
 
Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker: Interactive Brokers
Trading: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,855 since Mar 2010
Thanks: 4,238 given, 26,726 received

Our attitudes are formed by beliefs.

If you believe that backtesting does not work, it will not work for you.

If you do not believe in anything, you might discover something.

Economists believe in their theories.

For this reason their theories have never been backtested.

Reply With Quote
The following 2 users say Thank You to Fat Tails for this post:
 
(login for full post details)
  #25 (permalink)
Cancun, Mexico
 
Experience: Advanced
Trading: Stock Index Futures / CFDs
 
Anagami's Avatar
 
Posts: 865 since Dec 2010
Thanks: 603 given, 1,951 received


Fat Tails View Post
Our attitudes are formed by beliefs.

If you believe that backtesting does not work, it will not work for you.

If you do not believe in anything, you might discover something.

Economists believe in their theories.

For this reason their theories have never been backtested.

One might even choose to experience Freedom from the Known... at least on occasion.

"The mind is its own place, and in itself can make a heaven of hell, a hell of heaven." - Milton
Reply With Quote
The following user says Thank You to Anagami for this post:
 
(login for full post details)
  #26 (permalink)
Site Administrator
Swing Trader
Data Scientist & DevOps
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 49,742 since Jun 2009
Thanks: 32,288 given, 97,473 received


wallsteetking View Post
I just used Candle charts with 180 minute data
because this is a large bar how would I overcome the issue of ninja backtesting

You have to add a second dataseries, like 1 range, and execute orders on it.

A quick way to see how much this impacts you is to sort the trade performance results by # of bars. All exits on bar #1 should be assumed to be stops, not winners.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #27 (permalink)
Austin, TX
 
Experience: Advanced
Platform: TradeStation
Trading: Futures
 
Posts: 882 since Mar 2011
Thanks: 124 given, 701 received


Big Mike View Post
You have to add a second dataseries, like 1 range, and execute orders on it.

A quick way to see how much this impacts you is to sort the trade performance results by # of bars. All exits on bar #1 should be assumed to be stops, not winners.

Mike

In addition, the way you can overcome OHLC disparities is to drop to smaller bars altogether or to increase the P/L amounts to the point where the OHLC becomes less significant.

If your P/L is much larger than your ATR for a 180 bar, then the odds of it entering and exiting on the same bar are much less.

It would be interesting for someone to do a study to determine the rate of outliers for OHLC on different bar lengths. Particularly with respect to candle length, wick length, overall range, etc, etc.

I find it's much more rare for a bar inversion on a 60 minute bar than say a 5 minute bar during high traffic.

Generally speaking, a "red" bar (close < open) tends to OHLC, whereas a "green" bar (close > open) tends to OLHC, but as Mike pointed out, not always.

I'm wondering if the exceptions are MORE for lesser bars, it's just that you don't notice it as much because the P/L is larger than the typical range for the bars. I see a lot more "flip flops" with 5 minute bars than I do with a 60 minute bar (assuming the bars have relatively "average" range.). Obviously for smaller range bars, much more is possible during the given time period.

From this site, I was under the impression that Ninja was able to execute tick/tick backtesting which would eliminate the OHLC issue.

Reply With Quote
 
(login for full post details)
  #28 (permalink)
Barcelona, Spain
 
Experience: Intermediate
Platform: NinjaTrader 7
Broker: Interactive Brokers/Kinetick
Trading: US stocks, EUR/USD
 
BCNTrader's Avatar
 
Posts: 119 since Sep 2010
Thanks: 138 given, 108 received

How do you set your stop and target?

Don't know what kind of entry (market/limit) do you use in this strategy but the most common issue is that ppl set the stop according the entry they get (based on pips, ticks or dollars at risk) and you never know it on realtime. You must use a stop/target based on the previous close bar [0] if you're using a market order entry or ask/bid limit order. At least, this way you'll get the same exact exit on each trade (stop/target hit) but you'll never know the exact entry in market orders or you can lose opportunities on limit orders (you'll only notice this comparing real vs backtesting). The same is true if you trail based on dollar amount or ticks/pips.

Forget this advise if you set both with technical info (past day low, sma, etc). Just own experience.

* Not native english, any correction is well appreciated.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #29 (permalink)
Austin, TX
 
Experience: Advanced
Platform: TradeStation
Trading: Futures
 
Posts: 882 since Mar 2011
Thanks: 124 given, 701 received


BCNTrader View Post
How do you set your stop and target?

Don't know what kind of entry (market/limit) do you use in this strategy but the most common issue is that ppl set the stop according the entry they get (based on pips, ticks or dollars at risk) and you never know it on realtime. You must use a stop/target based on the previous close bar [0] if you're using a market order entry or ask/bid limit order. At least, this way you'll get the same exact exit on each trade (stop/target hit) but you'll never know the exact entry in market orders or you can lose opportunities on limit orders (you'll only notice this comparing real vs backtesting). The same is true if you trail based on dollar amount or ticks/pips.

Forget this advise if you set both with technical info (past day low, sma, etc). Just own experience.

* Not native english, any correction is well appreciated.

Technically, a backtest has a degree of inaccuracy regardless of your entry/exit criteria.

If you're trading market orders, the backtest cannot account for adverse market effects, which become increasingly more for larger orders.

If you're trading limit orders, the backtest must assume a total fill. You can set the backtest to only fill the order when price has moved beyond the limit price, however, again, this is an assumption and does not account for adverse effects (the amount of resistance/support your order created).

Regardless of how you enter or exit the trades, a backtest cannot anticipate adverse market effects. Depending on your entry side and the type of strategy, these effects can have varying degrees.

For example, if you have a "top side" entry strategy and use limit orders, the backtest might assume a sellshort fill, once the price exceeded your limit price by 1 tick, negating the concept that your limit may not have been filled totally if it had actually been in the marketplace.

Using the same example with a market order creates dififculties, as the engine won't consider the market order executed until it observes an equal number of contracts in the proper direction (which does not account for the adverse effects those shares/contracts would have had in providing momentum in that direction.)

Reply With Quote
The following 2 users say Thank You to RM99 for this post:
 
(login for full post details)
  #30 (permalink)
Barcelona, Spain
 
Experience: Intermediate
Platform: NinjaTrader 7
Broker: Interactive Brokers/Kinetick
Trading: US stocks, EUR/USD
 
BCNTrader's Avatar
 
Posts: 119 since Sep 2010
Thanks: 138 given, 108 received


RM99 View Post
Technically, a backtest has a degree of inaccuracy regardless of your entry/exit criteria.

If you're trading market orders, the backtest cannot account for adverse market effects, which become increasingly more for larger orders.

If you're trading limit orders, the backtest must assume a total fill. You can set the backtest to only fill the order when price has moved beyond the limit price, however, again, this is an assumption and does not account for adverse effects (the amount of resistance/support your order created).

Regardless of how you enter or exit the trades, a backtest cannot anticipate adverse market effects. Depending on your entry side and the type of strategy, these effects can have varying degrees.

For example, if you have a "top side" entry strategy and use limit orders, the backtest might assume a sellshort fill, once the price exceeded your limit price by 1 tick, negating the concept that your limit may not have been filled totally if it had actually been in the marketplace.

Using the same example with a market order creates dififculties, as the engine won't consider the market order executed until it observes an equal number of contracts in the proper direction (which does not account for the adverse effects those shares/contracts would have had in providing momentum in that direction.)

I totally agree with you but I'm talking about another issue. If you use market orders "on last bar close" strategies and you want it to have the same stop/entry in backtesting (and the same trail if you say "if goes X pips move to break even") you must use the information you have on the chart and not ticks/pips/dollars because your entry. This way, at least when comparing real trades and backtest trades (the same trades) you'll see that you had had the same stops and targets -real vs backtest- (don't matter if you had a fill on your target. You used exactly what your strategy is supposed to use), btw it depends a lot of each strategy and I assume that scalping strategies don't use market orders but limit orders. Aaand It can kill you a lot of profits or put you on a large stop situation if the market is moving a lot.

I totally agree... If you use limit orders, you'll lose entries even if the price went beyond your order, even more if you trade multiple contracts.

I agree too that the number of contracts you're trading can move the market against you.

And I totally totally agree with ThatManFromTexas:

ThatManFromTexas View Post
I have never found a correlation between backtesting and actual results.

At least, if you believe in your strategy and want to trade it you must understand it and play according to it, that's why I never use automated trades but I trade manually what my strategies say, using the exact same price to say OK, this is not doing what I was expecting for, and the same price to say OK, this is where I expected it to go. This way, I know where these points are at the present time (while on a trade) and I know that these points will be the same in backtesting (backtesting the same trade in the future).

Can be very frustrating to refresh your chart, or backtest the day you traded for real and see that for just 2 pips (don't matter which way +2 or -2, think about it) you had your stop or your target in a different place. Again, if you believe on it and use market orders, you must use chart info for your stop, target and trails (if any), by the other hand if you use limit orders expect to loose some entries. This is only valid for good risk/reward ratio trades where you backtested with market orders.

* I use limit orders for my real trades.

Don't get me wrong, my backtesting is highly correlated with my actual results BUT not my entries, sometimes slippage, sometimes I even have a better entry.

I think that you will never get the same results but you can play with it if you understand the strategy in your mind (you can explain it in simple words or even your grandmother can get it ).

Expect the worst case scenario per trade, if your backtesting don't even work when you include slippage and commissions on it, don't expect it to work.

In plain words, expect:
- Slippage.
- Real market is moving and sometimes is really fast.
- Price can go beyond your stop.
- Use commissions.
- In order for you to buy a contract, somebody has to sell a contract to you.



* I think I wrote far beyond my english skills allow to me, sorry if any mistakes. Any correction will be highly appreciated.

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to BCNTrader for this post:
 
(login for full post details)
  #31 (permalink)
Asia
 
Experience: Beginner
Platform: NinjaTrader, TOS
 
Posts: 798 since Jun 2009
Thanks: 109 given, 799 received

Back to the OP's question, one thing to think about is how to define if a strategy is working or not. It needs to be a binary answer (yes or no) that has a bit more behind it than whether you are making money or losing money.

If you have fully back- and forward-tested an idea, you should have a few datapoints that you can use to evaluate the strategy when it is live. This is an area I'm still working on, but most of the criterea I'm using are based on deviations in P&L from model vs. actual, and deviations in the volatility of returns.

Those deviations need to be monitored both on the positive side and the negative side. If the system is doing significantly better (or worse) than the model, it would indicate I've missed something and need to be very careful. While it is a lot easier to pull the plug on a system that is losing money than one that is making money, if you are deviating more than n%/period from your model, your model is wrong and you need to find out why.

The challenge (for me at least) is to work out parameters that indicate All Clear, Caution, and Stop and apply them objectively. Personally I've found it helpful to accept that all systems will fail at some point, and part of the design process therefore needs to include ways to recognize and mitigate that failure.

My 2 pesos.

Reply With Quote
 
(login for full post details)
  #32 (permalink)
Austin, TX
 
Experience: Advanced
Platform: TradeStation
Trading: Futures
 
Posts: 882 since Mar 2011
Thanks: 124 given, 701 received

As I said earlier, a backtest can provide useful information if paired with forward/sim test data.

If you're able to backtest and get a promising result, with respect to any parameter that interests you, say win rate. (assume a win rate of 67%) over a backtest period of 18 months.

If you then forward test/sim for 3 months and you observe a simulated win rate of 63%, you then conduct a backtest over the same period you just sim'ed and you get a return rate of 65%, you can then start to draw some linear interpretations of the relationship between your backtest and forward test.

It's indicating that your forward test has a variance of about -4% with respect to long term backtest and -2% with respect to in kind back/forward tests.

You can do the same thing for drawdown, profit ratio, etc, etc.

However, as I stated earlier, there's always a "fudge" factor that must be employed do to the inherent limitations for simulation and backtesting.

This "fudge" or safety factor will depend on your own tolerance taste and your observed data for live trading.

If say for instance, you were really thorough (like some people) and you paid to have 2 different data streams and 2 different accounts, that way you could live trade and sim trade at the same time, then you could get all three, back, forward and live, and compare all of them to develop your own factor to normalize or compare the various methods for results.

Reply With Quote
 
(login for full post details)
  #33 (permalink)
London, UK
 
Experience: Beginner
Platform: NinjaTrader, home-grown Java
Broker: IB/IQFeed
Trading: EUR/USD
 
Adamus's Avatar
 
Posts: 1,085 since Dec 2010
Thanks: 471 given, 786 received

I think I am missing the terminology here, or perhaps you're just not using a well-defined terminology, because I think there are 2 seperate issues here:

1) the performance of your system in live or sim-account real-time vs the performance in backtest against data collected over that same period

2) the performance of your back test over one period against which you build and possibly optimise your system vs the performance forward tested over another period vs the performance traded real-time over yet another period.

Number (1) is something you can control by throwing resources at it to get the best data that is closest to the real-time data that streamed to the system during live trading

Number (2) is something that is in the lap of the gods, and the only control you have is to make sure you build a robust system that is not curve fit, e.g. a long term trend following system such as the Turtles used.

What terms would you use to refer to (1) and (2) ?

You can discover what your enemy fears most by observing the means he uses to frighten you.
Follow me on Twitter Visit my futures io Trade Journal Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > How can I tell I my system will work in the Future


Last Updated on April 13, 2011


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1500 prizes from Topstep!

February
 

Identifying Setups & Targets Using Profile Charts w/Trevor & Tradovate

Feb 25
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts