Trouble with LIVE v. BACKTEST - NinjaTrader | futures io social day trading
futures io futures trading


Trouble with LIVE v. BACKTEST
Updated: Views / Replies:5,455 / 18
Created: by patbateman Attachments:9

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 9  
 
Thread Tools Search this Thread
 

Trouble with LIVE v. BACKTEST

  #1 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received

Trouble with LIVE v. BACKTEST

Hello all,

I put a new system of mine (60 minute chart, non intraday) through market replay after testing it via strategy analyzer, and had differing results.

I undertand this is not uncommon with Ninjatrader crapware, however generally backtesting minute bars with 1 tick of slippage is accurate enough.

What is really confusing is that the results of replay (live) versus backtesting are the same, minus the number of trades. The profit and consistency rates are the same, however the live trading is generating fewer trades. In fact, the live trading's overall results are better; generally it's the other way around

I'll go back to the charts and analyze each individual trades, but I was wondering if anyone has any thoughts. Here are some points on what I'm doing:
  • Using non-blackboxed Jurik tools (i.e. 100% ninjascript indicators) from jurikres.com - I've used them before in backtest versus live with no trouble
  • 60 minute bars, exit on close = false
  • Trading the ES
  • Uses a leading indicator (made this week)
  • Completely independent from other platforms, i.e. matlab is not connected

As a final bit, I only optimized three coefficients/inputs. All but three of the combinations had positive results... very confident that the system works. I'm going to investigate if there is a discrepancy between my Ninjascript in backtest versus live. If NT developers ask for the system... the answer is no, and you'd probably blame me anyways if I did.


Does anyone know of alternative software platforms that will run C# (NOT easylanguage) with similar functionality as NT? I've heard Xtrader and T4 are pretty popular... I got into NT because of recommendations and liked the up sides of it very much, at least at the time.

"A Jedi's strength flows from the force."
-Yoda
Attached Thumbnails
Trouble with LIVE v. BACKTEST-market-replay.jpg   Trouble with LIVE v. BACKTEST-backtest.jpg  

Last edited by patbateman; April 28th, 2012 at 06:19 PM. Reason: error
Reply With Quote
 
  #2 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received

Here's what I'm going to do in the next hour to try and solve this...
  1. Retest the market replay and see if I get differing results
  2. Analyze the market replay's indicator output versus static versus backtest output
  3. Compare if the trades in market replay are at the same times as backtest
  4. Try another system, even more basic, even though I'm certain my algos are NOT the problem
  5. Compare my minute data (paid for) to market replay (from NT7) - Market replay has worked well for me in the past with live versus replay.
  6. Any other ideas?






"A Jedi's strength flows from the force."
-Yoda
Reply With Quote
 
  #3 (permalink)
 Vendor: www.tools4trading.de 
Erfurt Germany
 
Futures Experience: Master
Platform: rtMagic, NinjaTrader
Broker/Data: AMP/CQG, IAB
Favorite Futures: ES, 6E, FDAX, FGBL
 
Posts: 338 since Aug 2011
Thanks: 54 given, 460 received



patbateman View Post
Hello all,

I put a new system of mine (60 minute chart, non intraday) through market replay after testing it via strategy analyzer, and had differing results.

I undertand this is not uncommon with Ninjatrader crapware, however generally backtesting minute bars with 1 tick of slippage is accurate enough.

What is really confusing is that the results of replay (live) versus backtesting are the same, minus the number of trades. The profit and consistency rates are the same, however the live trading is generating fewer trades. In fact, the live trading's overall results are better; generally it's the other way around

I'll go back to the charts and analyze each individual trades, but I was wondering if anyone has any thoughts. Here are some points on what I'm doing:
  • Using non-blackboxed Jurik tools (i.e. 100% ninjascript indicators) from jurikres.com - I've used them before in backtest versus live with no trouble
  • 60 minute bars, exit on close = false
  • Trading the ES
  • Uses a leading indicator (made this week)
  • Completely independent from other platforms, i.e. matlab is not connected

As a final bit, I only optimized three coefficients/inputs. All but three of the combinations had positive results... very confident that the system works. I'm going to investigate if there is a discrepancy between my Ninjascript in backtest versus live. If NT developers ask for the system... the answer is no, and you'd probably blame me anyways if I did.


Does anyone know of alternative software platforms that will run C# (NOT easylanguage) with similar functionality as NT? I've heard Xtrader, T4 and TT are pretty popular... I got into NT because of recommendations and liked the up sides of it very much, at least at the time.

please say your used ordertypes... Without self coding, NT7 don't can backtest stop&limit orders with correct intrabar fills.

Your without more own coding, please use only marketorders and CalcBarOnClose=TRUE
-> market entry orders now filled "OnOpen"
-> please write little own code for your exit conditions and use "CloseMarket" (also filled on next Open)

With this simple OrderLogic, Backtest and Replay run nearly equal. The last difference: Replay use only TickData files from NinjaServer, BackTest use the from real DataProvider requested data's. If time bars greater than 1min used, then NT7 dont used tick data for backtest.

For the next month i hope to publish my (tickbased) history solution for the LOM project. This well realize a accurat Fill incl. IntraBar Exit/ReEntry for the BackTest mode. The concept is simple current you deglare myStrategy:Strategy, then you change only this to myStrategy:LOMtickStrategy.

Your current problem can base on NinjaServer ReplayTicks <-> MinData of your DataProvider or your strategy use OrderTypes with needed correct Intrabar handling of Hi/Lo sequence and/or IntrabarExit/ReEntry or you use partial scale In/Out.

Reply With Quote
 
  #4 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received


TimeTrade View Post
please say your used ordertypes... Without self coding, NT7 don't can backtest stop&limit orders with correct intrabar fills.

Your without more own coding, please use only marketorders and CalcBarOnClose=TRUE
-> market entry orders now filled "OnOpen"
-> please write little own code for your exit conditions and use "CloseMarket" (also filled on next Open)

With this simple OrderLogic, Backtest and Replay run nearly equal. The last difference: Replay use only TickData files from NinjaServer, BackTest use the from real DataProvider requested data's. If time bars greater than 1min used, then NT7 dont used tick data for backtest.

For the next month i hope to publish my (tickbased) history solution for the LOM project. This well realize a accurat Fill incl. IntraBar Exit/ReEntry for the BackTest mode. The concept is simple current you deglare myStrategy:Strategy, then you change only this to myStrategy:LOMtickStrategy.

Your current problem can base on NinjaServer ReplayTicks <-> MinData of your DataProvider or your strategy use OrderTypes with needed correct Intrabar handling of Hi/Lo sequence and/or IntrabarExit/ReEntry or you use partial scale In/Out.

Okay. A little confused by what you said frankly, however in backtesting I ran market orders only, and in live I ran limit AND market orders. Both generated similar results (live).

"A Jedi's strength flows from the force."
-Yoda
Reply With Quote
 
  #5 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received

Update... my algo is showing different values. Attached is a screen shot of backtest versus replay output for the same time period. Combined with the information that both scenarios are producing similar performance results, I'm going to look at how far the system examines past data. I already tried this once by setting the look back period for my replay data at 6 days, which should be plenty for the maximum period in my system.

Anyone run into a similar problem? Any ideas? Is JMA in fact not backtestable, despite possible similarities in the past?

"A Jedi's strength flows from the force."
-Yoda
Attached Thumbnails
Trouble with LIVE v. BACKTEST-replaychart.jpg   Trouble with LIVE v. BACKTEST-backtestchart.jpg  
Reply With Quote
 
  #6 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received

When I adjust the "min bars required" field in strat analyzer, the results vary hugely. They all have similar performance data, however the trades are very different. I suppose I should of picked up on this for a leading indicator. There appears to be no correlation between the look back period and number of trades. It also appears that the lookback period in back testing does NOT produce similar results when the same lookback period is set in market replay.

"A Jedi's strength flows from the force."
-Yoda
Reply With Quote
 
  #7 (permalink)
Membership Revoked
Oslo, Norway
 
Futures Experience: Advanced
Platform: CQG, Excel
Favorite Futures: CL
 
Lornz's Avatar
 
Posts: 1,198 since Apr 2010
Thanks: 500 given, 1,235 received


patbateman View Post

Does anyone know of alternative software platforms that will run C# (NOT easylanguage) with similar functionality as NT? I've heard Xtrader and T4 are pretty popular... I got into NT because of recommendations and liked the up sides of it very much, at least at the time.

Automated Trading Strategies Development Platform

Reply With Quote
The following user says Thank You to Lornz for this post:
 
  #8 (permalink)
Elite Member
NY, NY
 
Futures Experience: None
Platform: T4, NinjaTrader
Favorite Futures: All!
 
patbateman's Avatar
 
Posts: 235 since Mar 2012
Thanks: 113 given, 193 received

To make things even more amazingly confusing, here is what happened...

I ran the system on a different contract, this time the Dow (YM). The backtest and the replay are 99.999% identical.

I don't know what to make of this, other than there is something in fact going on with the ES contract in replay or backtest. I'll have to investigate tomorrow, it's bedtime now. The NT team probably thinks I'm making this all up to sabotage them... I'm not!

Thank you lornz!

"A Jedi's strength flows from the force."
-Yoda
Reply With Quote
 
  #9 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received

Anytime you are using non-tick bars (like 60 minute bars), you need to be aware of how NinjaTrader processes orders. It knows only the OHLC of the bar, and it does not know the order of those. Did the L happen before the H, or vice versa? It does not know when you use non-tick data.

From memory, it assumes the best. Unfortunately. This means if you have a 1 tick stop and a 20 tick target, if the H of the bar has exceeded your target, it will count as a winner, even if the L of the bar has exceeded your stop.

The easiest way to see how this effects you is to look at the individual list of trades from the backtest report. Sort by number of bars in trade. All trades that show '1 bar' should be swapped from winners to losers. You can do this by exporting to Excel pretty easily.

The other option is to add a secondary dataseries to your strategy, of 1-range. Then send all orders to the secondary series, so it has a 1-range (1-tick) accuracy.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #10 (permalink)
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,651 since Mar 2010
Thanks: 4,226 given, 25,599 received
Forum Reputation: Legendary



patbateman View Post
Update... my algo is showing different values. Attached is a screen shot of backtest versus replay output for the same time period. Combined with the information that both scenarios are producing similar performance results, I'm going to look at how far the system examines past data. I already tried this once by setting the look back period for my replay data at 6 days, which should be plenty for the maximum period in my system.

Anyone run into a similar problem? Any ideas? Is JMA in fact not backtestable, despite possible similarities in the past?

If I see the different results between the replay and backtest, there are two possible explanations

-> either you used two different strategies
-> or your strategy as coded is flawed

If your strategy is coded correctly, you will never see differences as between the two charts above.


The only differences you will see are

-> data related, replay data is built from many ticks, historical data is single tick data
-> fill related, the backtest will usually enter at the open of the next bar

If you want to do proper backtesting with intra-bar fills, you need to add a one-tick series to the strategy for you backtest.

Reply With Quote

Reply



futures io > > > > Trouble with LIVE v. BACKTEST

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Renko charts | Live versus Backtest patbateman NinjaTrader Programming 8 July 31st, 2012 01:00 PM
Typical ES slippage from backtest to live frostengine The Elite Circle 4 June 2nd, 2011 04:37 PM
DoubleMAPaint_SharkyPkg trouble desander NinjaTrader 7 February 4th, 2011 06:16 PM
A little trouble with limit orders... BigDog NinjaTrader Programming 8 June 21st, 2009 08:07 PM
Trouble with ChannelThis and DoubleMA lolu NinjaTrader 11 June 21st, 2009 06:38 PM


All times are GMT -4. The time now is 09:02 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-11 in 0.17 seconds with 20 queries on phoenix via your IP 54.83.122.227