NexusFi: Find Your Edge


Home Menu

 





WARNING: MC 8.5 walk forward optimizer is FLAWED


Discussion in MultiCharts

Updated
      Top Posters
    1. looks_one treydog999 with 16 posts (30 thanks)
    2. looks_two badabingbadaboom with 6 posts (1 thanks)
    3. looks_3 Jura with 3 posts (1 thanks)
    4. looks_4 quantarb with 3 posts (2 thanks)
      Best Posters
    1. looks_one swz168 with 2 thanks per post
    2. looks_two treydog999 with 1.9 thanks per post
    3. looks_3 quantarb with 0.7 thanks per post
    4. looks_4 Jura with 0.3 thanks per post
    1. trending_up 15,494 views
    2. thumb_up 38 thanks given
    3. group 12 followers
    1. forum 46 posts
    2. attach_file 0 attachments




 
Search this Thread

WARNING: MC 8.5 walk forward optimizer is FLAWED

  #1 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039

I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

Started this thread Reply With Quote
The following 4 users say Thank You to treydog999 for this post:

Can you help answer these questions
from other members on NexusFi?
Request for MACD with option to use different MAs for fa …
NinjaTrader
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
NexusFi Journal Challenge - April 2024
Feedback and Announcements
ZombieSqueeze
Platforms and Indicators
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Retail Trading As An Industry
67 thanks
NexusFi site changelog and issues/problem reporting
47 thanks
Battlestations: Show us your trading desks!
43 thanks
GFIs1 1 DAX trade per day journal
32 thanks
What percentage per day is possible? [Poll]
31 thanks

  #2 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,645 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,336


treydog999 View Post
I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

Yikes, that is a bit scary!

Advice for people new to WF, not for Treydog: For anyone doing walkforward, I always recommend doing a case by hand when first starting out with it. It is tedious and time consuming, but at least for me I understood the concept and application so much better afterwards. Try a simple 5 year in period, and 1 year out period, and walk it by hand.

I'm guessing Trey did something like this, to uncover this issue. Thanks for alerting everyone Trey!

Follow me on Twitter Reply With Quote
  #3 (permalink)
 
teyano's Avatar
 teyano 
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
Posts: 22 since Feb 2013
Thanks Given: 67
Thanks Received: 14



treydog999 View Post
I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

@treydog999 Did Multicharts solved the issue ?

Reply With Quote
  #4 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039

They sent me a hot fix a few days ago. But I haven't had time to test it. There is no general fix out there that i know of. They did not fix the ability to recreate results using date and time flags as I understand it but i would need to test it before I say anything conclusive.

Started this thread Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
  #5 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039

Alright, I just did some manual tests vs the WFO engine. The date and times no longer overlap which is good. however the trade results are still incorrect. So i would say its still broken. I have again submitted my issues to multicharts, i been working with Andrew and Henry. But they take forever, took them weeks to come up with the hotfix. So basically do it by hand if you have to. I feel like they should be paying me to be a debugger or something. Seriously this is ridiculous. To bad I have the lifetime license

Started this thread Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
  #6 (permalink)
 
teyano's Avatar
 teyano 
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
Posts: 22 since Feb 2013
Thanks Given: 67
Thanks Received: 14

This is really quite pathetic for @MultiCharts If they consider this as a no priority issue (it seems though) for a backtester to be flawed in a software that is designed for algo trading than they need to seriously reevaluate their business. Really.

Reply With Quote
  #7 (permalink)
 
Jura's Avatar
 Jura   is a Vendor
 
Posts: 775 since Apr 2010
Thanks Given: 2,352
Thanks Received: 690


teyano View Post
This is really quite pathetic for @MultiCharts If they consider this as a no priority issue (it seems though) for a backtester to be flawed in a software that is designed for algo trading than they need to seriously reevaluate their business. Really.

Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

That is, of course, not to say that if there is an error in the WFO, this needs to be fixed (obviously). Can you keep us updated about this Treydog999?

Reply With Quote
  #8 (permalink)
 
teyano's Avatar
 teyano 
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
Posts: 22 since Feb 2013
Thanks Given: 67
Thanks Received: 14


Jura View Post
Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

Thats true, I admit my statement was overdone, but considering the fact the this thread is here almost one month and it's on the very top of the whole Multicharts side it would be desirable if someone could make a comment form Multicharts regarding this. What is more a work around that makes "the trade results are still incorrect" is also not the best case scenario.

Reply With Quote
  #9 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039


Jura View Post
Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

That is, of course, not to say that if there is an error in the WFO, this needs to be fixed (obviously). Can you keep us updated about this Treydog999?

I will definitely keep this thread updated with new developments between me and the developers/engineers. the outlook does look grim though. They told me when i first came to them about it that it was "too complicated" at first. Then they sent me the hot fix for the easiest part of the problem, but its like putting new paint on a lemon. Looks a lot better but still under the hood is broken. Supposedly I will get in update next week to see how it goes with the programmers. They have extracted so much data from my including my strategies, workspaces and data stream to isolate it. But upon their own admission it is systemic. So its just going to be a waiting game at this point.

Started this thread Reply With Quote
The following user says Thank You to treydog999 for this post:
  #10 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039


Ok so after working with the developers at multicharts, I am able to use a work around to independently verify the walk forward tests. You are required to get the hot fix they gave me, or as them to post it on the forum or something. Without it this is useless. This does not work with time and date flags inside the code itself, you have to manually adjust adjust the look back window in the instrument settings area. They key aspect is to incorporate a lead time of X bars back for however many bars back your walk forward used. SO if you have a 200 bars back on a daily, you need to move your starting window 200 days earlier in the instrument settings. I tried this on the worksheet and strategy optimization they sent me as an example. I haven't done it on any of my strats so far because they are not daily bar based, so figuring out how many bars back you need is going to be a pain. However at least you can verify that now the walk forward does in fact give you proper results post hotfix.

I will continue to work on this and see if i can verify the results with intraday bars and on more complicated strategies.

Started this thread Reply With Quote
The following 3 users say Thank You to treydog999 for this post:





Last Updated on March 9, 2017


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts