WARNING: MC 8.5 walk forward optimizer is FLAWED - futures io
futures io futures trading



WARNING: MC 8.5 walk forward optimizer is FLAWED


Discussion in MultiCharts

Updated by GuppyDRV
      Top Posters
    1. looks_one treydog999 with 16 posts (30 thanks)
    2. looks_two badabingbadaboom with 6 posts (1 thanks)
    3. looks_3 Jura with 3 posts (1 thanks)
    4. looks_4 quantarb with 3 posts (2 thanks)
      Best Posters
    1. looks_one swz168 with 2.0 thanks per post
    2. looks_two treydog999 with 1.9 thanks per post
    3. looks_3 quantarb with 0.7 thanks per post
    4. looks_4 Big Mike with 0.3 thanks per post
    1. trending_up 10,815 views
    2. thumb_up 38 thanks given
    3. group 12 followers
    1. forum 46 replies
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

WARNING: MC 8.5 walk forward optimizer is FLAWED

(login for full post details)
  #1 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received

I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

Reply With Quote
The following 4 users say Thank You to treydog999 for this post:

Can you help answer these questions
from other members on futures io?
OPEN_CLOSE_DOTS
ThinkOrSwim
DLL with webrequest function for MC
MultiCharts
Looking freelancer for MT4 Indicator
Platforms and Indicators
help to convert from thinkscript to ninjascript
NinjaTrader
is there an easy way to obtain by code the price variation percentual
MultiCharts
 
 
(login for full post details)
  #2 (permalink)
Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,793 since Jul 2012
Thanks: 1,440 given, 5,385 received


treydog999 View Post
I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

Yikes, that is a bit scary!

Advice for people new to WF, not for Treydog: For anyone doing walkforward, I always recommend doing a case by hand when first starting out with it. It is tedious and time consuming, but at least for me I understood the concept and application so much better afterwards. Try a simple 5 year in period, and 1 year out period, and walk it by hand.

I'm guessing Trey did something like this, to uncover this issue. Thanks for alerting everyone Trey!

Follow me on Twitter Reply With Quote
 
(login for full post details)
  #3 (permalink)
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
 
teyano's Avatar
 
Posts: 22 since Feb 2013
Thanks: 67 given, 14 received



treydog999 View Post
I have been banging my head against the wall because of the WFO results I have been getting. So i started to look deeper into it. Turns out there is overlapping time periods in the WFO engine, instead of true shifting of time windows. Also you can not replicate the WFO results inside a strategy using Date and Time flags. I have been working with tech support for over a week, and they finally today decided to acknowledge the flaw. So anyone out there developing using it, be very very careful.

@treydog999 Did Multicharts solved the issue ?

Reply With Quote
 
(login for full post details)
  #4 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received

They sent me a hot fix a few days ago. But I haven't had time to test it. There is no general fix out there that i know of. They did not fix the ability to recreate results using date and time flags as I understand it but i would need to test it before I say anything conclusive.

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
(login for full post details)
  #5 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received

Alright, I just did some manual tests vs the WFO engine. The date and times no longer overlap which is good. however the trade results are still incorrect. So i would say its still broken. I have again submitted my issues to multicharts, i been working with Andrew and Henry. But they take forever, took them weeks to come up with the hotfix. So basically do it by hand if you have to. I feel like they should be paying me to be a debugger or something. Seriously this is ridiculous. To bad I have the lifetime license

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
(login for full post details)
  #6 (permalink)
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
 
teyano's Avatar
 
Posts: 22 since Feb 2013
Thanks: 67 given, 14 received

This is really quite pathetic for @MultiCharts If they consider this as a no priority issue (it seems though) for a backtester to be flawed in a software that is designed for algo trading than they need to seriously reevaluate their business. Really.

Reply With Quote
 
(login for full post details)
  #7 (permalink)
The Netherlands
 
Experience: None
Platform: MultiCharts, TradingView
Trading: ...
 
Jura's Avatar
 
Posts: 774 since Apr 2010
Thanks: 2,347 given, 685 received


teyano View Post
This is really quite pathetic for @MultiCharts If they consider this as a no priority issue (it seems though) for a backtester to be flawed in a software that is designed for algo trading than they need to seriously reevaluate their business. Really.

Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

That is, of course, not to say that if there is an error in the WFO, this needs to be fixed (obviously). Can you keep us updated about this Treydog999?

Reply With Quote
 
(login for full post details)
  #8 (permalink)
Amsterdam, Europe
 
Experience: Intermediate
Platform: Various
Trading: ES
 
teyano's Avatar
 
Posts: 22 since Feb 2013
Thanks: 67 given, 14 received


Jura View Post
Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

Thats true, I admit my statement was overdone, but considering the fact the this thread is here almost one month and it's on the very top of the whole Multicharts side it would be desirable if someone could make a comment form Multicharts regarding this. What is more a work around that makes "the trade results are still incorrect" is also not the best case scenario.

Reply With Quote
 
(login for full post details)
  #9 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received


Jura View Post
Well, we don't know how much work these issues are and how many programming hours have been assigned to this already, and if they are really errors or just 'design choices'. So I think that your 'really quite pathetic' is a little bit overdone - even treydog999 was unassuming when he said "i would say its still broken".

That is, of course, not to say that if there is an error in the WFO, this needs to be fixed (obviously). Can you keep us updated about this Treydog999?

I will definitely keep this thread updated with new developments between me and the developers/engineers. the outlook does look grim though. They told me when i first came to them about it that it was "too complicated" at first. Then they sent me the hot fix for the easiest part of the problem, but its like putting new paint on a lemon. Looks a lot better but still under the hood is broken. Supposedly I will get in update next week to see how it goes with the programmers. They have extracted so much data from my including my strategies, workspaces and data stream to isolate it. But upon their own admission it is systemic. So its just going to be a waiting game at this point.

Reply With Quote
The following user says Thank You to treydog999 for this post:
 
(login for full post details)
  #10 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received


Ok so after working with the developers at multicharts, I am able to use a work around to independently verify the walk forward tests. You are required to get the hot fix they gave me, or as them to post it on the forum or something. Without it this is useless. This does not work with time and date flags inside the code itself, you have to manually adjust adjust the look back window in the instrument settings area. They key aspect is to incorporate a lead time of X bars back for however many bars back your walk forward used. SO if you have a 200 bars back on a daily, you need to move your starting window 200 days earlier in the instrument settings. I tried this on the worksheet and strategy optimization they sent me as an example. I haven't done it on any of my strats so far because they are not daily bar based, so figuring out how many bars back you need is going to be a pain. However at least you can verify that now the walk forward does in fact give you proper results post hotfix.

I will continue to work on this and see if i can verify the results with intraday bars and on more complicated strategies.

Reply With Quote
The following 3 users say Thank You to treydog999 for this post:


futures io Trading Community Platforms and Indicators MultiCharts > WARNING: MC 8.5 walk forward optimizer is FLAWED


March 9, 2017


Upcoming Webinars and Events
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts