WARNING: MC 8.5 walk forward optimizer is FLAWED - MultiCharts | futures io social day trading
futures io futures trading


WARNING: MC 8.5 walk forward optimizer is FLAWED
Updated: Views / Replies:8,274 / 46
Created: by treydog999 Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

WARNING: MC 8.5 walk forward optimizer is FLAWED

  #11 (permalink)
Elite Member
Wayne, NJ USA
 
Futures Experience: Intermediate
Platform: Multicharts, Deltix DCS
Broker/Data: Interactive Brokers/IQ Feed
Favorite Futures: ETF Options
 
Posts: 41 since Oct 2012
Thanks: 7 given, 36 received

Hey Treydog I think I might know why you are unable to recreate the walk forward results with time stamp flags. Multicharts WFO breaks up the data into insample and out of sample data segments. When it runs the strategy on the out of sample segment it does not have the data from the insample data set. Ex. you use the 20 day moving average your strategy will need to wait 20 days before it can start trading in the out of sample data set. When you use time stamp flags your strategy has access to all the data so it does not need to wait 20 days. I mentioned this flaw design to Multicharts on their forums.

Reply With Quote
The following user says Thank You to quantarb for this post:
 
  #12 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


quantarb View Post
Hey Treydog I think I might know why you are unable to recreate the walk forward results with time stamp flags. Multicharts WFO breaks up the data into insample and out of sample data segments. When it runs the strategy on the out of sample segment it does not have the data from the insample data set. Ex. you use the 20 day moving average your strategy will need to wait 20 days before it can start trading in the out of sample data set. When you use time stamp flags your strategy has access to all the data so it does not need to wait 20 days. I mentioned this flaw design to Multicharts on their forums.

Yes this is part of the problem I have been addressing with multicharts. However if give it the entire data set and use the time and date stamps of the trades in that period since you have the lead time it should be OK. i also have done it out of sample session by session and you are able to replicate the correct number of trades if you give it the extra lead time. However if you use commissions and slippage its still incorrect. So if you go window by window + lead time you can get very close to the WFO answer, though not exactly. I thought if you use the date flags, on the entire set the preload time would be built in so it should be ok. Thats not exactly the case but I haven't investigated it further until i can replicate the WFO results exactly using commissions slippage window by window. Honestly i work on this almost every day with these guys at support. I just want to get back to strategy development, its a huge hinderance and annoyance but what else can i do? Anyway they were supposed to get back to me about that yesterday, maybe i will get something today. when i do i will update.

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
  #13 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


So I got an update from Andrew. Basically its confirmed that adding commissions and slippage does not allow you to replicate the WFO. Although we had some minor success with no commissions and slippage. He told me their engineers are working the new beta and that they don't have time for my problem. Which is really not just my problem but every multicharts users problem. I am really getting frustrated with the amount of time I have to go back and forth with them to prove that I found some severe flaw. Not to mention the fact that they keep sending me fixes that really don't fix the entire problem. So now I am on the back burner again, I have no idea when they will get back to me with an answer. This is really holding up my development process. The worst part is they asked me to take down this thread, or say that the problem was fixed.


XXXXXXX, I saw you opened a respective thread on Big Mike’s Forum. Since we discussed the Walk-Forward Optimization and steps to reproduce it manually in MC, would you be able to comment that there is no issues with that? Thank you.

Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST


No the problem is not fixed, you are able to reproduce it on your end so no I will not comment that there are no issues when there blatantly are issues.

Reply With Quote
The following user says Thank You to treydog999 for this post:
 
  #14 (permalink)
Elite Member
Hannover
 
Futures Experience: Beginner
Platform: MultiCharts
Favorite Futures: EUR/USD
 
Posts: 5 since Jan 2012
Thanks: 1 given, 1 received


treydog999 View Post
So I got an update from Andrew. Basically its confirmed that adding commissions and slippage does not allow you to replicate the WFO. Although we had some minor success with no commissions and slippage. He told me their engineers are working the new beta and that they don't have time for my problem. Which is really not just my problem but every multicharts users problem. I am really getting frustrated with the amount of time I have to go back and forth with them to prove that I found some severe flaw. Not to mention the fact that they keep sending me fixes that really don't fix the entire problem. So now I am on the back burner again, I have no idea when they will get back to me with an answer. This is really holding up my development process. The worst part is they asked me to take down this thread, or say that the problem was fixed.


XXXXXXX, I saw you opened a respective thread on Big Mikeís Forum. Since we discussed the Walk-Forward Optimization and steps to reproduce it manually in MC, would you be able to comment that there is no issues with that? Thank you.

Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST


No the problem is not fixed, you are able to reproduce it on your end so no I will not comment that there are no issues when there blatantly are issues.





IMHO this is very important issue. I'm on the step to WFO my strat which uses timestamps. What should i do ? take a break for a couple of monts/years ?

This is absolutely a NO GO. We have to force the support of MC to resolve this issue. In my opinion the new beta is not such important like WFO problem on an existing version.

Reply With Quote
 
  #15 (permalink)
in a meditative state...
Portugal
 
Futures Experience: Advanced
Platform: Multicharts, Jigsaw
Broker/Data: Stage5, OEC
Favorite Futures: Treasuries
 
arnie's Avatar
 
Posts: 801 since May 2010
Thanks: 709 given, 976 received


wbailey View Post
In my opinion the new beta is not such important like WFO problem on an existing version.


Actually, the new beta is probably the most important beta/version released in the last couple of years.

MC 8.7 introduces millisecond data and this forced the database to suffer a major upgrade.

So you basically have now a complete new millisecond database that needs to have backward compatibility with the previous time keywords/functions and new ones were built to take advantage of the millisecond data.
Also, a couple of other tools will see significant changes due to the new database which will enrich MC's experience.
I'm sure this was not an easy task and most of MC resources must have been on this the last few months.

If I become half a percent smarter each year, I'll be a genius by the time I die

Last edited by arnie; April 27th, 2013 at 10:03 AM.
Reply With Quote
 
  #16 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


wbailey View Post
IMHO this is very important issue. I'm on the step to WFO my strat which uses timestamps. What should i do ? take a break for a couple of monts/years ?

This is absolutely a NO GO. We have to force the support of MC to resolve this issue. In my opinion the new beta is not such important like WFO problem on an existing version.

I agree with this. Basically we just have to wait until it is fixed before we can continue our development. Which is total BS.

Reply With Quote
 
  #17 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received


arnie View Post
Actually, the new beta is probably the most important beta/version released in the last couple of years.

MC 8.7 introduces millisecond data and this forced the database to suffer a major upgrade.

So you basically have now a complete new millisecond database that needs to have backward compatibility with the previous time keywords/functions and new ones were built to take advantage of the millisecond data.
Also, a couple of other tools will see significant changes due to the new database which will enrich MC's experience.
I'm sure this was not an easy task and most of MC resources must have been on this the last few months.

I am not denying that an upgrade with great features is going to be good. But why not fix a feature that has been claimed to be working for several releases? I mean its a bedrock feature they advertise on their website. Basically everyone who owns multicharts has a broken piece of software, or at least one that is falsely advertising it has the capability to do something it really can't. I may be emotionally charged here because its holding up my own progress, as well as how annoying it is to work with them. So I am just going to stop here. Either way, I want this thing fixed.

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
  #18 (permalink)
 Vendor: tradingcode.net 
The Netherlands
 
Futures Experience: None
Platform: MultiCharts, TradingView
Favorite Futures: ...
 
Jura's Avatar
 
Posts: 774 since Apr 2010
Thanks: 2,347 given, 661 received


arnie View Post
Actually, the new beta is probably the most important beta/version released in the last couple of years.

That sounds a little bit bold, why do you think that? (also related to my question below)


arnie View Post
MC 8.7 introduces millisecond data and this forced the database to suffer a major upgrade.

Where did you find that? Don't get me wrong, I'm not saying that I don't believe you , but I like to know what things we can expect and why this will be the most important version in the last couple of years. If I look in the Project Management I see four features and one bug fix for MC 8.7 Beta 1.


arnie View Post
I'm sure this was not an easy task and most of MC resources must have been on this the last few months.


treydog999 View Post
I am not denying that an upgrade with great features is going to be good. But why not fix a feature that has been claimed to be working for several releases?
(...)

I agree with Treydog999 here, since I don't consider it a very valid reason that they were busy with new features. Of course, new features are great and I look forward to these as much as any other user, but introducing new features can not be (in my view) an excuse to not fix the existing product.

Fixing bugs should always take precedence since we can easily wait two more months before a new feature is introduced, but, as others have already said in this thread, we can not wait two months before we can perform a reliable WFO.

Reply With Quote
The following user says Thank You to Jura for this post:
 
  #19 (permalink)
in a meditative state...
Portugal
 
Futures Experience: Advanced
Platform: Multicharts, Jigsaw
Broker/Data: Stage5, OEC
Favorite Futures: Treasuries
 
arnie's Avatar
 
Posts: 801 since May 2010
Thanks: 709 given, 976 received


Jura View Post
That sounds a little bit bold, why do you think that? (also related to my question below

Where did you find that? Don't get me wrong, I'm not saying that I don't believe you , but I like to know what things we can expect and why this will be the most important version in the last couple of years. If I look in the Project Management I see four features and one bug fix for MC 8.7 Beta 1.

I agree with Treydog999 here, since I don't consider it a very valid reason that they were busy with new features. Of course, new features are great and I look forward to these as much as any other user, but introducing new features can not be (in my view) an excuse to not fix the existing product.

Fixing bugs should always take precedence since we can easily wait two more months before a new feature is introduced, but, as others have already said in this thread, we can not wait two months before we can perform a reliable WFO.


Hi @Jura

Since HFT and algorithmic trading is the hottest thing around these days, I thought "algo" guys would love the idea of having milliseconds available to them, giving the extra granularity to program their strategies, hence - the most important version release in the last years.
Sorry for that, apparently my assumption was wrong
A good friend of mine always told me that my assumptions will be the death of me

Please don't misinterpret me, bugs should have always precedence but there are bugs, and I have no idea if this is such a case, but I assume (again, I'm assuming) it's not since several fixes have been made available and none solved the issue, that will be easily fixed with some new feature that will be released in the next version.
So if some bug will be easily fix when that new feature becomes available, why lose time trying to fix something now if in a couple of months it might stop working again since with that new feature, the way that tool works will change again?

I share treydog pain, heck, we all share his pain, we all had our share of working with something that was only half working and had to wait for the next version to see the problem solved but the only thing I could do was nagging support about it, so they could move faster in solving the problem.

I hope I'm not putting words in his mouth but I believe treydog said he was losing his time testing the fixes MC sends to him.
He discovered the bug. The tool is not working accordingly to his settings and knowledge.
Apparently, developers thought the tool was working fine, accordingly to their knowledge.
Assuming treydog is more knowledged on the matter, since he's a trader, he's the most qualified to be testing this, telling developers that the tool continues to be faulty, which he is.

Helping MC developers with this treydog is helping himself on the long haul because he will have a tool set accordingly to his settings.

Personally I consider this to be a very useful and friendly relationship between developer and final user. Both helping each other in building a better platform.

If I become half a percent smarter each year, I'll be a genius by the time I die
Reply With Quote
The following user says Thank You to arnie for this post:
 
  #20 (permalink)
Elite Member
Wayne, NJ USA
 
Futures Experience: Intermediate
Platform: Multicharts, Deltix DCS
Broker/Data: Interactive Brokers/IQ Feed
Favorite Futures: ETF Options
 
Posts: 41 since Oct 2012
Thanks: 7 given, 36 received


My biggest complaint with Multicharts is their half-finished portfolio backtester. I already have given up on their walk forward optimization. Single symbol walk forward optimization is pointless for me since I backtest on my strategies on many symbols to ensure robustness. Besides no walk forward optimization there is no bar magnifier making the execution model less accurate. Worst you canít even trade the portfolio strategy without having open up an individual chart for each symbol. This is very time consuming and greatly limits the number of symbols and strategies you can trade.

Each new version of Multicharts introduces both bug fixes and new bugs. I feel like youíre stuck between a rock and a hard place. Do you have to continue to suffer from existing bugs or upgrade and deal with the new bugs?

At this point, Iím waiting for either Multicharts 9.0 with the feature to trade multiple instruments from chart or Deltix DCS Trading. I thought version 8.5 would be the last release before 9.0. Now there is a Multicharts 8.7 betaÖ

Reply With Quote
The following user says Thank You to quantarb for this post:

Reply



futures io > > > > WARNING: MC 8.5 walk forward optimizer is FLAWED

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Jigsaw Trading: TBA

Elite only

FuturesTrader71: TBA

Elite only

NinjaTrader: TBA

Jan 18

RandBots: TBA

Jan 23

GFF Brokers & CME Group: Futures & Bitcoin

Elite only

Adam Grimes: TBA

Elite only

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Is Walk Forward Optimization always a must? Albnd Traders Hideout 15 October 28th, 2016 10:00 AM
Synthetic Data Walk Forward Optimization quantarb Elite EasyLanguage Automated Trading 5 December 23rd, 2012 01:52 PM
Walk Forward Experiment Silver Dragon Elite Automated Trading 327 September 9th, 2012 12:56 AM
Walk Forward Optimization - memory consumption Mondrakete NinjaTrader Programming 2 September 7th, 2011 06:04 PM
Walk Forward caprica NinjaTrader 4 August 31st, 2009 06:51 PM


All times are GMT -4. The time now is 02:09 PM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-15 in 0.16 seconds with 19 queries on phoenix via your IP 54.82.56.95