|
New York, NY
Experience: Intermediate
Platform: TradeStation
Broker: TradeStation
Trading: Forex
Posts: 33 since Feb 2011
Thanks Given: 16
Thanks Received: 16
|
Hi, I have received permission from MC to share this email exchange here.
It might be useful for people running portfolio optimization, it is in relation to my earlier post
I realize it is a very long post, but please spend a few minutes to read if you are portfolio backtesting. A good starting point if in red.
I am currently studying the answer of MC support.
Attachments mentioned will be added in a separate post.
Please provide your own feedback, ideas, etc.
------------------------------------------------------------------------------------------------------------------------------------------------
Hello Philippe,
Thank you for your kind patience. We have analyzed the situation and It works correctly as expected. Let me explain how it works:
1. We have performed the followings tests:
One symbol @ES + set of signals. Population Size = 100; Maximum Number of Generation = 100; Genetic optimization.
Two symbols @ES + set of signals for each symbol. Population Size = 100; Maximum Number of Generation = 100; Genetic optimization.
One symbol @ES + set of signals. Limited set of inputs, exhaustive optimization.
2. The first case showed max. NetProfit = 2800
The second case showed max. NetProfit = 4250 (2000 for the first symbol @ES and 2250 for the second symbol @ES)
The third case showed max. NetProfit = 3450
3. Conclusion: the best results are provided by exhaustive optimization, then the genetic optimization of one symbol and then the genetic optimization of two symbols.
• Exhaustive optimization calculates all points.
• Genetic optimization does not. It is limited by population size.
• Population size is not relevant for exhaustive optimization. It is calculated on all points.
o In your case you have Population Size = 100 for both variants (100 for one symbol and 100 for two symbols).
o By default the Population Size for one symbol = 675; Maximum Number of Generation = 4726;
o By default the Population Size for two symbols = 741544; Maximum Number of Generation = 5190812;
So the Population Size = 100 is not enough and it doesn’t cover all points for calculation. This causes the difference.
Solution: 1. Use exhaustive optimization with limited range of inputs.
2. Increase Population Size and Maximum Number of Generation for genetic optimization if you use 2 symbols (Population Size = 10000 and Maximum Number of Generation = 10000 for 2 symbols. Results: max net profit was 5312,5 (3000 for 1 symbol and 2312,5 for the other symbol)).
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto [email protected]]
Sent: Friday, May 25, 2012 3:35 AM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hi Andrew,
Thank you for the feedback.
You did not confirm if you had access to the TradeStation data to reproduce the behavior on your end.
As I wrote earlier, the data exports are 54 MB and 53 MB, I could try to send them via email, but I doubt it will go through. I’ll try something else if you have an idea.
Philippe.
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Thursday, May 24, 2012 1:43 PM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Philippe,
The engineers have investigated the situation, but I need to discuss couple of details with them and I’ll send you the email first thing in the morning tomorrow.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Thursday, May 24, 2012 3:31 PM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Here is a very rough optimization result looking at 2 possible values per parameter (12 parameters total) or a total 4096 combinations (see screen grab).
optimization - net profit Time @EC @ES Total
MC portfolio #1 (w/ HT on*) Exhaustive
4096 tests 3h30 $4,845 $11,340 $16,185
MC portfolio #2 (w/ HT off*) Genetic
100x100 50 mins $4,845 $11,340 $16,185
*As a side note: I have an Intel i7-2600K CPU. When using Hyper Threading (HT), the Genetic Optimization with 100 generations of a population of 100, the total number of simulations is 900 (taking 50 minutes). When not using the HT, the total number of simulations is 500 (taking 24 minutes).
More tests with all possible parameters like previous emails but w/ HT off
optimization - net profit Generation x
Population Time @EC @ES Total
MC portfolio #1 100x100 25 mins $0 $41,125 $41, 215
MC portfolio #2 100x100 25 mins $0 $41,700 $41, 700
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Wednesday, May 23, 2012 2:50 PM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Philippe, could you decrease the number of simulations for exhaustive optimization to make it possible and try to reproduce the situation meanwhile? Thankly uo for your understanding and cooperation.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Wednesday, May 23, 2012 8:54 PM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Andrew,
Here are my results / comments:
• Could you make your initial capital in Portfolio Backtester 100 000 000 000 and perform the same test you did, Sir?
Yes, here are the results:
optimization - net profit Generation x
Population Time @EC @ES Total
MC portfolio #1 w/ 100,000,000,000 Capital 100x100 50 mins $95,837 $19,877 $115,715
MC portfolio #2 w/ 100,000,000,000 Capital 100x100 50 mins $97,642 $5,152 $102,795
MC portfolio #3 w/ 100,000,000,000 Capital 1000x1000 8 hrs and
50 mins $87,510 $37,475 $124,985
Analysis: the test #1 and #2 are in line with the results of my email of May 8th (in this emails thread): 30% lower than ran individually. Test #3 total is closer to the individual tests shared in my email of May 8th, however the ES result in lower (20%) than when run as an individual strategy.
My goal is to run the genetic optimization on a custom criteria of (Max Profit / - Max Drawdown). I must be sure that each occurrence of the strategy for each symbol is optimized as needed. In a portfolio, it seems that the 2nd strategy/symbol optimization is not as good the 1st and this is even worse when a 3rd strategy / symbol is added (ref. email of May 15th).
• Could you perform the same test using exhaustive optimization?
900 tests take approximately 50 minutes, exhaustive optimization (number of simulations shown at the bottom of the attachment) would take 235 million years on my computer.
Regards,
Philippe.
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Tuesday, May 22, 2012 8:46 AM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Philippe,
• Could you make your initial capital in Portfolio Backtester 100 000 000 000 and perform the same test you did, Sir?
• Could you perform the same test using exhaustive optimization?
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Thursday, May 17, 2012 5:31 AM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Andrew,
Thank you for your answer. I understand the non-repeatability of the genetic optimization.
I ran all my optimizations in Portfolio. None in MultiCharts (ref. your comment highlighted below).
Actually, my point is that I obtain consistently results that are 30-45% lower while optimization 2 symbols at a time as compared to 1 symbol at a time. I repeated this observation on many runs, I only show in my table a small sample.
Moreover, a test ran on 3 symbols led to a negative Net Profit for @EC whereas I would typically obtain the best results for this symbol as compared to the other ones.
I’ll gladly provide more information if needed.
Regards,
Philippe.
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Tuesday, May 15, 2012 3:23 PM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Philippe,
Thank you for your patience, Sir.
You are using the genetic optimization, that does not give you the same results even if you run once again and again. It means that you should not compare genetic optimization results of each symbol in Portfolio to each symbol in MultiCharts. Judging by your screenshot 5-7-2012 6-32-16 AM.png we can see that if the initial capital value is really great in Portfolio Backtester you can try exhaustive optimization. If all the input parameters including data series settings are the same, it should provide you with identical results.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Tuesday, May 15, 2012 3:00 AM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Thank you for the update.
Actually, I ran another test with adding a 3rd duplication of the strategy with a 3rd symbol (@TF) and the result of the optimization for one of the symbol is negative!
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Monday, May 14, 2012 12:22 PM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Philippe,
The engineers are still in the process of investigation. We will get back to you as soon as possible, I believe that will be tomorrow. I am sorry for keeping you waiting. Thank you for understanding.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Friday, May 11, 2012 7:46 PM
To: 'Andrew Shuvlenov'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hi Andrew,
Any news?
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Wednesday, May 09, 2012 11:48 AM
To: 'Philippe Schulligen'
Subject: RE: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hello Philippe,
Unfortunately I have to ask you to wait. It may take some time to investigate the situation since our engineers are off for today due to public holiday. I will get back to you with some news as soon as possible.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Tuesday, May 08, 2012 3:41 PM
To: 'Andrew Shuvlenov'
Subject: Optimization of 1 strategy on 2 symbols in the Portfolio Backtester - discrepancy
Hi Andrew,
- Description of the issue: I am trading a portfolio using the same strategy duplicated on multiple symbols at the same time. I want to optimize the inputs of the signals of the strategy individually for each symbol but in the context of the portfolio. The individual optimizations provide consistently better results than doing an optimization as a portfolio using genetic optimization with the same number of generations and the same number of individuals (100 each). As you can see below, the total of both net profits of the top result is 30-40% higher than the results of the portfolio. The results are similar when using the fitness function NetProfit or NetProfit / -MaxStrategyDrawDown, (the built-in custom criteria). My goal is to build a portfolio with the best NetProfit / -MaxStrategyDrawDown possible, and so far the individual optimization provides better results than the optimization as a portfolio.
optimization @EC @ES Total
MC individual #1 $83,142.00 $45,977.00 $129,119.00
MC individual #2 $92,740.00 $47,927.00 $140,667.00
MC portfolio #1 $73,220.00 $18,245.00 $91,465.00
MC portfolio #2 $53,272.00 $48,367.00 $101,639.00
MC portfolio #3 $65,552.00 $34,765.00 $100,317.00
- Screenshots: see attached.
- Exported from PowerLanguage Editor signals: Export of Stochastic Slow LE PhS signal (StoSlowLEPhS.pla)
- Exported from QuoteManager symbols (with data): I did my test on @ES and @EC TradeStation data. I prepared the exports but they are 54 MB and 53 MB respectively. I doubt that these files will go through the email. Can you access the data on your own or do you have a way for me to provide it to you?
If possible I would like this question to be tracked in the PM tool.
Regards,
Philippe
________________________________________________________________________
From: Andrew Shuvlenov [mailto:ashuvlenov@MultiCharts.com]
Sent: Friday, May 04, 2012 8:41 AM
To: 'Philippe Schulligen'
Subject: RE: 8.0.5203 Beta 2 64bits - difference in optimization result on charts vs portfolio backtester
Hello Philippe,
Please describe the issue in details, + screenshots. If you can also send me the Portfolio workspace, exported from PowerLanguage Editor signals and exported from QuoteManager symbols (with data) to reproduce the issue on our end.
Best Regards,
Andrew Shuvlenov
Customer Care Specialist
1-888-340-6572
Ashuvlenov@MultiCharts.com
6:00 am - 4:00 pm EST
Follow us on Twitter, Facebook, LinkedIn, and our Traders’ Blog
______________________________________________________________________
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the system manager.
This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.
________________________________________________________________________
From: Philippe Schulligen [mailto[email protected]]
Sent: Thursday, May 03, 2012 8:39 PM
To: 'Andrew Shuvlenov'
Subject: RE: 8.0.5203 Beta 2 64bits - difference in optimization result on charts vs portfolio backtester
Thank you Andrew, actually one of my screen capture was incorrect.
Since then, Beta 3 corrected this issue (this is actually specifically in the list of changes made in Beta 3).
How do you suggest I submit another issue regarding the optimization of 2 strategies together in the Portfolio Backtester giving net profit results that are way below the optimization of the strategies individually when optimized in the same conditions?
Philippe
|
