CrudeDude 

Date System(s) My P/L Hypo P/L Trading Diff Start/Stop Diff My start/stops License fees Conv. Rate My Total P/L 05/22/2019 GriBor Intra 6 _ E-mini Crude Oil QM ($270.00) ($270.00) - - - - - ($270.00) 05/24/2019 GriBor Intra 6 _ E-mini Crude Oil QM ($445.00) ($441.43) ($3.57) - - - - ($715.00) 05/27/2019 GriBor Intra 6 _ E-mini Crude Oil QM ($320.00) ($331.54) $11.54 - - - - ($1035.00) 06/03/2019 GriBor Intra 6 _ E-mini Crude Oil QM $155.00 $157.08 ($2.08) - - - - ($880.00) Net totals (USD converted) ($880.00) ($885.89) $5.89 $0.00 0 $0.00 USD totals ($880.00) ($885.88) $5.88 $0.00 EUR totals €0.00 €0.00 €0.00 €0.00 I bought into this isystem last month and promptly lost on 3 trades. Their performance info made it look like all I had to do is sit back and wait for the money to roll in. ha ha. Now I'm wondering if I walked into a buzz saw??

SMCJB 

Here's your system. As you can see unless June is at our above average performance it will already be out sample after just 6 months. Remember the darker grey band shows a one standard deviation performance band, and the light grey two standard deviations. Hence if the backtest is predictive of future performance we would expect 95% of system performance to fall in the light grey band.

Another major disappointment...

Tradesignalgo 

been tracking Google Trends Monthly NQ and Google Trends Weekly NQ since Dec 2018, vastly different results even on the same logic but different parameters, if such structured systems by the professionals do not work in the long term, what is the chance of the retail that tries to build these on his own?

SMCJB 

One of the drawbacks of the way isystems work is that it's impossible to use a walkforward development. While not everybody uses walkforward I do believe it's more common than not with successful systematic traders. That could be a factor as to why so many of these systems appear to struggle. Looking at those two systems specifically... The weekly after only slightly underperforming for the first year of tracked, has really crashed in the last 5 months. Conversley the monthly has been over performing for most of it's 2+ years of tracked performance. Unfortunately in the last seven months it's experienced two large drawdowns never before experienced. Saying that, it made the money back in both cases and is currently at new highs and has made more per month since being tracked than it did in its backtest.

And another bad one...



And finally, our first system, that a year after first being mentioned you would actually wanted to have been trading! Even this one though is in a drawdown over double what has ever been seen before.

Some Thoughts.

So my original portfolio contained 6 systems, other people mentioned 8 other specific systems in this thread, so that's a total of 14 systems that have all been live for between 14months and 3 years since being mentioned.

Of those
- 1 is outperforming expectations over the long run - but is currently in a massive previously unseen drawdown!
- 3 or 4 are breakeven to slightly positive in USD results but all underperforming expectations
- 9 or 10 are negative despite having great looking backtests.

Conclusions? Don't believe backtests!

SMCJB 

Some Thoughts. So my original portfolio contained 6 systems, other people mentioned 8 other specific systems in this thread, so that's a total of 14 systems that have all been live for between 14months and 3 years since being mentioned. Of those - 1 is outperforming expectations over the long run - but is currently in a massive previously unseen drawdown! - 3 or 4 are breakeven to slightly positive in USD results but all underperforming expectations - 9 or 10 are negative despite having great looking backtests. Conclusions? Don't believe backtests!

Would agree.. dont trust them. Breaking it down to its simplest terms; Back testing is nothing more than a data model which tries to predict the future. A good friend of mine who is a data scientist once told me "All models are wrong but occasionally they provide good information." I believe this to be absolutely true. After messing around with automated systems for years I came to conclusion it just not possible to accurately produce the a automated system based on a long periods of historical back testing. In fact, I tried to get my data scientist friend to help me design a automated system to which he thoroughly mocked me (to this day) for my naivety for believing it could be done.

If you look at the guys who run algos, they are not using indicators or bar movement to predict the future. They are looking at order entry and capitalizing on imbalances of buy and sell orders and taking advantage of how orders are routed through the market to get better fills.

With this said, I am not saying its impossible to find a system that works. What I am saying is a broken clock is right twice a day and it possible to get lucky. However, its not sustainable long term. There is far better uses of your time and capital which can be found.

Robert

SMCJB 

Update... Went up for 1 more month, and then straight down ever since. For clarification, not trying to call out @CannonTrading in any way, just highlighting specific systems that were discussed in this thread and how they have performed since. As you can see, in almost every case the answer is 'poorly'.

Good pointers and really sums up the biggest challenge many clients/ prospects face when they decide to go with auto system or managed accounts.

Will the system continue to perform when market conditions change? how long / IF can it return from drawdown?

Very FEW systems and money managers have shown that ability and very hard to find and then the biggest question is the unknown....

I have seen quite a bit of the issues you pointed with more than a few systems and money managers these past few months. Volatility spiked, market relations and behavior changed across the board and simply put this is not the same market today as it was 8 months ago - across MANY sectors.

Diversification, evaluation and setting max risk settings before starting out any auto trading is key in my opinion.

I do something in a copier i build for Mt4 and Mt5 to get around the problem of underperfroming strategies so far it has work great but it is only 100 days since i started. But orignal portfolios are at loss which was expected since there is a lot of exotics.

For example :
Copier only copy trades from a strategy if Strategy has above 60% winrate last x trades but it will copy only when it have had 2 loosers in a row and stop when it have acheived 2 win and start over. The combinations of winnner / loosers/winners and aswell rolling x trades Profitfactor all seems to improve result when combined with winrate looking back x trades.

One thing that seems to be correct in backtest is actually winrate . Take break outs their winrate is usally around 37-45 % overtime but when they are working and making money they have a winrate above 60 %. So now i am trading 1350 strategies and are sorting out the ones that work plus adding some safty by trading them first after x loosers. a bit Crazy i know but had to try it and so far it looks great. I also can sort portfolios based on their performance using indicators simple and nested (whatever) and only copy the good ones but this do not seems to work that well and it is probably better to copy them in some drawdown % so it works alot better on indivdual strategies using current last x trades winrate as base performance factor.

Here is another way to analyse portfolios and strategies in a portfolio. It is a quota from a private chat. More realated to what has been done here.

Last Portfolio with 80 strategies made 7.2% that was good. I found something about strategies. The overall portfolio forecast was very good. I had only 5% of chances to get below 20% of growth and got 7.2% appearing that portfolio forecast is superestimated. So I analyze each strategy in portfolio and found some "rotten apples" strategies between them that could impact of future forecast. There was a strategy that the forecast was only $40 dollars with standard deviation of $930 dollars. For this example of strategy, profit or loss is a flip coin case. Completely random and there is no reason to keep this strategy in portfolio. So, I made a final of final version of building portfolios hehe. The difference first I filter all strategies using last 100 weeks to get the ones with 99% of chance to have a mean above 0. From 1750 strategies that I have, only 25 passed all criterias (T statistic > 2.85, no autocorrelation and normally distributed for forecast validation). And the best portfolio built, use only 18 from 27. small number of strategies but all of them have a high chance to profit in future. Lets see the result, but until now, is performaming better then a big portfolio.

Thanks @Mabi. Both posts very interesting.

Mabi 

For example : Copier only copy trades from a strategy if Strategy has above 60% winrate last x trades but it will copy only when it have had 2 loosers in a row and stop when it have acheived 2 win and start over. The combinations of winnner / loosers/winners and aswell rolling x trades Profitfactor all seems to improve result when combined with winrate looking back x trades. One thing that seems to be correct in backtest is actually winrate . Take break outs their winrate is usally around 37-45 % overtime but when they are working and making money they have a winrate above 60 %. So now i am trading 1350 strategies and are sorting out the ones that work plus adding some safty by trading them first after x loosers. a bit Crazy i know but had to try it and so far it looks great. I also can sort portfolios based on their performance using indicators simple and nested (whatever) and only copy the good ones but this do not seems to work that well and it is probably better to copy them in some drawdown % so it works alot better on indivdual strategies using current last x trades winrate as base performance factor.

I think that is what people would think of as equity curve trading? I belive the famous Turtle Trading System did something similar, but in their case they would skip the next trade if the previous trade was a winner. For me and iSystems I only built the ability to retrieve the monthly PnL so could not do anything as elaborate as that.

Mabi 

Here is another way to analyse portfolios and strategies in a portfolio. It is a quota from a private chat. More realated to what has been done here. Last Portfolio with 80 strategies made 7.2% that was good. I found something about strategies. The overall portfolio forecast was very good. I had only 5% of chances to get below 20% of growth and got 7.2% appearing that portfolio forecast is superestimated. So I analyze each strategy in portfolio and found some "rotten apples" strategies between them that could impact of future forecast. There was a strategy that the forecast was only $40 dollars with standard deviation of $930 dollars. For this example of strategy, profit or loss is a flip coin case. Completely random and there is no reason to keep this strategy in portfolio. So, I made a final of final version of building portfolios hehe. The difference first I filter all strategies using last 100 weeks to get the ones with 99% of chance to have a mean above 0. From 1750 strategies that I have, only 25 passed all criterias (T statistic > 2.85, no autocorrelation and normally distributed for forecast validation). And the best portfolio built, use only 18 from 27. small number of strategies but all of them have a high chance to profit in future. Lets see the result, but until now, is performaming better then a big portfolio.

Be curious to know what "get the ones with 99% of chance to have a mean above 0" actually means?

While I didn't try exactly what you are saying here, those were exactly the type of rules I was trying to use. Highlighting a previous post though, past performance definitely was not a predictor of future performance, even when looking at the best performing systems..

SMCJB 

Highlighting how difficult systems picking is I have 15498 samples of data, with 15 consecutive months of since tracked, not backtest, data. For now I am keeping back the most recent data and using 13924 samples. If a system has a 50% chance of being profitable each month, then the probability of being profitable 12 months in a row is 0.5^12 which would equate to 3.4 of my 13924 samples. In fact I have 4! Maybe surprisingly all 4 of these lost money over the next 3 months!

To know the confidence pl monthly mean for example, calculate t statistic = monthly mean / monthly standard deviation * sqrt(number of months). With t statistic you can use a function in excel tdist to found the probability given t statistic and sample size (months).

Made an Excel last year were i could test diffrent combinations of Winners,Looser, Winrate to see if it had any impact on result and it had so thats why i implemented it automated of course. I made sevral versions could only find this one that still works. It is 6000 real trades on mini accounts FX ( breakout trades only). You can try it out on Isystems maybe. But each strategy need to have a identification and i suppose you do not have this. .MagicAnalyser_working.zip