Kevin, In the test period, did you have such an imbalance between the size of the winners and the losers? I am concerned to see you taking such large losses (almost 10 handles) with so little profit. Maybe I am misunderstanding things here.

Ddawg

Kevin,

I don't have a strong intuition on the meaning of that lower 10% line. I have a vague sense of what you are aiming to measure, but only a vague sense - and it's unclear how that lower line realizes that vague sense.

Suppose you wanted to know if your system is underperforming over any 18 trade period. Based on what I've learned from your threads so far, I might expect you to run a monte carlo simulation of 2500 18-trade sequences, and take the equity that demarcates the lower 10% of those results from the upper 90% as your cut off point. That would be a well-defined "lower 10%."

But I can see that it would be tedious to do that for each N-trades - you'd basically have to re-run the simulation every day - and so you're modeling the "lower 10%" in a different way. From your combine thread I have the following formula for your %-bands around the average result:

N * AvgTrade + Sqrt(N) * (Sigma) * X

Where I just assumed that the "X" corresponds to the number of standard deviations away from the mean that marks off the lower 10% of your individual trade results. If I understand Z-scores correctly (I probably don't!), I'm guessing X = -1.28 or so.

Even if I've interpreted the "X" in that formula correctly, I still don't have a very good intuition of just how that realizes what you want to be measuring.

Feel free to respond briefly, or not at all. A link to a well chosen wikipedia page might be enough to clear up my confusion.

thanks,
RK

Ddawg 

Kevin, In the test period, did you have such an imbalance between the size of the winners and the losers? I am concerned to see you taking such large losses (almost 10 handles) with so little profit. Maybe I am misunderstanding things here. Ddawg

Thanks for the question. It is a good one. For the NGEC strategy#1, which trades overnight, you are indeed right - the winners are small but frequent (75% of time), and the losers are small but infrequent. Overall, they yield a positive expectancy.

What happened last week is that there were 2 consecutive big losers in start #1, which had not happened in a while. As you can see from the equity curve, this has happened before.

rk142 

Kevin, I don't have a strong intuition on the meaning of that lower 10% line. I have a vague sense of what you are aiming to measure, but only a vague sense - and it's unclear how that lower line realizes that vague sense. Suppose you wanted to know if your system is underperforming over any 18 trade period. Based on what I've learned from your threads so far, I might expect you to run a monte carlo simulation of 2500 18-trade sequences, and take the equity that demarcates the lower 10% of those results from the upper 90% as your cut off point. That would be a well-defined "lower 10%." But I can see that it would be tedious to do that for each N-trades - you'd basically have to re-run the simulation every day - and so you're modeling the "lower 10%" in a different way. From your combine thread I have the following formula for your %-bands around the average result: N * AvgTrade + Sqrt(N) * (Sigma) * X Where I just assumed that the "X" corresponds to the number of standard deviations away from the mean that marks off the lower 10% of your individual trade results. If I understand Z-scores correctly (I probably don't!), I'm guessing X = -1.28 or so. Even if I've interpreted the "X" in that formula correctly, I still don't have a very good intuition of just how that realizes what you want to be measuring. Feel free to respond briefly, or not at all. A link to a well chosen wikipedia page might be enough to clear up my confusion. thanks, RK

You seem to understand what I am doing almost perfectly. The only difference I can see is that I run the day by day simulation (to get the +/10% curves) at the start of live trading, using all data to that date. So, I don't have to
re-run it every day - I only have to add in the actual results.


I look at the chart this way: after trading for a while, if the historical walkforward test can be believed still, what are the chances that my actual performance is not within the upper and lower 10% curves?

That means, for the lower line, that there is a 90% chance that my performance will be above the curve. If it isn;t, I start to wonder 1) has the system stopped working, and/or 2) has the market changed to something different, rendering my system ineffective?

This curves really becomes an early warning system that something might be wrong.

I hope this explains it - if not just let me know.

Kevin
your details are really great and like to try Practice sim in Topstep
topstep says we can't use auto strategy and Ninja trader but you seem to be using strategies. is that allowed in Trade station?

emini_Holy_Grail 

Kevin your details are really great and like to try Practice sim in Topstep topstep says we can't use auto strategy and Ninja trader but you seem to be using strategies. is that allowed in Trade station?

Right now I am not doing anything with TopStep - the system I am currently running is automated through Tradestation, and traded through their brokerage.

When I was doing the Combine, I was still using the same strategies, but I was manually entereing orders in the T4 platform, which connected to TopStep.

I hope this explains it - if not, please feel free to ask more questions.

thks Kevin

I heard Topstep allowed Ninjatrader for a while , but later they stopped.
T4 is ok, but is not as easy as ATM strategy in Ninja's DOM

emini_Holy_Grail 

thks Kevin I heard Topstep allowed Ninjatrader for a while , but later they stopped. T4 is ok, but is not as easy as ATM strategy in Ninja's DOM

Sierra Charts might be an option, if you are looking to automate with TopStep. I have never used it though.

Kevin,

Is the system really performing as designed? How many trades would it normally have taken over this period? What do the per-trade stats look like in testing vs live?

My fear here is that it is going sideways and you are giving it more time, but what happens should it take a couple of losses in a row? The ultimate question comes down to the first question above -- is it performing as expected, or not.

Mike

Big Mike 

Kevin, Is the system really performing as designed? How many trades would it normally have taken over this period? What do the per-trade stats look like in testing vs live? My fear here is that it is going sideways and you are giving it more time, but what happens should it take a couple of losses in a row? The ultimate question comes down to the first question above -- is it performing as expected, or not. Mike

Good question. After 7 weeks of live trading, is the system performing as designed?

First, let's look at the number of trades it has taken. A sharp increase or decrease in the number of trades, when compared to the walkforward history, would suggest that the market action is different than the historical market, causing many more, or far fewer, trades to be taken than normal. After 7 weeks, the system has traded for 18 days, or 53% of possible days. Historically, it trades about 151 days per year, or 60% of days. So, the system is trading less than the average. But some years it has traded as few as 130 days (51%) and as many as 175 days (70%). Based on all this I'd conclude that the number of trades is generally in line with expectations, although at the low end. Anecdotally, I have felt that the volatility has been lower than usual. There have been a handful of days where an entry was wished by only a few ticks (the big Fed move day Sept 18 was a case in point). A little more volatility in the hours before the announcement and the system would have entered a big winner.

Second, let's look at average performance, versus actual performance. The average performance of the system gives $1441 profit after 18 trading days. Actual performance, however, is at -$746. This is a HUGE discrepancy, and the conclusion obviously is that the system is not performing as well as it should.

But here is where is gets tricky. Take a coin, and flip it 100 times. If you get 60 heads, do you conclude that the coin is "broken," i.e. biased? What if you got 70 heads, or 80, or 90? Even if you flipped 100 heads in a row, could you absolutely conclude that the coin is biased? No! There is a chance, albeit very, very small, that a fair coin could be flipped to heads 100 consecutive times.

It is the same way with a trading system. You can ask "is it broken?" but the answer will always have some degree of uncertainty. That uncertainty sometimes makes all the difference to your conclusions.

Right now, the actual system performance is at the 10th percentile of what was expected. The 50th percentile would be right at the average, so the 10% mark is pretty bad. But, it is still within the realm of possible outcomes. If it was below the 0 percentile - let's say the system had lost $8,000 through the first 18 days - then the system is obviously not performing as expected. That would be an easy decision. It is where there is uncertainty that things get tough.

I know there are statistical tests that could be run to show whether the current trades could be part of the historical distribution of trades, but even that analysis is not definitive. There is always a gray layer.

So, how do I navigate the gray area? First, I try to determine the parameters that will cause me to quit trading a system. I could certainly use the percentile number approach, and have a quitting rule that says "if after X days the performance is below the Yth percentile, I will cease trading." X and Y would be at your discretion, based on your personal preferences. As long as you stick to the rule you create, you'll be doing fine.

For me, and this particular system, earlier I decided that I would quit only when I hit a $5000 drawdown. So, I am not using the data in the tracking graph to decide when to quit. Sometimes I do use it, though.


So, to summarize:

Is the system performing as expected? No, not even close. It is performing much worse than expected.

Is the system "broken?" Maybe, maybe not. It depends how you define broken. One cannot say definitively it is broken, or it is not broken.

Am I going to quit? No. My quit point, established earlier, calls for a single contract drawdown of $5,000. This was a well thought out amount, and I can't just toss it out the window. I'm going to stick to the plan. I realize, though, that sticking to the plan might be akin to the captain sinking with the ship. It was a risk I was comfortable with at the start, and still am.