Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Sorry if some of these thoughts and observations are repeats... I have not read your entire thread but thought it was important to give some insight from a data perspective.
First and foremost; There is no way you should quit or even think about quitting unless you have data to back it up your decision. You posted some guidelines for quitting here:
Now that I have decided to start trading my NGEC ("Not Good Enough for the Combine") strategy live starting on Monday, I have to address the question that everyone likes to avoid when starting to trade a new strategy: If things go bad, when do …
Have these changed? Have you hit one of these conditions?
From a data perspective you need more data points before you start asking if it is working or not. A data set needs a minimum of 30 data points to be statically significant. For trading you need more and from what I can see you are not there yet. Think of your entire equity curve as your arm. The live testing is about size of your fingernail. Your not comparing apples to apples . Give it time to develop and dont let your emotions get involved when determining if it is a good system or bad system. Emotions cloud your judgement and cause irrational decisions. If the system is truly bad, the data will bare it out in the in the end.
If you gave this system to me to manage here is the one question I would ask you:
What is max draw down for the system. In other words- What dollar amount are you willing to risk to properly test the system? Think of it as a longer term investment.. You have looked at the prospectus now you have to determine how much you want to risk and for how long you want to invest. I believe you have defined this in the post above as 10%
After that I would just let it run and DONT MESS WITH IT!!! Doing so nullifies your testing and any future comparisons... IMO you need to have at least 6 months to year worth of data for a proper comparison. This will capture different market conditions and news cycles. After this time is up then I would compare it to the past. (This assumes the max draw down has not been hit.)
My other suggestion would be to run the same strategy in SIM alongside live account. Maybe you are already doing this?? This will give you invaluable information. If there is a major difference in the day to day trades between the live and sim then I would stop immediately to find out whats going on. If it is taking similar trades, taking into account slippage and market conditions then you know it is working as designed. Finally I would run another back test after the 6 months is up to compare.
At this point you will have 3 data sets. In theory the two SIM back tests should be nearly, if not identical. The live set should be within acceptable levels of the SIM sets taking into account slippage and unforeseen market conditions which cause bad entries etc. If you do not want to wait the full six months to run the 3rd data set then run it monthly. Again, this will tell you if there is a problem in the code or the data. If it is significantly off then run a second month to see if the results are the similar. Stop the system if it is. Otherwise let it continue to play out.
The thing to remember is proper analysis takes time. Its easy to back test because of the 5 minute instant gratification factor. It is harder to stay the course and let the system play out for a year for better or for worse.
At this point I would not worry about $$ performance. Thats not important from an analysis standpoint. All that is going to do is cloud your judgement. Instead focus on the nuts and bolts of your operation. Make sure everything is running as expected. Do a weekly and monthly comparison of trades between SIM and Live and define the differences in a journal. In fact if you have time do it daily. Define the market conditions and what caused the difference. Once you get done with all that then you will have a detailed document of what went right and what went wrong. Then you can make a determination on if it was successful or not.
Finally, something to put things in perspective; Its possible to do everything right and still lose the game. So, you can have a losing system $$ wise but it still be a successful system from a execution stand point. In this case the detailed notes of the SIM vs Live executions will tell you where the adjustment needs to be made. Fact is systems have been known to fail $$ wise on as little as 1 tick slippage per trade.
BTW- I have had the same thing happen to me when I went live with my automated system; Results varied from the back test. I determined the issue had to do with the fills and the exotic bar type I was using. So your not alone.
just looking at your hypothetical curve diagrams and assuming diagrams had significance then you are nowhere near quitting point.
1) would you consider the curve as representation (or sum total) of various statistics?
2) what about the curve during the backtesting and forward testing portion made you decided to use the system?
3) what about the curve during the backtesting and forward testing portion would make you decide to not pursue the system?
since you used period of about 4 years
5) you would have to wait perhaps at least that long to decide whether system is worth it?
6) or if you have drawdown much longer in time duration (1.5x or 2x) and/or in much greater dollar loss (1.5x or 2x)
in my dayjob we do accelerated testing to validate the life of the product. the test has duration, predefined stresses and reliability with confidence attribute in the event sample test parts successfully pass the test.
if the backtest period ( 4 years in this case) is the life of the product, do you have to wait for 4 years to find out whether your product survives the life or is there an accelerated way to stress your product in much shorter period of time ( this would not be a walkforward test and period)? i could further expand on the analogy. perhaps it could be a useful analogy to what you are trying to do or it is just misguided rambling....
Thanks for the comments. I agree with pretty much everything you said.
One important part you mention is comparing SIM (or strategy results) vs. live. This is critical, especially for exotic bar types or strategies where slippage is a factor (and initial estimates of it were low).
I appreciate your excellent reply, even if you are a Bengals fan. But, they are doing much better than my Browns this year!
Not sure I understand this question. Can you explain?
I set out my goals and objectives before developing the system. Since it met those goals, I then considered it for live trading. For me, the primary number I look at is Return/DD ratio.
There are a lot of reasons why I might not trade the system, even if the number look good. Let's say, for example, that the numbers are great, but the equity has been down the past 2 years. That might be a deal killer.
I would use the 1.5-2.0 x drawdown before I'd wait 4 years. Your point is good though: the longer the real time out of sample period, the more confidence you'd have in the system.
I understand what you are getting at, but I'm not sure how you could do this. A couple of thoughts come to mind:
1. Run the system on simulated data - data that isn't real, but has same general characteristics as real data. I think Tushar Chande's book describes this method.
2. You could arbitrarily reduce all trades by a certain percentage. Knock 25% off wins, and add 25% to losses. Does the system still work? This might simulates future performance.
Thanks for the comments. You've put forth some good ideas to think about.
you have yourself pointed to the curves as basis for opinion whether to continue trading or not. yet you have statistics calculations which are calculated independently of the shape or form of the curve?
could you visually judge/compare Return/DD ratio of live trading period to that of backtested/walkforward period. If the answer is no, then you should not have offered the curves as the basis of opinion, but rather stats only. Anyhow a minor point...
Thanks for clarifying. I would not recommend looking at Return/DD for such a short period of time as I have so far in live testing. Maybe in a few years that will be appropriate. Hopefully I will still be trading it at that time!
I've noticed that Big Mike was interested about generating random data a couple years ago. There is a thread about it here in futures.io (formerly BMT) but nobody really seemed to be interested at the time. Anyway, I believe it's a very good idea …
I took the equity curve chart (most of it) and windowed it (cutting it in half and then cutting the pieces in half) until I reached a point where there were sections that significantly underperformed the "average performance" of the total equity curve. Place smallest oval possible around each section that contains the start, finish, and all equity curve action in between. Make copies of oval or ovals and place in such a manner that would be inherently pessimistic to the "live trade section". Exits outside the bottom side of the risk ovals would indicate problems.
Going forward, move the ovals to new start point as new data points come in at intervals the size of the windowed region the ovals originated from. Looks like a few weeks ago there should a new start point for the small oval if you keep my windowing points.
The large oval appears to be about $7,500 profit top to bottom. Since the oval won't allow for placement that takes the full range, probably 2/3rds of that would be enough to get the equity curve outside the range...your $5k drawdown power-off button seems right.
Can't say I know of a proof or theory that legitimizes this as a means to test an equity curve going forward. I'll let someone else do the math. Fun with Ovals....yay!!!
If nothing else, it shows that you have plenty of time to consider performance.
I should have given a bit more background about my analysis but it was late. Your survey about continuing or quitting the system got me thinking.
The statistics of how much data you need to say that your live sample is different from the in-sample period seemed to be the most important factor. That is what got me slicing up the data into windows to see how much time constituted a period of underperformance in the in-sample data.
Also I've known traders that have used ovals on price/time squared charts. The basic shape is interesting when you think about how positive and negative volatility works over time.
Sure the analysis is subjective, but it adds a visual component that is representative of underperforming periods in the in-sample data. In this case it shows that you need a good bit more time before trashing the system. It also frames the volatility of in-sample underperforming periods which should helpful in determining drawdown potential going forward.
You may laugh, but I am going to be adding this as a subjective component to my system tracking. I'll be doing it in excel instead of paint though...lol
