I am currently backtesting a new model in excel, where I tested 47 trades (I am doing this manually; don't have any backtesting software, thats why I currently only tested 47 trades) here are my statistics:
So I was reading a thread that Mike posted and downloaded this Monte Carlo simulation model (which is attached), I ran numerous simulations and it reported positive profit after 47 trades (ranging from $400 to $2000)
I am not sure if this is too good to be true or not, or how valid/relevant the model Mike posted is.
Here is just a quick rundown of what I saw and how I interpret my own backtests. so take this with a grain of salt. I am not judging manual vs automated testing or any of that at all, just purely at the results given.
So, the first thing i notice is low sample size 47 trades. Yes statisticians say 30+ is "significant" for a sample however more is always better. A quick plug into the old formula you get 14.43% standard error with this sample set alone. So any monte carlo based on that is going to have a built in 14% error either way. Monte carlos are only as good as the data set they can take from.
Looking at the trade stats you NET .5(pt) per trade and win 55% of the time. Is this with or without commissions? If not then they are going to take a sizable portion of your edge and make this system very difficult to trade. If it is with commissions then go on right ahead to incubation mode.
Another thing you did not mention is how many rules you have or filters. When you have a sample size this small it really can effect and skew both results and monte carlos. If you have 500 + trades and have 5 filters your ok, but if you 50 trades and 5 filters thats not ok. Statistically speaking, although i am a huge proponent of less filters, and less rules are more robust.
Also mention the date range that your using to test. Is this over 1 day, 1 month, 1 year? Both insample and out of sample to cover atleast 1 full market cycle. Though optimal is not very realistic, however more time is better. I look to have at least 3-5 years segregated for out of sample.
Hope to hear how this all works out.
The following 2 users say Thank You to treydog999 for this post:
I have CQG and DTN iq feed. You can search their websites to see which service fits your needs. i would say DTN is best bang for the buck. However the best free resource is quandl, you can download a lot of free good data there.
The following 3 users say Thank You to treydog999 for this post:
Glad you've built your first model. Just like you, when I started my first model from excel I had tons of questions and wanted to share any some info that might be of use to you.
1. On Backtesting Software - Treydog has given excellent points. Apart from the speed and scale which you can backtest many markets/asset classes, there is also the huge benefit of automated execution. I found my usage of time much more productive in research instead of monitoring trades. One can now trade up to 20 or more positions simulatneously (long term or short term). The caveat of course is that the model developed is accurate and robust and monitoring systems from time to time is still a prerequisite.
2. On Monte Carlo - It is an interesting measure but wouldn't put too much faith in it. Actual drawdowns are a function of a specific market events/environment in relation a series of bad trades produced. Trade scrambling removes the serial correlation which happens during this period and how the drawdown was actually produced. It would be useful to look into the periods which drawdowns occured and understand what caused them. This is crucial to deduce if risk management approaches are needed (stopping the trading model, develop non-correlated models etc.) ... All these decisions have 'costs' to the long term outcome of the eventual pnl. Alternatively, one could accept that this is a function of markets where drawdowns are inevitable from time to time and just continue to stick to the plan.
3. Number of trades - Perhaps instead of measures of statistical significance, one could take a qualitative approach and see what the system was designed to capture. Warren buffett as well as many long term trend followers although might fail statistical tests by their number of trades, but their performance is certainly more than random. One needs to take into account the holding frequency and frequency of conditions present.
4. Clean Data - Good models need to be built on clean and more importantly adjusted data. Even accredited data providers do not provide properly adjusted data for backtests and will lead to incorrect trading conclusions. Thus, the need for backtesting software to adjust data to generate a proper backtest.
Lastly, I personally found the most crucial parts in building good trading models was in thinking and capturing the stationarity of the market. I got this general idea from visiting the local library and reading books by clifford j sherry. The 2nd part was asking myself how long this 'stationarity' would tend to persist. The better which these 2 questions could be answered, the better the models have performed going forward. I hope some of my personal experiences would be of use to you going forward.
The following 2 users say Thank You to Ming80 for this post: