Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
How many trades are needed for significant results?
This subject came up in a different thread that went off the rails before I think this question was properly addressed. I want to discuss it more formally because more and more I am seeing people buy into claimed trading systems without sufficient enough data to prove their efficacy. I think that the retail trading community maybe sets the bar too low.
The question is essentially how much data do you need to justify moving forward with a system? Some results are just random, and so to prove that a strategy's performance is not just random you need to show its performance over a large enough data sample.
There's certainly some rules of thumb. I like to have four years worth of data and at least 1000 trades in my backtests. If there's any kind of fat tail involved with the net profit per trade or win/loss streaks then I might look for 10,000 trades or more. That is if my particular strategy allows for it. I'm finding myself more and more in situations where my system just can't handle what I'm throwing at it. Then of course I run it against live data in a sim account for a time, and always scale up slowly.
It sure upsets me though when people try to claim they've developed a holy grail with only a month's worth of trades. That's someone that's either clueless or trying to deceive you. I've had multiple occasions where I've had good profitable periods for several months, and then the market changes.
Perhaps there is a mathematical way of determining if your strategy has enough trades based on the stats of your backtest. We'd probably need a math phd to figure that out though. In the meantime explain to us what benchmarks your own systems have to meet before you are satisfied.
I would much prefer a discussion about how much data do you need to justify moving forward with a system as a discretionary trader? I would bet the majority of traders fall into this category. What do you think?
In my opinion it's no different from an automated strategy. I backtest it over multiple years of data, then I run it for a month or two on sim, and then maybe I trade it live. Why bother if you don't have data suggesting some historical efficacy?
Now one might argue that plenty of traders were successful without going through this process. We've all heard stories of a trader joining a prop firm, trading on sim for a month or two, and then going straight to live. However, I'd argue that such traders actually did have that data suggesting historical efficacy. The prop firm already did that research for them. There's real math behind why those trades work, and the real life performance matches the theory. So all they needed was a month of the trader's stats to verify if they can actually perform the strategy or not.
Unfortunately, we don't really have those strategies anymore. Robots made it impossible to compete in that world. There's plenty of people out there claiming to have such strategies, but they never provide that empirical evidence to verify that it is effective. Hence, anything that we're trading these days are edges we discover on our own.
And so you have to do the verification of the strategy on your own too. Otherwise discretionary just becomes "I'm not skilled enough to actually verify my strategy so I'm yoloing it".
At least a thousand incidences at a bare minimum would give you an idea of what's going on
"Now one might argue that plenty of traders were successful without going through this process. We've all heard stories of a trader joining a prop firm, trading on sim for a month or two, and then going straight to live. However, I'd argue that such traders actually did have that data suggesting historical efficacy. The prop firm already did that research for them. There's real math behind why those trades work, and the real life performance matches the theory. So all they needed was a month of the trader's stats to verify if they can actually perform the strategy or not."
I think that's one of the biggest fallacies perpetuated in this industry. People don't join prop firms and become successful. Prop firms have nothing to do with anything. They take your money for a desk to pay their rent in a NYC or Chicago. They don't give a damn what you do.
There are almost no traders that are successful, and nobody that had an inkling of what they were doing would join a prop-shop
Don't mean to derail the thread I think it's very valid. Get a thousand trades under your belt the cheaper the better then you'll figure out what's what.
Depends, for my purposes I use year worth of data. I agree with Shrike, number of trades here are the key, if system is not taking enough trades on backtest with that amount of data then I'll have to increase it. I trade intra and my avg in trade time is around 25 min, so I usually do not face such an issues as all the strategies I test have enough trades over the year for me to get good picture.
Also, on front of short amount of data backtests, I think culprits here are also platforms which offer short period data on their basic packages, new users simply do not know better or have the required money to invest in data packages and buy tools separately or just never study the subject enough to know not to trust them over such short periods. For example, Tradingview backtests for a very short amount of time in its basic subscriptions, you will have to hard code the ability to increase the periods and even then there are multitudes of problems that can come up. This has led to most people doing backtests on TV to be either on completely misguided path till they blow their accounts or turn into snake oil salesmen who post fancy backtest results over short periods to attract attention of new blood.
it's not easy to answer these question because the trade activity of a system is very variable. It could be 100 trades per hour, day, week, month or year. So there isn't an valid answer to your question.
But there is a common approach to get over these discussion and get into the markets faster (with SIM, forward testing or real trading) - Equity Curve Trading. If you have made your backtests you know how your expectancy is and how your drawdown is distributed. So you know how YOU think that your model will perform, which leads to an range of possible equite curves. You can then build your "stop trading rules" for the case that these expectations didn't realize. Often there is an plain moving average applied to the equity curve to do this.
This works better the more smoother your equite curve runs. It's perfect for systems which run excellent in specific market conditions. But it's only one weapon more in trading, to achieve the goals. It doesn't match to all systems.
I may know the thread you mention here. ( ) It did implode, for reasons unrelated to the testing issue.
Personally, I think this is a valid question that does not really have an answer. Or not an easy or unequivocal one.
On one hand, I think that most traders look at two or three charts (perhaps covering two or three days of intraday data), say, "Hey, this works," and go with it. A lot of "trading" is pure sim, in which case there is no financial harm in this, but harm is still done from time wasted and from a constant churn of new trading ideas and no consistency. If this is done with real money in live trading, it's a good way to spiral down the drain. There are reasons that most traders get nowhere, and jumping from idea to idea with no testing to speak of is one of them.
On another hand, if you elaborately test thousands of data points, you will be looking for something that holds up over many changes in market conditions -- but some things work well in high trending conditions but crash and burn outside of trends, and others are good in non-trending periods but keep you short in big uptrends (or long in downtrends) as they fight the trend. Unless you have something in your system that is intended to differentiate different market types, testing over long periods basically involves looking for homogenized systems that try the same approach under all conditions, something I am skeptical of.
I do think it's important to know how an idea has worked out in the past, but it's also important to know how far you can trust your sample of the past to be like the future.... which is ultimately just something you will have to assume, because it's not something you can know.
But the question is still extremely valid. In terms of the issues facing traders, the question is how much can be essentially the trader's judgment and how much should be systematically tested. Too much of either will get you nowhere, as will not enough, or so I believe. Many people who focus on testing think "judgment," something not quantified in an automated system, has no place at all and should be eliminated. This does make testing easier, but are the best traders those who only use automated systems? I have seen successes from automated and non-automated trading, although often more as aspirations than as reality, in both cases. The question of how to test it is still relevant for either type. How do you know if it works is always relevant.
I hope there are more perspectives on the question in this thread, as well as some reasoning of why x amount of testing is the right amount (you sometimes see some number asserted without much evidence), and some discussion of what the issues are that need to be addressed by traders that the recommended testing is designed to solve.
Note that the question is not the same as how much testing should you insist on from someone who is trying to sell you a system. The best idea is simply to assume that if someone is selling something, it isn't working for them and that you should not buy it. There is no need to find a defense against the claims of system sellers, just simply don't buy them. (Buying them is another reason traders don't succeed... there are so many reasons, but this is easy to fix.) This applies to turnkey systems as well as trading rooms and anything else that is sold for money. If it works, there is better money for the seller in trading it than in selling it. If they are selling it, there is better money for them in selling it than in trading it. A simple rule. There can be value in courses and books, or in trading tools, but not that much and not if they are expensive. Remember the rule about whether they can make more selling it or using it. It's a good rule to follow.
But what should we insist for ourselves, for our own trading? Great question. I hope for many answers and a good discussion.
Bob.
When one door closes, another opens.
-- Cervantes, Don Quixote
) It did implode, for reasons unrelated to the testing issue.