Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Traders with 5-10 years of experience but still not profitable
There is a lot of talk about retail vs institutional. I have traded retail and while I am not with a hedge fund or a large financial institution, I have seen that side as well. This post is about whats different. I havent seen all of the institutional side obviously. So this is based on my experience only.
And I am speaking strictly from a technical trading point of view and specific to the petroleum markets. I am not talking about firms that trade around their physical assets (tank farms, pipeline credits, barges, rail road etc) that they have access to. So, this is just about purely speculative trading that uses exchange generated market data - Time and sale, BBO (L1), market by level (L2) and market by order (MBO).
Broadly speaking, for speculative, algorithmic (or manual) technical trading you need market data and analysis. You want to analyze market data and establish an edge before you start trading it. I hope we can all agree to this last part.
Market Data: Institutions and retail traders can access the same market data.
Analysis: Math is math, its not proprietary.
So, whats different?
At trading firms, it will typically be something like this. They will first establish a bunch of market hypothesis - either discover them by research or brainstorm. They will then write programs to test each hypothesis. They will follow a well defined process with pre-defined performance metrics. Then compare hundreds of such tests to see which ones are historically profitable among other factors. This is known as backtesting. Then go on to the next phase like available liquidity, margin requirements etc to determine position sizing and risk management aspects. Out of hundreds of tests, maybe one or two will make it to actual execution. For this process, they will get the best, most comprehensive historical market data and use high performing infrastructure for their computing environment to speed things up.
Whereas, a retail trader will typically open up a few charts with platform vendor provided data, superimpose platform provided or community programmed "indicators" on the data. Watch it live or replay for some time and develop a mental hypothesis. Build enough conviction that it has an edge and start trading it. Lets say this takes a month. Then, most likely this fails so the retail trader then goes back to the drawing board and develops the next hypothesis. And on and on until time or money or both runs out. I have been there so I know this cyclical, frustrating process very well.
So, the process followed by both is similar except, the trading firm followed a well-defined, repeatable process and went through hundreds of tests in minutes or hours using a programmed process to clearly document an edge while the retail trader is doing the same thing but manually and very slowly.
The point I am trying to make is that retail traders run out of time and money before they can get to the point where they actually find their edge in the markets. What keeps most new comers from doing what the trading firm does is the lack of programming skills and poor funding.
My advice to new comers is that please:
Learn how to program, you can buy well organized courses for various programming topics at udemy.com for as little as $12. My suggestion is to learn python.
Fund yourselves enough for the time required to go through this phase, to purchase some decent computing hardware and accurate and comprehensive historical market data.
Program your own charts using many available open source charting libraries.
If you read these books and feel the need to purchase Amibroker - please dont. All the programming can be done in python.
Write programs to test whatever you think will make money using historical market data, keeping a very critical eye on the results.
Stay away from live markets during this phase.
When you find something with stellar results, then take it to market live. No sim. There are various micro futures available now to have your skin in the game without committing too much money.
This is one way I know for a fact that gets you closer to becoming successful at at least analyzing the markets well. If you can get this far, you are among the top 1% of new people starting out and a lot closer to being successful at trading your analysis.
If you need any help with any of this stuff, please reach out to me. I have done this for more than a decade. I cannot help you with specific trading strategies but I can help with navigating the market data and computing space.
I was initially going to write more about the Sharpe ratio, but then realized that what I cared about was more what Long-Term Capital Management's fate told us.
And in fact, as I remember very well from that time and from other large institutional events over the years, it tells us a lot. One thing is not to ignore the standard phrase, "Past results do not guarantee future profits," or however it goes. It really is not just a throw-away disclaimer of responsibility. It means "Don't count on this working tomorrow," which is about all you can say.
I like the Kaufman quote about profits. As you know, I don't do any automated trading, but I respect the process, and especially the testing procedures. From reading your stuff and Kevin's ( @kevinkdog ) on this forum, I can see that, once you have a proposed system that has passed your tests, you then take the fateful step of running it in real time with real bucks, and you're in the same position as all traders: staring into the same abyss (sorry, I like to dramatize sometimes ) of not knowing what will happen next. You've gotten there a different way, but that's where you are. And like anything else, if it works for you, that's what matters.
My thought about the Sharpe ratio (or any other measure of long-term profitability, however it is put together) is simply that it predicted Long-Term Capital Management's profitability very well, right until the time that it didn't. I think it's good to know about history, but the one thing it won't tell you is what will the market be like in the future, and how will something or someone respond to it. So these types of measurements have a place, but only up to a point.
Bob.
When one door closes, another opens.
-- Cervantes, Don Quixote
This is a really good, solid look at the differences between retail and institutional trading. The takeaway I see is that institutions have a much deeper bench, and can put more resources against a problem and work toward the solution more thoroughly.
I should point out that there are individual traders here on the forum, including some developing automated systems, who are both successful and are one-person shops. Not to keep beating the same drum, but LTCM had the best possible depth of expertise, staffing, equipment and experience, and didn't keep going.
There are also traders who never program, and never use a computer for anything but to display their charts, who also do well.
But certainly few, which is the point of this thread in a way: trading is not the simple way to get rich that it may seem to be. It will take time, effort and money, and not everyone will get there. Anyone who is inclined toward the highly data-intensive, programming-intensive side of things should read your post, if they need to know more about what is required.
And thanks for your generous offer of help. This is what this forum is about, ultimately.
Bob.
When one door closes, another opens.
-- Cervantes, Don Quixote
There are many advantages that institutional investors have over retail investors, but I have found that largely they're just playing a different game
I spoke with block traders when I worked at a bank. Their big advantage was they didn't really need to make any kind of prediction. A client would come to them with an order, and all they were doing was executing it as efficiently as possible. To some extent they knew what the market was going to do because they were the ones making it move. So the whole game is about limiting your impact. This is where you'll use things like a VWAP buy program, or Almgren-Chriss to minimize your execution impact as much as possible. They can also watch their institutional order flow which is a big help.
Working at a bank also gave me some insight into HFT's. I worked on trade execution systems, and HFT firms were our primary customers. One of the jobs I worked on was improving the accuracy of time measurement within our trading systems. We had special hardware that allowed us to measure in microseconds. That's a millisecond divided by 1000! All to give the HFT's we served an advantage in the queues. This is a place that retail traders can try to compete. To stand a chance though you'll probably need a co-located server and machine learning algorithms.
What retail traders are trying to do is probably more like hedge and cta funds really. Their edge has more to do with predictions and risk management. I never spoke with the hedge fund customers, but I did manage systems that allowed me to see the stock positions of every institutional client we had. It was usually stock positions that I had never heard of before, and they held positions for several weeks to months. So not really the speculative day trading that retail traders do.
Still you can easily see where the institution has advantages we don't have. They can pay for professional research. They can pay for a Bloomberg terminal. They have lots of employees to be able to broaden the search for opportunities. When it comes to a more algorithmic fund like Renaissance Technologies they do have data that retail traders don't. For my backtests I have to use tick replay data to test strategies based on order flow, and even then they'll only let me go back 2 years. I'm working on a system that records live data for me, but I'll have to collect all the data myself to get 5 years of data. I also have yet to find a good solution for syncing that data up with breaking news events or economic numbers. What's more is that such institutions will pay for past order flow data. This can help them see through some of the noise, and try to find patterns in the behavior of larger intuitions. This can then improve the accuracy of their models for things like end of month rebalancing behavior. Not to mention all the computing resources that often goes into machine learning algorithms.
This is not all to say that successful retail trading is impossible. There's a few hints in there. It's easier to trade a higher timeframe, and collaborating with others is a huge edge. We just can't do it exactly the way institutional traders are doing it. For those few successful retail traders out there the banks would love to know how they're doing it too lol. Don't tell them though as that probably ruins the edge. Keep it to yourself and start a fund instead please.
Ahh you're going to need a lot more trades than that. Remember that the more fat tailed a distribution is the more samples you'll need in order to get an accurate view of the distribution. If I'm doing something more high frequency intraday trading I'm shooting for 10,000. That might not be realistic for a more swing strategy, but I would always try to test on as large of a dataset as you have access to.
I agree. These are really not the retail trader's competition. The execution broker filling large orders for clients, either on the screen or block has a very good feel for the immediate order flow but he can hardly see past the near term market. For large orders or thin markets, I too prefer to use my brokers rather than execute myself. They always get me a better fill than I could have executing the orders myself. Market making HFT traders operate in a very different space too. Much shorter term. They can shave the retail trader's edge a little but not really compete directly. The retail trader is more like CTAs, hedge funds, prop trading firms that analyze historical data to find their edge, and this is where I think the retail trader is too slow and runs out of funds/time before finding theirs.
As far as data disseminated by the exchange is concerned, everyone has access to the same data if you can afford it. You could check out data vendors like dxFeed (this is who I use), algoseek, CME datamine for historical, CME SmartStream for real-time or ICE data services. All messages disseminated by exchanges are available historically and real-time including time and sale, best quotes and the entire order book. Although, data before mid-2017 is sketchy no matter which vendor you go to. Even CME does not offer market depth or MBO data before 2017. But you can get time and sale and best quotes data since 2007-2009 (for CME at least).
Anyone can purchase these data services, not just algo trading firms. You really dont need to record it unless you find it too expensive.
Or go join them. Thats what I did. If you have a documented edge but are unable to trade it or trade it well for any reason, get interviewed at trading firms. Its far easier to trade this way than going at it solo.
I don't have @TWDsje's experience in banking, but I think I can add to this line of discussion. I am not sure about how much market share (in AUM) the HFT's command, but the majority of the players that I have researched (ETFs/ETNs, CTAs, hedge funds, etc.) do not participate in HFT. HFT's are extremely sensitive to price data and they need that millisecond precision to remove as much slippage as possible and to execute at a price that may run away from them. A small difference in bid-ask or a tick in one direction or the other can be the difference between profit or loss.
Some people may think there is no cost to HFT, but all firms pay commissions and exchange fees, no matter how big they are. The exchanges are going to get paid, which is why it is my firm belief that trading is not a zero-sum game. The house always wins. So the HFTs have to take this into consideration for their systems. The CTA and hedge fund managers that I follow found that the added costs required for HFT (hardware, development, trading costs, human capital) did not justify adding
Data itself can be a funny thing. Live price and volume data is notoriously dirty. Imagine in the old days of pit trading, when a number might have been fat-fingered on the big board, before being corrected. A pit trader acting on the incorrect price is analogous to an HFT entry or exit triggered by a bad tick. Consider that the daily closing price does not always match the closing price intraday. Adjustments are happening while the market is closed and contracts are being settled. I would venture to guess price data is being corrected along the way.
I have dabbled in HFT (8-20 trades per day) from the retail side, using stop orders as a method to manage slippage. The signals would happen several minutes before my expected entry point, so I was not looking for immediate entry or exit. My trading platform was running on an Azure VM to eliminate potential power and internet outages. I had mixed results. Having a system running 23 hours per day that needs to be monitored was tiring. Profitability was about flat, maybe a little on the plus side. My broker loved me though and my commission costs fell with the volume. Execution rate was about 80-90%, meaning 10-20% of my orders were not filled. For a few hundred bucks that I may have made, it was not worth it. I spent more time maintaining these systems than they were worth. Also, just trying to determine profitability and match against the signals that should have happened was difficult and time-consuming, so I pulled the plug.
One thing I will note based on my what I have learned out institutional traders: They are not always as sophisticated as we may think. I know one major hedge fund sends their orders to their broker via an Excel workbook every end of day. A multi-billion dollar hedge fund sends their orders via a spreadsheet.
But what do I know?
Circling back to the Sharpe discussion, I saw this on Twitter this morning and got a chuckle: Jerry Parker's signature (former turtle and founder of Chesapeake Capital):
If we are truly at the start of a commodities super cycle as some analysts believe then the CTA space will be a good place to look for opportunity. I don't know much about that industry or how to get your foot in the door, but it's definitely something I'm keeping an eye on. The professionals are not omnipotent, and some of the retail traders here could have valuable experience that CTA's could put to work. Getting a series 3 license is on my list of things I'm looking into this year.
One dead simple way to get into a commodities super cycle is go the ETF route, which @kevinkdog pointed me to after I was unable to trade futures due to some industry restrictions (my wife works for an investment bank). I am currently working on a specific basket of commodities ETFs on a system this week. They have cool names like: CORN, HOGS, COW, GLD, SOYB, CPER. After that I will be comparing and contrasting these against futures, but that may take a few weeks to get done, as I have other things on my plate.
These ETFs offer exposure, but with less risk than futures, at least as far as I see it. The underlying instruments are futures contracts in a lot of cases, and are a blend of mutiple contracts (front month, 2nd, etc.) and maybe some calendar spreads.
The analysts who get paid for their analyses are wrong probably as often as they get it right. Amateur analysis can offer about the same hit rate. What is important is the ability to research, critically think, and articulate an argument or position. I am annoyed when I read of an analyst who states: I predicted the 2008 crash. Listen....I could predict a crash every year, and be right eventually. As they say, even a broken clock is right twice per day (or once if you use a 24 hour clock ).
Series 3... yeah, interesting proposition.... hmmm.....
) of not knowing what will happen next. You've gotten there a different way, but that's where you are. And like anything else, if it works for you, that's what matters.
Thats what I did. If you have a documented edge but are unable to trade it or trade it well for any reason, get interviewed at trading firms. Its far easier to trade this way than going at it solo.
