Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
Summary of All Types of Day Trading/Scalping Systems/Methods?
Enclosed is a link to a data sharing thread in the elite section on here. I have never used it because I get my raw data through a custom extraction program I built but others might comment on this.
I haven't spend much time bifurcating the various sub-categories within the directional TA group like you have, but I can mention generally speaking which styles are the most widely used successfully in practice.
1. HFT / Arbitrage: This is where algo's front run one exchange based on information from another. This is the most risk free, profitible strategy out there with sharpie ratios well into the double digits. Us mere mortals will never be able to play in this space, but I at least want to put it out there that I think this is # 1.
2. HFT / Market Making: Again, mere mortals will never play in this space.
3. Trading with size / pushing the market: I believe this is the top way that financial institutions do it... It's not a very sophisticated way to trade but it works very well. Here the idea is to exploit an area where the market is very thin on one side and very thick on on the other for the top 2-4 price levels. So a trader throws a huge order into the market clearing 2-3 price levels and driving the price in the direction they want. Then they exit on the opposite side where the market is thicker. It's an easy way to make 1-3 ticks. I see this happen all the time in the ES. This is not included in your mind map, I don't even know what to call this, but this is way that most big players do it I believe when they are swinging size.
4. Investing / Hedging: This is more of a long term holding strategy, not exactly day trading, so this may not fall into what you are looking for.
5. Non Directional TA: (Algo or Manual): There are not a ton of classes in this bucket, but this works off volatility levels, risk management, advanced order types, capturing the spread, etc.
6. Directional TA: (Algo or Manual) I just think of this as everything else. This is most of your mind map btw. Within this group you have everything from potentially viable to obviously fraud. snake oil, magic, etc.
I think a good exercise would be to find anything not in this last group and pursue it first. Very few, if any viable directional edges that are repeatable over time have ever been successfully quantified. Far more times than not, directional edges that have worked were often only successful over very high volatility periods, extremely bearish or extremely bullish periods. I have yet to see anyone articulate a directional edge like this: When X occurs, 51% of the time or greater Y will happen next and actually be able to pass the smell test. It may just be that those that have cracked this have never talked about it.... But I see a lot of of people talking about directional edges and none of them can simplify all the charting, DOM, TA down to a simple binary bet that can be tested..... It's almost always presented intentionally vague so it can't be tested. But if there is any out there with real lines in the sand that can be tested, look at those too. But just stay away from anything that tried deliberately to be vague for obvious reasons.
I keep getting requests for people to share their GomRecorder data. A few threads have been started on the subject but not many are following thru it would seem.
I am collecting data from IQFeed via QCollector. All data contains bid/ask and is tick …
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
I just became a member to gain access to the data + resources on futures.io - something I was considering doing anyway.
Your posts have been very helpful to me (thank you!) - but also bring up a lot of questions. Some of these are probably answered elsewhere (so if you could point me to whatever threads are relevant...):
1. What are you using to backtest the data you are extracting? Are you writing custom code, importing into NT, or just doing statistical analysis (in XLS, matlab, R, somewhere else?) on the data?
2. Where are you pulling the bid/ask/level 2 data from? is it from your data feed provider (FCM)? Is there any reason to use a data feed provider or can you get this data directly from the exchange / CME? Do they have APIs or is it something a little more janky? I'm assuming Rhytmic, CQG, etc. all have differences to be aware of.
3. You wrote your own software instead of using something "off the shelf", correct? Any tips / pointers on how to do that would be appreciated. I'm not looking for source code, just an idea of how you actually hook into the data if I need to.
4. What is "GOM" that I keep on hearing referred to?
5. Are there any books / tutorials that you recommend on algo trading and/or data extraction?
6. Have you looked into genetic programming / ML techniques?
I think no matter what style I pursue, I'll want some ability to backtest it / provide statistical data on trades - even if it is a technical + directional strategy. I know I wouldn't trust it any other way.
1. For testing purposes, if using NinjaTrader I recommend using market replay, not the strategy analyzer. The SIM engine in NT is fair for the most part. If you are doing scalping or HF stuff though, there are some design flaws that can mess you up though. The OnBarUpdate method front runs the level 1 data feed that processes the simulation, so if you scalp off of tick data using a 1 tick time series and you execute your code off of the GetCurrentBid() or GetCurrentAsk() calls you will front run the SIM engine and potentially trick it depending on what you are doing. The Fill engine regarding limit orders in NT is very conservative to the point that you virtually get no fills unless the market trades through you, so to this end I have built my own system offline. I coded something custom in SQL that I think is more realistic. It's always tricky to get right, and everyone does it differently, but I think the best approach to limit order fills / estimating your place in the queue is to take credit for cancels in proportion to your place in the queue. So if you are in the back 90% of the queue, you get 90% credit for all the cancels. When you get to the 50% mark in the queue, you take credit for 50% of the cancels. When you get to the top 10% of the queue, you take credit for only 10% of the cancels. In all cases you always get 100% credit for transactions, because these are obviously in front of you. NT does it better than most, but they only move you up in the queue if the entire queue moves below your position, so you are essentially only ever filled from the back which is very pessimistic. Also keep in mind latency modeling. The market replay tool uses 0 latency for order executions. By contrast their live SIM has around 100 - 200 millisecond latency on every order. These are not settings that you can change, and neither were realistic to my use case... But these can impact your result considerably depending on what you are trying to do.
2. As far as extracting the raw data. You can get this from creating a strategy in NT to just print to the output window, or write to a file. I have code that runs the level 2 OnMarketDepth and level 1 OnMarketData feed through a series of calculations for each price level. For every price level (Bid / Ask Change) I extract the total transacted volume for bids and asks, total canceled volume for bids and asks, total added volume for bids and asks as well as starting and ending volume for bids and asks. I also snap shot the full 10 level bid, ask level 2 book on each price level change, so I can test for any correlations there. The data is pretty dirty, so I have to run it through some custom programming to clean it, sequence it, and fill in any gaps. I am taking a few million rows a day and synthesizing it down to around 20K to 50K rows per day fully analyzed. This is likely an overkill for most traders, but for me I need the entire microstructure for the type of trading strategies, edges I am researching. As far as where to get the data (Data feed, vs. platform) you can get this from ninjatrader via their market replay which uses kenitic or you could run it live with your data feed, CQG, Rithmic, etc. They should all be about the same, except I have heard some things about IB's historical feed which leads me to believe that they will have less detail and possibly miss some data.
3. As far as building a trading strategy, there are a number of systems that have very user friendly APIs such as trade station, multi-charts, or the Managed approach with Ninjatrader. These systems can do 90% of what traders want. If you want to take the training wheels off then you can use an API that lets you do more at your own risk / demise. R- API, IB API, or Ninjatrader Unmanaged. Right now I am using the Ninjatrader unmanaged API. You can write anything you want, but you will have to handle all your own logic for if anything goes wrong, the kids gloves are fully off. But you can do more, so that's the trade off.
4. GOM: I believe this is a reference to a person named Gomi who makes and sells indicators for Ninjatrader. I am not too familiar with his work, but he is on this forum and has a lot of content out there.
5. For algo trading in general there is a member here named KevinDog who teaches algo trading. He has a website, book and some other information. He would be a good resource to start learning. Generally speaking if you are looking for coding tips and other algo trading resources, there is a good subreddit called Algotrading that has some useful information as well.
6. I have done some things with ML / AI in the past. This is a very challenging space for trading though because these tools are meant to solve a problem based on largely defining a correlation between dependent and independent variables. Most that attempt this type of work get the main part wrong and feed the wrong data into it, so it becomes garbage in, garbage out. I have an old trading journal on here that I posted some information about one of my old systems that uses a cartesian join with a volatility classification matrix to optimize different risk reward bets. I didn't document it 100% on here, but I put the general idea out there with some screen shots, downloads of the output. You may find this interesting.
I 100% agree with your last point. This is the way that I frame trading research, I look at it from the perspective of betting. What bet are you making, why are you making it, and does it have a viable edge over the long term. Most trading schools of thoughts can barely define the bet, let alone quantify the statistics for it. So the best place to start is with the microstructure and just run some tests to see how it moves, and if there are correlations that exist that you can exploit. There are, but they are not what most people expect, and the real secret sauces are virtually unknown to most traders because they never look at the raw data.
Hope some of this can be useful on your journey.
Best of luck
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
I am not familiar with those platforms, so maybe others could help you with extracting data from those.
If you want free, and easy, then I recommend downloading NinjaTrader.... I can give you a few different scripts you can run with their platform where you can extract a variety of different data.
What data are you looking for specifically?
You also may want to just start by defining the rules you want to use for trading, and figure out a basic betting strategy. You can do this without data.... You just have to understand order types, the mechanics of an exchange and some basic modeling technical and probability.
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
Nice! Yeah - that would be great. I'm probably only looking for tick data right now (having bid/ask prices - would be nice to know if it's an uptick/downtick, as well as volume traded + timestamp). Of course if you can also point me to extracting the orderbook, that might come in handy but probably won't use it initially (I know it will be a lot of data). But the main thing is the ability to get access to the data.
BTW: Ran into this today which I figured sounded something similar to what you were trading:
If you are comfortable using Ninjatrader you can use this basic code I have enclosed to extract bid / ask price and volume data as well as transacted volume.
Run this on a 1 tick time series and capture the output to the output window. Once you have a few days of data, you can select all the data from the output window via Control A, and the copy Control C: and paste this into Excel. The data will paste into one single column, so to break this up you will have to delimit it. I have already included delimiter logic in the code, so all you have to do is the following:
Data > Text to Columns> Delimit > Select Other and in the box input / I have every column separated like this in the code:
This will get you started. It's not the full micro-structure by any means, but this is a fair start. I would be willing to show you more once you get the hang of this...
And to the point you made about the link you posted: I used to trade 100% off of volatility using a similar method that was more advanced that the one described in the article. For example
Since there have been a few folks interested in some of my research regarding prop betting / volatility trading I figure I would add a little more color to the conversation to explain exactly how I approach this. This should be of keen interest to a …
That was a long time ago and I don't trade like that anymore. It definitely has a solid upside but my models showed that the extreme draw down scenario was close to $10k per month in rare cases.... I never ran it live because even though it modeled 6 figures per year on paper, overall it did have risks I wasn't willing to take up front. But I likely took this concept further than most with the info in the post in the link. Interesting read if you have the time.
These days I am modeling all my trading based off of microstructure bets I can pull off given my latency, and lack of HF infrastructure. So just the bread crumbs basically.
Ian
In the analytical world there is no such thing as art, there is only the science you know and the science you don't know. Characterizing the science you don't know as "art" is a fools game.
NT does it better than most, but they only move you up in the queue if the entire queue moves below your position, so you are essentially only ever filled from the back which is very pessimistic. Also keep in mind latency modeling. The market replay tool uses 0 latency for order executions. By contrast their live SIM has around 100 - 200 millisecond latency on every order. These are not settings that you can change, and neither were realistic to my use case... But these can impact your result considerably depending on what you are trying to do.