Outside the Box and then some.... - futures io
futures io futures trading



Outside the Box and then some....


Discussion in Trading Journals

Updated
      Top Posters
    1. looks_one iantg with 66 posts (321 thanks)
    2. looks_two artemiso with 12 posts (34 thanks)
    3. looks_3 SMCJB with 12 posts (15 thanks)
    4. looks_4 pen15 with 10 posts (2 thanks)
      Best Posters
    1. looks_one iantg with 4.9 thanks per post
    2. looks_two wldman with 3.7 thanks per post
    3. looks_3 artemiso with 2.8 thanks per post
    4. looks_4 SMCJB with 1.3 thanks per post
    1. trending_up 18,973 views
    2. thumb_up 406 thanks given
    3. group 65 followers
    1. forum 137 posts
    2. attach_file 52 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Outside the Box and then some....

(login for full post details)
  #1 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

First let me say that I am not a day trader, I am a programmer, statistician and mathematician by trade. I started analyzing the futures markets about 10 years ago as more of a hobby / gag just to bust on a friend that though he could predict the market with some charting tools. He would always tell me he knew exactly the direction the market would move just because he could look at old charts with lots of sloppy looking overlapped objects and such.....

So fast forward a few years later and I started programming some basic entry / exit logic in excel and I thought I had something pretty neat. I was just following simple moving average lines, but I thought it was neat that I stumbled onto without having any prior knowledge of trading. Like most things I tried early on it failed miserably in the long run.

About 3 or 4 years ago I got board over a long weekend so I wrote a program that would extract price data from yahoo finance and put it into a SQL Server I had. From there I build a linear regression based system to enter and exit based on a few things. It was an interesting project and I actually got the system to break even, but once you factored in commissions and all...... But keep in mind I had never actually seen a real trading platform, I was just making stuff up for fun.

About 2 years ago a friend of mind told me about ninja trader and asked me if I would take a look at it and program hist trading system for him. So I downloaded it and started playing around with it. It was very easy to use, C# is one of my more commonly used languages, so I figured I would have a go at it. Like a lot of people his system was not programmable because it relied on a lot of discretionary rules of thumb but no real binary parameters, so I never built him that system but I did however learn how to program in ninjatrader.

So for a little over 2 years now I have been hacking around in ninja trader and I have had a ton of experiences that think would be very interesting to share. As I have sim traded somewhere north of 1,000,000 trades and written over 1,000 different strategies all with varying degrees of success.

More recently I have developed several strategies that are highly profitable via Sim trading so I will likely roll the dice with a live account in the coming months.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 18 users say Thank You to iantg for this post:

Journal Challenge February 2021 results (so far):
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(165 thanks from 30 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(111 thanks from 32 posts)
looks_3Wisdom is Emptinessby Mtype
(68 thanks from 25 posts)
looks_4Deeteeís DAX Trading Journal (time based)by Deetee
(31 thanks from 16 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


Observation 1: AKA my golden rule.

Let me start my blog by saying after years of backtesting my most conclusive finding that may be helpful to other traders is this: Everything will work some of the time but nothing will work all the time.

This applies to any type of entry or exit system via technical analysis, indicator, etc. The concrete proof of this is evident by the very nature of trading in markets with more than 100 participants. Think about it this way... if you put 100 people in the room and tried to get them to agree on trading philosophy odds are they would not. They would be divided between various trading styles, and for even those that agree they may trade different time frames, use different target sizes, etc. So the outcome would be that these 100 people in a controlled simulation would move prices randomly by employing various strategies that cancel each out. Compound this by a factor of 1 million and you have a real market. For this reason I stand by the claim that no entry or exit system will ever consistently pick winners. The market moves randomly in the long run and there is no way that any crystal ball can predict this.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 10 users say Thank You to iantg for this post:
 
(login for full post details)
  #4 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Back when I started out, the first system that I thought I found success with was based on using Line Breaks. In the end I realized that I just found a way to trick the ninjatrader simulator and that the system produced a false positive. For a while though it seemed rather clever and some of the concepts I came up with at the time weren't that far off base, but I abandoned the system long ago and moved on, so I will share my system and the ninjatrader glitch that I was able to produce a false positive with.

Using a line break system I put together a pretty decent system for entries:

Entries: for 3 line break settings: After a single bar color change use a stop order to enter in the direction of the new trend.
Exit: I built a programmed trailing stop exit using a profit ladder that would access the current profit position and moving the stop loss up by a tick or two at each level. For example if profit = 4 move stop loss to 3, then when profit hits 5 move the stop loss to 4, etc. I ran this about 5 to 10 levels, I forget exactly what the recipe was. If anyone is interested I may just dig out the old code and post it. But the effect that it would have would be to protect your profits.

Now for the NT bug: In the strategy analyzer the simulator does not have access to every tick within line break bar types so it can only see a nice string of green bars followed by a nice string of red bars, etc. So the effect will always be with decent entries followed by at least one or two bars in your favor you will essentially never lose. I think the system was like 75% winners. I tested it over a few months and it never lost a single day, so I figured it was too good to be true, and it was. When I finally tested it with market replay... BOOM! All the ticks that were previously invisible came in and kicked my ass. So instead of getting a run of 5 to 10 bars before my exit got triggered with a heavy profit, I would get a 20 point swing against me inside of a bar that was still going my direction and get knocked out before the line break streak continued. I think line breaks are neat and likely the best visual way to trade a trending system, but you have to have some serious room to play.

I think in retrospect I didn't have enough appetite for a large enough stop loss. I think a stop loss of about 2 to 3 times the size of your profit target would be needed to keep you in the game. Alternatively you could just stay in until the line break bar switches colors I think I tested this some in market replay with decent results but I remember like every other trend following system I crushed it on days where there were trends but on sideways days it got killed.

This was my last attempt at traditional trading. On to the next story and lesson.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 5 users say Thank You to iantg for this post:
 
(login for full post details)
  #5 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

After trying every possible type of entry / exit based system using past to try to predict future I got board one day and programmed the martingale into ninja trader just to see what would happen with an unlimited bank role and just how much skin you would need to have in the game to win so to speak. For those unfamiliar with this it is a negative progression betting system that was used with varying degrees of success to gamble with in casinos. In recent years casinos have raised the table minimums and lowered the table maximums to combat this system. Without this dirty trick Bill Gates could walk into any casino and bankrupt them in one night.

So just for fun, I built it to run via ninjatrader and I saw some pretty neat stuff. With the default margin account of 100K you will typically get up around 30K to 50K inside of the first month, but usually within 3 months you will hit the big loosing progression that produces the dreaded "You don't have enough buying power in your margin account to complete the trade" or something to that effect. This means game over you lose. Your upside down more than 100K and to dig out of the hole you just fell in you would need another 200k. Just for fun I played with a few different starting margin account sizes to see what it would take to "beat the game". I only tested a year, and I would highly discourage anyone from ever doing this but starting out with a margin account of 10,000,000, I was able to beat it and make it through without a margin call. If anyone wants that code I would be glad to share. But use it or reverse engineer it at your own risk / demise.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 7 users say Thank You to iantg for this post:
 
(login for full post details)
  #6 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

What I will share of my current strategy is that it is all based on statistics. Anyone looking to trade seriously needs to understand the concept of "the house edge". If you play games in Vegas you have at best around 49% chance of winning in the long-run in any game. The first thing I did was deconstruct what the house edge is in trading. This may be helpful to those starting out, for more advanced traders this is nothing new.

1. Commissions: This makes any system with 50% wining trades or less a loosing system right off the bat. Depending on which market and how close your profit target is to your commission line this will cut your expectancy down significantly.

2. Market Orders: If you are entering a trade, you lose a tick or 2 right off the bat, so if your PT and SL are = your chances of losing are higher than your chances of winning.

3. Slippage: This is a wild card and it may be positive or negative or non existent but this can be a potential negative for you so I will add it to the house edge.

So all in: Using market orders and betting with an even stop loss relative to your profit target you are pretty screwed. In the long run the house edge will be close to 60% to 65%. I have tested every possible flavor of this on 5 different futures market and get the same results.

Some times the dice will roll in your favor and you may seem to beat these odds, but before you bet with real money you need to test your system with at least 1 years worth of back-testing to see if your system can beat the built in "house edge"

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 8 users say Thank You to iantg for this post:
 
(login for full post details)
  #7 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

After data mining and testing for a year or so I thought I finally cracked it. I started producing quarterly backrest results like the following:



Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to iantg for this post:
 
(login for full post details)
  #8 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

As I mentioned the market has a built in house edge of about 60% to 65% if you are doing the following:

1. Even odds (Same value) with PT / SL
2. You are using market orders


While I won't give up the secret sauce that I use to get my edge, I will share with you a few generic ideas that you could use to deconstruct and find your own edge.

1. Each instrument has a built in bias for a normal range that it will move in. Some move up and down in 10,20,50 point or tick increments, others more and others less. Understand this and exploit this. Going 50% / 50% with your PT / SL will almost always produce loosers, but if you go 2 to 1, 1 to 2, 2 to 3, 3 to 2, etc with the right settings you can exploit the fact that market you are trading has a build in bias to stay inside or outside your favorable range. For example if a market only moves up or down 10 ticks normally and on rare occasions it will move up or down 15 to 20 ticks, using a PL of 10 ticks vs a SL of 20 ticks would beat the statistical bet. You are beting 2 to 1 which moves your natural odds of winning from 50% to 75%, but all things even if you hit the 25% loosers with 2X the SL value you are just as screwed as if you had gone 50% / 50%. However, and this is the key if you can exploit a market with a built in bias to favor one range over another you may get > 75% winners using 2 to 1 odds for example.

Find this and you will find a winning system!

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 12 users say Thank You to iantg for this post:
 
(login for full post details)
  #9 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

I have seen a lot of discretionary traders lose big by moving from monopoly money to real money far too fast. As someone who has backtested and built countless systems here is my advise.

Before you put down one dollar of your own money ask your self this seriously: Do you have a system or not?

if your "System" involves the following rules: When X occurs go long, when Y occurs go short, when Z target is reached get out. Then ask your self this important question: in real time trading if you try this "System" and hit a loser or a series of losers how likely will you diverge from your own "system" and do something completely different? If you are 0% likely to change your rules, then you have something tangible, you pass the test and you should be ready to trade with real money. But if on the other hand you will likely diverge from your rules the first time you hit a few losers than you should not be trading.

I have seen way too many idiots in trading chat rooms with no real hard rules, but 100's of rules of thumb that are vague and loosely defined. And as soon as they hit a loser or two they abandon their rules and start making up new things and just clicking randomly trying to get in and out of trades as if they were children.

The first thing you have to accept is that trading is gambling and you can not win every time. Every system at best can only be slightly better than 50% / 50% when normalized for the spread between your betting parameters. If you have a PT of 5 and an SL of 10 you will likely hit 60% + winners easily, but when you normalize this you will still be somewhere around 50% / 50% if you are lucky and good. This is okay though. As long as you believe that your system will perform well over time and you will not abandon it at after the first bad trade you will hit your statistics in the long run.

I advise everyone to stick with their rules every time for every trade or else you don't have a "real system" and shouldn't' be trading.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 11 users say Thank You to iantg for this post:
 
(login for full post details)
  #10 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


I have always been amazed by the various nuances of the time frames available to traders and the fractal nature of it all. Everyone has their own reasons for using their own preferential time frames. For manual traders 5 minute, 10 minute and longer time frames are nice for point and click control, but for automated traders tick level data provides more volume and earlier signals.

I don't use technical indicators or charts for my trading system but I do use limit orders. After lots of trials and errors I realized that limit orders can help give you a nice edge or they can screw you depending on where you are in the queue. If you get in line too late you will never catch any of the good moves, where price temporarily swings down to the trigger and then quickly moves up to your profit target. This action is only available to traders that were in the front of the queue, so unless you are using a longer resting limit you will never see these Instead the limit orders that will always get filled will be the ones that break through your trigger and hit you 1 to 2 ticks at least against you. These will be losers about 60% to 70% of the time.

How do you combat this? You have to have a resting limit order far enough in advance to get to the front of the queue. Here is where the great debate over time frames circles back to this topic. If you have a resting limit up on a 600 tick chart that resets every 10 bars, this will likely not be as effective as a resting limit in a 100 chart with a 30 bar reset. Depending on the volatility, the range, the volume and your limit target ( 1 tick, 5 ticks, etc.) you can find that you can catch the good limit orders better or worse using various settings.

I found a very nice recipe that works for me by oscillating between .5 to 3.5 ticks above or below the current price and resting between 10 to 50 bars all depending on the volume, the range, and a few other market dynamics.

The difference between using a successfully placed resting limit compared to (GetCurrentBid, or GetCurrentAsk) is stunning.

Cheers.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 11 users say Thank You to iantg for this post:
 
(login for full post details)
  #11 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

In the spirit of giving back and sharing...... I am enclosing in interesting strategy that you can incorporate into your automated NT trading. This will work in any market and it will typically work to give you a serious edge. Here is the concept: A static stop loss vs. a static profit target will give you predictable results but it suffers one major draw back and that is: You can't have both a high % winning system and a system that catches the long runs. You have to pick your poison in the form of either being a scalper and taking more winners at less profit per trade, or try to only catch "the big ones" and you will usually have far more small losses than big wins.

The code syntax I am enclosing will help you split the difference and get both. Start by setting your PT and SL close to even or with one slightly more than the other, and let these both be a decent size. From here, slip in my code below and you can scalp out small winners when the market would normally turn against you and not hit your full profit target. In times where the move went more in your direction, you may hit your full profit target or come close, but the idea is that having this in place will protect you.

If you are a scalper and you typically play a PT = 50% or less of your SL, then this is for you! Add this to your trading system to catch the larger trades you would normally have missed. My advise to scalpers is to is to set the first level of this one tick lower than you would normally set your PT. So if your normal PT = 3 for example, then your first level is: If your position has reached a positive avg price of 3 then move your SL to 2. If this was the only thing that ever occurred you would be much worse off then just having your normal PT at 3, but in reality you will hit multiple levels throughout your trade history and your average PT will actually be much higher than 3, and your winning % will be about the same. So you end up getting the best of both worlds.

For trades only looking for "The big one", I advise that you use this to give you a few levels below your "Big trade profit" you usually like to hit. This will take your average profit down slightly but increase your winning % enough to offset it.

If you have any questions hit me up and I will expand further. But for anyone that codes this is fairly straight forward.

Happy trading:









// Moving Profit Targets

//Shorts

if (Position.MarketPosition == MarketPosition.Short
&& Close[0] <= Position.AvgPrice + -4 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice - 3* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Short
&& Close[0] <= Position.AvgPrice + -5 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice - 4* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Short
&& Close[0] <= Position.AvgPrice + -6 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice - 5* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Short
&& Close[0] <= Position.AvgPrice + -7 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice - 6* TickSize, true);
}
if (Position.MarketPosition == MarketPosition.Short
&& Close[0] <= Position.AvgPrice + -8 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice - 7* TickSize, true);
}






// Now the Long Positions



if (Position.MarketPosition == MarketPosition.Long
&& Close[0] >= Position.AvgPrice + 4 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice + 3* TickSize, true);
}


//Next Long

if (Position.MarketPosition == MarketPosition.Long
&& Close[0] >= Position.AvgPrice + 5 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice + 4* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Long
&& Close[0] >= Position.AvgPrice + 6 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice + 5* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Long
&& Close[0] >= Position.AvgPrice + 7 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice + 6* TickSize, true);
}

if (Position.MarketPosition == MarketPosition.Long
&& Close[0] >= Position.AvgPrice + 8 * TickSize)
{
SetStopLoss("", CalculationMode.Price, Position.AvgPrice + 7* TickSize, true);
}


// Now reset back to defaults

if (Position.MarketPosition == MarketPosition.Flat)
{

SetProfitTarget("", CalculationMode.Ticks, 9);
SetStopLoss("", CalculationMode.Ticks, 9, false);
}

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 12 users say Thank You to iantg for this post:
 
(login for full post details)
  #12 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

This is an interesting topic that is often overlooked in trading, so I wanted to share my insights in the hopes that it can help some traders out there.

Any system no matter how good will always hit a losing streak. This is just unavoidable, and it should be expected. The best systems out there will only ever succeed by the smallest margins. So you need to know the following statistical likelihoods:

1. Every system will hit a string of losing trades. Taken out of context a few bad trades may lead people to abandon a perfectly good system. Losing streaks are unavoidable, losing days are unavoidable. This is part of the game, and rather than fighting this fact you need to embrace it. Does your 401k that makes you 2% to 5% each year do so by never losing a single day? No, quite the opposite. It likely looses half the time and wins half the time and squeeks by with the narrowest of margins to give you a small gain every quarter. Start thinking about trading this way and you will get much further.

2. How can you tell if your system will work in the long run or not? You need to test at least 12 months, but preferably 24 months or more. If you can win 20 months out of 24, and the losing months don't wipe you out, you have a viable system. There are plenty of traders that manually trade a few days, and make their decisions about the merit of their system based on that alone. A few days won't tell you anything. You need at least 1,000 trades over several months before you can even tell what a realistic winning percentage would be. Any good statistician will tell you only a large sample size will give you a statistical baseline for decision making.


If you have an automated trading system you have a huge advantage because you can quickly test 12-24 months and check your performance. Without this though, you will likely be making decisions about the merit of your system with too little data.

I have a wide range of systems that produce 6 figure returns each year, but they all lose around 40% - 50% of the days they trade. This is to be expected. The market does not move the same way each day, so embrace this fact instead of fighting it and you will see much better results in the long run.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 11 users say Thank You to iantg for this post:
 
(login for full post details)
  #13 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Some say draw downs are to be frowned upon and that any system that takes any significant loss should be abandoned.
To this: I give you exhibit A: (Q3 2015 ES with a net profit after commissions of > $30K)
and B: (part of Q4: 2015 ES with a net profit after commissions close to $9k).

Here is the lesson to be learned:
1: Draw downs are a part of trading and are unavoidable, and statistically speaking a system that wins 51% will theoretically do what Vegas does and make millions over time.
2. Draw downs are the payment you have to make to earn your eventual profit. Stay in the game for the long run and you can expect to survive the bad to realize the eventual good.
3. There is no such thing as the perfect system. At best, trading systems can only squeek out tiny profits over a long period of time. In the short run there will be bad, days, bad weeks and even potentially bad months. But in the long run a system with a 51% or better expectancy will pay off.

Attached below are a few screen shots from one of my many automated systems that run 24 hours a day. The daily results look crazy. Every other day is a loser, and some days are large losers, but if you look closer, you will notice that the weekly view looks better with fewer losing weeks, and if you look even closer you will see the monthly view is even better. If you step back further and look at it quarterly or annually you will see the real picture. A system that actually makes money!

Happy trading.

Attached Thumbnails
Click image for larger version

Name:	Q3 2015 Adaptive PTs.jpg
Views:	222
Size:	140.5 KB
ID:	203790   Click image for larger version

Name:	Weekly Q3 2015 Adaptive PTs.jpg
Views:	201
Size:	89.1 KB
ID:	203791   Click image for larger version

Name:	Monthly Q3 2015 Adaptive PTs.jpg
Views:	179
Size:	86.3 KB
ID:	203792   Click image for larger version

Name:	Q4 Adaptive PTs Cummultive.jpg
Views:	226
Size:	123.0 KB
ID:	203793   Click image for larger version

Name:	Q4 Adaptive PTs.jpg
Views:	191
Size:	124.1 KB
ID:	203794  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 8 users say Thank You to iantg for this post:
 
(login for full post details)
  #14 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

I figure this topic would help some trend traders as it is a very popular topic. One of the keys to being successful in trend trading is to know when it is appropriate to use a trend trading tool box. In markets that move up and down there are variety of tools that work fairly well, but in sideways markets they completely break down and fail. So the first rule is to define when markets are trending and when they are ranging. Here are some of the ways you can easily achieve this:

1. Time filters: Markets tend to trend during the US open for around the first 2 or 3 hours and then it's a coin toss from there. If you are trading an auto-strategy you can just use a time based parameter and only trade between X and Y hours. Fairly easy to program and run and it should get you at least 60% trending market conditions.

2. ADX: This is the only indicator I know of that claims to tell when the market is trending or not. it does a decent job from what I have seen. There are a variety of settings to play with but if you used a larger time frame say 10 minutes or so, you could tell if the market was in an up trend or down trend and how strong it was. If the trend was below a specific level you could set your code to just not take trades, this would essentially filter out sideways action.

3. Range Filtering: One of the easiest ways to find out if you have favorable conditions for trend trading is to analyze the range of the last N number of bars. Just call a few variables such like this:
double point1 =MAX(35)[0];
double point2 = MIN(35)[0] ;
double range = point1 - point2;

if(range >10)
{
//Enter trade
}

Using something like this you can only enter trades when there is enough movement in the market. If you are trying to pick off a profit target of 10 ticks for example, you will likely fail if the market is only moving in a tight range of 5 ticks over the last 35 bars. So a better strategy would be to just filter these periods of sideways movement out, sit on the sidelines and wait until the market has a little more to work with. If you have a market moving 20 ticks in the last 35 bars for example, then you will be more likely to get 10 ticks in your favor with the right additional setups.


Just throwing out some ideas that may be helpful.

Happy Trading!

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 18 users say Thank You to iantg for this post:
 
(login for full post details)
  #15 (permalink)
Chicago, USA
 
Experience: Advanced
Platform: TS, NT, TOS, SSE, IRT
Trading: ES, currency futures, options, stocks
 
Posts: 195 since Dec 2010
Thanks: 328 given, 273 received

Thanks for the great posts. I especially like your Moving targets/stops code. Thanks for sharing. I never liked Trailing Stops, but this piece of code may work better. I will check it out. I do like to move stops as soon as a reasonable move is made though, otherwise, you will be holding a loss in what could be a smaller winning trade if you had moved your stop. Of course, as you have shown in your code, you need to leave some wiggle room for the trade to work, but let the trade come back and take you at a loss is not good to any strategy.

Filtering choppy markets is one of the toughest things. I never found any value on ADX, as it also filters some good runs and you are almost always late in your entry. Another suggestion is to use a synthetic bar (UniRenko for example). Try it, see how it performs on your strategies (btw, synthetic bars donít work well with the Strategy Analyzer, results are way off to the optimistic side). I like UniRenko and a modified version that I use on my own tests.

Another thing I can suggest (if you have already not done so) looking at your post 13 charts is that you introduce a Ďdaily loss limití, and check how it will affect your overall P&L curve. I learned that for some strategies, days that are bad are really bad and limiting the losses on those days can be very helpful.

Like you I couldnít find much value using the Strategy Analyzer (SA) or Optimizer tools of NT. I do use them at a very high level, maybe to give me an idea if a certain MA is better than another, but I always go thru Market Replay. I also use SA sometimes with intrabar code in my strategy so I can sometimes get better results but as usual they are way off or way too optimistic. Market Replay makes NT much more powerful than Tradestation to test intraday strategies (or any other platform to my knowledge), and NT is the only tool I use for testing intraday strategies. I also use Tradestation for stocks (which is great btw as you can use many market breadth indexes and variables in your strategy) and VectorVest to test portfolio strategies.

Reply With Quote
 
(login for full post details)
  #16 (permalink)
Vancouver, BC
 
Experience: Beginner
Platform: Ninjatrader
Trading: emini's
 
Posts: 33 since Nov 2011
Thanks: 15 given, 18 received

Thanks for this awesome thread, iantg. Just goes to show how freakishly hard it is to create something viable and profitable. I wonder if you have a specific way to deal with chop? Chop is my ultimate nemesis. I'm tempted to use Renko bars but they hide real price movement so I don't see how it's possible to auto-trade with them in a strategy. I tend to favour Range bars, but even they have their (significant) limitations.

Reply With Quote
The following user says Thank You to w00dmann for this post:
 
(login for full post details)
  #17 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

w00dmann and dstrader,

Thanks for reading my posts and taking an interest in some of my ideas. I appreciate your dialog and contribution to the topic of filtering through choppy / sideways markets. I think this is one of the most important topics out there, so perhaps I should expand on my thoughts further in the hopes that it can help out.

Let me first state for the record that I am not a trend trader, and my style of trading is completely directionless from the market. So my current success has little to do with what I am about to recommend. That being said, I used to successfully trend trade before I arrived at my current trading system.

Here are my observations over several years of trend trading:

1. There is no silver bullet in predicting the future direction of the market. All indicators / technical analysis and any other attempts at predicting the future will have the same 3 cycles:

A: The Current market conditions have just changed (A down trend is now an uptrend, or an uptrend is now a downtrend, or a strong direction market is now a sideways market, etc.) Because this measurable change event has occurred I should now do X. (In this scenario traders often take entry or exit positions based on what I will refer to as Change Event 1. The hope is that the past condition before Change Event 1 will stay in the past and the new condition that has started will continue. Because the market is made up of thousands of participants that see different things in different time frame I will simply state that Change Event 1 has around a 50% / 50% chance of materializing in your favor. Some things may seem to work better than others in the short term, but when doing a large scale back-testing of several years and hundreds of thousands of trades you will find that 50% / 50% odds is a pretty fair characterization because half of the moves in the market are fake-outs and have the moves are break-outs.

B. Once Change Event 1 has occurred, People will now use their technical analysis to predict that current market conditions will continue (an uptrend will stay an uptrend / a downtrend will stay a downtrend, etc.). Again, sometimes trends run long sometimes trends run short, but I would say you have around a 50% / 50% chance again. I will characterize this trading cycle as Continuation 1.

C. Once Chance event 1 and continuation 1 have occurred traders now have a difficult decision to make. They now have to decide if Continuation 2 will occur to keep the current trend in place or if Change Event 2 will occur to reverse the position. Given the obvious fact that there are two binary outcomes here I will once again place the odds of success at 50% / 50%.

If you take away all the fancy indicators and trading jargon this is essentially the most minimalist view of the market and how it behaves in relation to any traders attempt to understand it or predict it. I won't discourage people that believe they have found a way to break the 50% / 50% odds in any of the cycles that I have mentioned, but if you normalize everything in the most simple way this is the reality of what you are up against.

All this being said, I believe there are only two real aspects of past market behavior that carry any level of accuracy or reverence to future market behavior. And these only work outside of critical event time frames such as market opens, market closes, and major news releases: But here they are:

1. Volume: Outside of the previously mentioned critical event times, volumes tend to gradually increase and decrease. Therefore light or heavy volume in the immediate past should predict light or heavy volume in the immediate future. Unless you are near a critical event time frame, volume rises and falls very gradually typically.

2. Trading Range: Outside of critical event times, a tight trading range will typically stay a tight trading range and a large trading range will stay a large trading range. If the last 50 bars had a range of 10 points it is more likely that the next 50 bars will also trade within 10 points and far less likely that it will trade within 100 points. Like volume, the trading ranges tend to rise and fall gradually outside of critical event times.

Nothing else will be a consistent predictor of the future market other than these. So the way to use these to your advantage is to us the rule of thumb to stay out of markets with light volume and thin ranges, and only trade markets with heavy volume and larger ranges. Even these are not silver bullets, so you will never get 75% or 90% accuracy with either of these, but you will beat the 50% mark constantly, and after years of testing I have observed these are the two most consistent predictors of what the market will do next. Unfortunately they do not apply at all to direction, but you should be able to use these to filter out thin ranges in the market > 50% of the time. If I had to rate this statistically I would say the overall accuracy is around 60%.

Hope this helps a little.

Happy Trading

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 10 users say Thank You to iantg for this post:
 
(login for full post details)
  #18 (permalink)
Paris France
 
Experience: Beginner
Platform: TradingView
Broker: CQG
Trading: ES1!, CL1!, DY1!
 
Pharmakon's Avatar
 
Posts: 45 since Jan 2016
Thanks: 119 given, 27 received

Hi !


First of all, I'd like to let you know that English isn't my mother tongue. So I beg your apologies if my English isn't so fluent as you.

I found your posts very insightful. I appreciate greatly your approach and methodology in market studies. I am very thankful about the time you spend to share your works. And, I have some questions to ask you.

I used to be a demo discretionary trader but I decided to look more about automated or mechanical strategy to trade market with real money. The main purpose to do this is to find the best reasonable way to trade market and to develop my system as much as precise I can. To limit any form of gambling. When I trade, I'd like to have strict rules to respect and to follow to avoid any emotional bias. So I, first try to find a logical model close to my trading strategy. It give me a reliable way to know at any time what to do : enter, hold on or exit markets. In doing so, I found that it will be easy to automate it with a small script to test my strategy on historical data. And so, this is what I got :



Results are mostly same on SPX500 and DAX. It's a small logical code that I could give you if you want. According to your definition of a good system of trading. This strategy wouldn't be a good one because of the lack of 20/24 of wining months even if it used to be profitable. So what do you think ? Could I expect any future positive return any longer with this strategy ?


Best Regards,
P.

Follow me on Twitter Reply With Quote
 
(login for full post details)
  #19 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Pharmakon thanks for reading my posts. I may have over simplified or overstated the exact ratio of winning months to losing months (20 out of 24). There is no real hard line in the sand per se, other than you need to make more than you lose in the long run. I think the concept of "The long Run" is relative to everyone.

Let me explain my thinking and where I got my 20 out of 24 months idea from. If you want to fund your margin account with a fairly low amount: Let's say $20K USD. Then you have around $10K to $15K to risk beyond your required margin account minimum just to be in the game. If your average monthly draw down on your bad months is around -$5,000 to $,8000 USD, then as long as your testing / market replay results hold up in real trading, you could theoretically weather the storm even if you hit your worse two months right off the bat. Your originally $20K could potentially survive and in month 3 you climb out of the hole. The reason I like to see at least 24 months of history is to know if winning 10 out of 12 months is statistically likely in the long run or if it is just an anomaly observed in the year you tested. Ideally more years and more testing would be nice, but everyone has a since of how much risk they are willing to take based on what they see from testing their systems.

To me, if I can get 10 out of 12 winning months for 2 years or more, I know that under a worse case scenario if I hit my two losing months as soon as I go in with real money, I can survive with only around $20k based on my trading systems and how they work. Everyone will have to adjust their own systems to find that sweat spot but this is just my preference.

If someone theoretically had $100K to play with and statistically they won 7 months out of the year, they will make money in the long run, but there is a real risk that the $100K will get knocked down close to a margin call at certain points under worse case scenario conditions.

Most people are fairly risk adverse when it comes to their money, so if a system loses them money the first month they trade, they will likely shut down their margin account and walk away. Now if you know that statistically speaking under very large scale market replay / backtesting your system wins 10 out of 12 months, you will be more likely to stay the course and not just shut down your account and give up. Odds are if you lost during the first month you jumped in, you would likely win the next 8- 10 months in a row before you hit another losing month. To me this would be the type of testing statistics I would need to keep me in the game. There may be others who could watch a system lose 4 or 5 months in a row right off the bat, but my guess is this is less than 1% of traders.

So it just comes down to how much skin you want to have in the game, and what that looks like relative to your trading style and system.

Hope this helps. If you have any other questions let me know.

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #20 (permalink)
montreal quebec/canada
 
Experience: Beginner
Platform: CQG/NT7
Broker: IB/AMP/CQG
Trading: CL
 
Okina's Avatar
 
Posts: 2,149 since Sep 2015
Thanks: 758 given, 5,464 received

I find this post very intresting. Thank you for sharing.

I'm not trading with bots so maybe I will make a stupid remarks.

I found little value in bots for retails investors because IMO their main added value is the fact that they give you the opportunitie to trade 50 market with only 1 or 2 traders and a small crew of IT guys vs having to hire 50 goods traders (which is difficult to find and fairly expensive to fund). So sometime if you have lots of equity it is better to be just average on 50 market than good on 1 or 2 ? But for us in general having too much $ to invest in not our main problem?

For the same reason I found them efficient in the equity world since they can scan thousands of stocks per second to find abnormal prices (which you can not do with one human brain). I remember one Guy in Japan who has made at the beginning of his career more than 1M$ in one trade because his bot has spotted a fat finger mistake from a broker who instead of selling 1 share at 600k yens has entered a sale of 600k shares at 1 yen... it was a very illiquid stock so he had the time to purchase a large block of them.

Other than that I don't see any added value except of course the purely intellectual challenge ?

Cheers

Okina Tora.

R.I.P. Olivier Terrier (aka "Okina"), 1969-2016.
Please visit this thread for more information.
Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #21 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Okina, thanks for reading and thanks for adding your perspective and interesting story! There are lots of opinions between discretionary trading vs. automated trading, and there are no right or wrong opinions as everyone has their own style or preference.

The key reasons that I enjoy automated trading so much are as follows:

1. Quicker entries and exits. My programming can do do things that I can not do as fast because it sees and analyzes conditions that would take me far longer to compute manually.

2. 24 hour a day trading vs. how ever many free hours I am willing to do it.

3. Any rules for trading that can be thought of can be programmed, though you have to be a fairly good programmer to do some of the more complex stuff.

4. Visibility to process or incorporate more information into your trading decisions vs. what a human can do. For example a computer can enter, exit or add or subtract contracts based on an infinite number of very specific conditions being true, but as a chart trader looking at a screen I could only ever catch a few conditions before it would be too late to make my move.

5. The ability to test the merit of different trading ideas: This is the big one for me. As a chart / discretionary trader you can only test as much as you can physically sit through in real time. So naturally with only so many hours in the day to dedicate to sim trading you can only really vet / test a system, or idea so much. With programming a system to trade for you it is possible to test it over hundreds of thousands of trades over several years.


I have never thought of an idea or concept for trading that I couldn't program, but I am a professional programmer first and foremost, so I have a slight edge over casual traders that are trying to hack around in C# and can only do basic things.

Just a few pro's that I have noticed that I wanted to throw out there for anyone that is interested in automated trading.


Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to iantg for this post:
 
(login for full post details)
  #22 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

So just for fun, I found a hack or cheat code if you will, that will allow you to profit an obscene amount of money using the ninja trader market replay in NT 7. Here are the settings:

1. Market Instrument = ES, though this would likely work in any market with a little bit of sizing work.
2. Date Range: (Any week in the last x years, it really doesn't matter, they all have the same results)
3. Time Settings: 300 tick to 1000 tick or anything in between seems to work find. I haven't tested much else but I assume they will get the same results.
4. Ninja Trader Settings: Enforce Immediate Fills = checked or true


I am just going to throw this out there to anyone that is interested in testing this and obviously there is a catch here, but at the same time quite a bit of practical application and theory that can be gained from this exercise.

I am interested to see some different perspectives from this community about why this works, what the catch is and hopefully some insight into practical application. I have already developed several real trading systems that can beat any sim on any market and some of them use a basic concept that can be learned from this. So obviously I know the answer the answer to the question: "What if anything can be useful from this, and how can it be applied in real trading". I am interested to see if anyone out here can grasp it and understand it.

For those that can, there may be a chance for future collaboration.

Happy Trading

Ian

Attached Files
Register to download File Type: cs EvenOddsBaseline.cs (2.6 KB, 71 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #23 (permalink)
Estonia
 
Experience: None
Platform: windows nt
Broker: windows nt
Trading: everything what moves
 
Posts: 10 since Mar 2014
Thanks: 170 given, 12 received

Lets kick it off then, this helps favor the odds

- SetStopLoss("", CalculationMode.Ticks, 3.5, false);
+ SetTrailStop("", CalculationMode.Ticks, 3.5, false);

Reply With Quote
 
(login for full post details)
  #24 (permalink)
Market Wizard
Los Angeles CA
 
Experience: Beginner
Platform: NT8
Trading: NQ YM ES ZN CL GC HG ZS 6E 6A 6J 6C
 
FlyingMonkey's Avatar
 
Posts: 654 since Jan 2016
Thanks: 2,968 given, 2,282 received

Edge is dependent on turning off NT's simulated fill engine (enable immediate fills = true). This alone will tip scales somewhat.

Additionally, placing orders into the engine at half-points off the last price could encourage long fills at bid and short fills at ask, as price must move toward the order.

? Maybe. Don't have nt7 installed at the moment, so this is without actually testing the code.

Real-world strategies can benefit by being on the right side of the noise, scraping a bit of it up into the PnL.

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to FlyingMonkey for this post:
 
(login for full post details)
  #25 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received

Slightly off topic... my favorite tshirt...


Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #26 (permalink)
Tulsa oklahoma
 
Experience: Intermediate
Platform: Ninjatrader
Trading: Emini Es
 
Posts: 2 since Apr 2016
Thanks: 1 given, 0 received

Very interesting concepts!!

Reply With Quote
 
(login for full post details)
  #27 (permalink)
Tulsa oklahoma
 
Experience: Intermediate
Platform: Ninjatrader
Trading: Emini Es
 
Posts: 2 since Apr 2016
Thanks: 1 given, 0 received

I tried to pm you, however I am not sure you received due to posting minimum?? I just have a few questions regarding one of your concepts... Ty

Reply With Quote
 
(login for full post details)
  #28 (permalink)
Paris France
 
Experience: Beginner
Platform: TradingView
Broker: CQG
Trading: ES1!, CL1!, DY1!
 
Pharmakon's Avatar
 
Posts: 45 since Jan 2016
Thanks: 119 given, 27 received


Bhenry29 View Post
I tried to pm you, however I am not sure you received due to posting minimum?? I just have a few questions regarding one of your concepts... Ty

Could you ask your questions there ? It may be good to share you thoughts to other people there.

Follow me on Twitter Reply With Quote
 
(login for full post details)
  #29 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

It's been a while since I have been on here. But I figured I would share a little more, get a little feedback, etc.. I will be posting some information about some algo trading I do and be providing a little bit of insight here and there. But not too much obviously. I have to have some secret sauce for my self.

The first system I am going to be sharing a little bit of is one I have been testing recently on NinjaTrader 7 market replay. It's been fairly profitable from my initial testing. I am still trying to optimize it and put all the pieces together but I think I have a good start.

The concept behind this one is fairly interesting, It involves some simple machine learning, and I am eventually going to connect this to a SQL database so I can improve the processing power. Right now it is doing a bit too much in memory. I don't use any off the shelf indicators or charts only a few KPIs and metrics I built. Below are some screen grabs from my latest test... (It took about 2 days to run all this) This is an example of a somewhat High Frequency Trading system. I say somewhat because I am only testing this with one contract. If I was trading 5-10 contracts then I suppose it might qualify as HFT system. I haven't seen many HFT discussions or posts on here, so I figured I would throw my hat in the ring.

This is one of about 30 different systems I have, (Each has around 50% in common and 50% variation with different metrics, logic, etc.) and they all have somewhat similar results in back testing. The goal is to take the parts that are working the best with each and combined them into a final algo. None of these are market or time frame specific. They will work with anything, I am still tweaking some of the conversions for various instruments, but I have tested the YM, ES, and NQ and after conversions I have these up and running. I have learned a lot from this process! Some stuff I will share, and some obviously I won't but feel free to take a peak at this journel from time to time because I will be posting some random insights and theories here and there from time to time.

Happy Trading!

Ian

Attached Thumbnails
Click image for larger version

Name:	MK1 250 Tick Monthly.png
Views:	184
Size:	166.3 KB
ID:	242610   Click image for larger version

Name:	MK1 Draw Down.png
Views:	152
Size:	219.1 KB
ID:	242611   Click image for larger version

Name:	MK1 Hourly Dist.png
Views:	150
Size:	240.4 KB
ID:	242612   Click image for larger version

Name:	MKI Weekly.png
Views:	146
Size:	213.9 KB
ID:	242613   Click image for larger version

Name:	MK1 Monthly Stats.png
Views:	186
Size:	295.4 KB
ID:	242614  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to iantg for this post:
 
(login for full post details)
  #30 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Enclosed are some screen grabs from another ninjatrader strategy I built. This one tends to take higher profit targets and lower stop losses on most trades. Though it is not a trend following system in any way, it has the type of statistics you would hope to find with a profitable trend following system. (If there is such a thing ). I ran this from Jan 2015 to Feb 2016. I had to break it into two sessions because it takes forever to run these in market replay. But overall this is hitting around $8-$10 dollars per trade. Draw downs are maxing out under $5K and cumulative profits are somewhere around $50K - $60K after commissions. This system is only trading one contract. So I imagine that I could scale this up fairly nicely. Just to be safe I would put $10K per contract just to manage draw dawns.

I will likely never trade this live though because out of all the systems I have this is one of my least favorite for a number of reasons. But someone asked me recently if all my systems were scalping: I.E (Lower Profit Targets, Higher Stop Losses) or something else. I don't scalp or follow trends or anything like that in the traditional sense, as I don't use charts or predict price movements. But this system, does look like a trending a system just by the stats: (Higher Profit Targets, Lower Stop Losses). So it can be done, but just not the way that I imagine most people are going about it.

Happy Trading.

Ian

Attached Thumbnails
Click image for larger version

Name:	WallClimber Backwards Monthly Overview Jan-Sep.png
Views:	117
Size:	162.9 KB
ID:	242857   Click image for larger version

Name:	WallClimber Backwards Monthly Overview Oct-Feb.png
Views:	93
Size:	146.5 KB
ID:	242858   Click image for larger version

Name:	WallClimber Backwards Details Jan-Sep 2015.png
Views:	132
Size:	261.1 KB
ID:	242859   Click image for larger version

Name:	WallClimber Backwards Details Oct- Feb.png
Views:	108
Size:	192.4 KB
ID:	242860   Click image for larger version

Name:	WallClimber Backwards DD Jan-Sep 2015.png
Views:	96
Size:	214.8 KB
ID:	242862   Click image for larger version

Name:	WallClimber Backwards DD Oct-Feb.png
Views:	95
Size:	221.1 KB
ID:	242863  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #31 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Even though these types of results will likely not be something easily repeatable for beginning traders, I do want to show this just as a lesson in managing the psychological side of the trading game.

Enclosed are screenshots of one of my various Algo systems: This was ran on NinjaTrader in Market Replay over a little more than a year. No cheating, no martingale, no checking the fill limit orders instantly box, and only using 1 contract.

If you look at the daily results, they are all over the place. It would be hard to even guess what this system netted out to since there is so much daily variability. But when you look at the weekly view, it cleans up a little bit. By the time you look at the monthly view and the hourly view, you realize the system is almost perfectly in the green. And out of 14 months tested all but 3 of the months would have carried a decent profit, (3 were basically break evens). The cumulative profit was around $80K and after commissions would have been around $50K-$60K.

And the Draw Downs never touched below $4k. But nearly every other day was a loser. Now I know most people aren't going to like the idea that every other day was a loser and every 4 or 5 days were big losers, but that is just part of trading. There is no way to avoid having outliers in your performance data both good and bad. You will need around 1,000 trades before they become statistically significant and hit your true performance statistics. Anything less that this will hold serious anomalies both good and bad.

The point is this: You are much more likely to have a successful system overall if you just stick with your system and do not deviate. A bad trade, or a bad day, will likely be followed by a good trade or a good day. But if you change your system and your rules for trading every 5 trades, then you really have no system and you are just getting emotional and pointing and clicking.

Empirical evidence enclosed.

Happy Trading!

Ian

Attached Thumbnails
Click image for larger version

Name:	SMPL2 2015 DD.jpg
Views:	117
Size:	61.7 KB
ID:	243079   Click image for larger version

Name:	SMPL2 Daily.jpg
Views:	116
Size:	73.9 KB
ID:	243080   Click image for larger version

Name:	SMPL2 2015 Weekly.jpg
Views:	121
Size:	75.8 KB
ID:	243081   Click image for larger version

Name:	SMPL2 2015 FY.jpg
Views:	122
Size:	66.5 KB
ID:	243082   Click image for larger version

Name:	SMPL2 Hourly.jpg
Views:	103
Size:	75.3 KB
ID:	243083   Click image for larger version

Name:	SMPL2 Net.jpg
Views:	129
Size:	61.7 KB
ID:	243084  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #32 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

2017 has had historically low volatility levels. These have been some of the worst trading conditions possible for futures traders. The market stops moving, everything is a correction, there are little to no trends, most indicators are useless, etc. Sound familiar? Most of my friends that trade futures got killed this year and have since moved on to other professions because of it.

Anyway.... I built a system that beats low volatility markets fairly well. Enclosed are a few NinjaTrader Market Replay screen grabs I ran yesterday.

One Contract only, the ES, Jan-Aug. (Very slow months). Somewhere around $20K after commissions, nothing to quit your day job over, but maybe with more contracts.... DD was almost nothing, so roll the dice?

The key: Base hits, not home runs....


Happy Trading

Ian

Attached Thumbnails
Click image for larger version

Name:	MatrixTFT17 Detals.png
Views:	137
Size:	46.2 KB
ID:	243152   Click image for larger version

Name:	MatrixTFT17 Monthly Graph.png
Views:	120
Size:	75.3 KB
ID:	243153   Click image for larger version

Name:	MatrixTFT17 Hourly Graph.png
Views:	101
Size:	78.9 KB
ID:	243154   Click image for larger version

Name:	MatrixTFT17 Weekly Graph.png
Views:	119
Size:	82.3 KB
ID:	243155   Click image for larger version

Name:	MatrixTFT17 DD Graph.png
Views:	106
Size:	107.9 KB
ID:	243156  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to iantg for this post:
 
(login for full post details)
  #33 (permalink)
France
 
 
Posts: 1 since Aug 2013
Thanks: 3 given, 0 received

Very interesting research

Thank you

Reply With Quote
 
(login for full post details)
  #34 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Greetings fellow traders,

Today I wanted to cover some basic blocking and tackling that I believe will help quite a bit of traders out there.

When I first started designing my algorithmic trading systems I was always perplexed by the fact that every time I built a new system I always hit the same issue. I would take a given day to back-test and almost curve fit a system that could beat that particular day very well. Then as soon as I started testing it on other days it would immediately blow up. Sometimes I would make it 2 or 3 days, sometimes a week, but ultimately they would all fail. And it took me a while to figure out why?

You can typically break any market down into 3 categories. Every expert trader has a way that they would characterize the markets. Some may bifurcate this into 2 types, 5 types, etc, but for me I see 3. Here they are: (Below I have images to illustrate)

1. High Volatility: This is characterized as a wide range of movement. The Max - Min over N period of time is high. There are a number of ways to quantify it, but they all tell you the same thing. This type of market works for just about any trend following system. Anything to do with this line crossing that line, or the idea that if something is moving in a certain way it will likely continue. Pundits often say this type market behavior is often found during the first several normal business hours the market is open. But of course this can break out at any point in time.

2. Low Volatility: This is characterized by having a very low range of movement. The Max- Min over N period of time is often very low. Everything is a correction of a correction in this type of market with no clear major trend or direction. This can occur at any time, but it often occurs in the middle of the day, or overnight.

3. Medium Volatility: This can be thought of as the middle ground between the two prior extremes, in a lot of cases this is hit during a transition from high to low or low to high volatility. There is no real typical time this is observed as it can break out at any time. This is the most commonly observed type of market behavior where there are some small trends, some sideways corrections and no clear constant pattern as this is a hybrid of the two more consistent patterned market behaviors previously listed.

So once I realized that the market only had 3 types, then I started realizing that most of my strategies were only built to handle one of these three types, and there was no consideration for quantifying what type of market was in play in the first place. I started realizing that instead of back testing months and years with only one type of system, I needed to create several different strategies that worked well in each type of system. Here I will draw some parallels to the classic video game Mike Tyson Punch Out.

1. Glass Joe: (Trending System): In Mike Tyson's punch out this was the proverbial tomato can first fight. Virtually everything you throw will work on this guy. It's a super easy win no matter what. I equate a high volatility day paired with any trending strategy the same way. Very easy and anyone can win. Take the Simple MA Cross over strategy that comes with NinjaTrader. This will beat 30% of all days in the market, and these are all high volatility days.

2. Kind Hippo: (Low Volatility System): This morbidly obese dude comes out throwing heavy punches but he is slow and he gapes for like 5 seconds giving you plenty of time to sock it to his mouth. I will pair this guy with the low volatility market because he is slow. When you are in a low volatility market it is better to try to call tops and bottoms based off of former resistance and support. Everything will typically correct once a previous high or low has been hit. In these types or markets counter trend traders clean up nicely.

3. Bald Bull: (Medium Volatility System): This dude was a pretty even fighter. He was big powerful and mixed things up. I would characterize the medium volatility similar to this. It is a mixed bag. There are some small trends, and some small to medium corrections. There is no real silver bullet. So your trend following system will win half the time and lose half the time and your mean reversal system will win half the time and lose half the time. For inexperienced traders my advice is to sit this out! For more seasoned traders that have some more experience, I think you can split this type of market into 2. More trending or more ranging. From here you can likely work a conversation system on either, but you won't make the same type of consistent profit you would with either the High or Low volatility systems.


Finally we have the ultimate challenge: Kid Dynamite, Mike Tyson himself. This dude was a straight up killer. The first round he just threw upper cuts a million times and any one of those would knock you out. The second round he mixed it up and the third round he mixed it up again. He was always changing and his transitions were unpredictable. But you could beat him, if you learned to read all the signals. The real market is exactly the same way. Most people are looking at the market from the lenses of a single trading style or system that beats only one type of market, and Kid Dynamite (AKA the real market) will straight up murk you!


So while I won't give too much away about my own personal system, I will offer this small bit of advice.

The conservative trader that picks his shots and is always careful will win. You need to know that no amount of past market data can 100% accurately predict the future. At best you can count on the immediate past predicting the immediate future in terms of only the type of market you are in. (High, Medium, or Low Volatility). Once you enter one of these you will typically have a small window where a certain tool box will apply. But the biggest challenge is when these switch from one extreme to the other. To this end, do the following:

1. High Volatility Markets: Use whatever signals you prefer to confirm you just recently entered into a high volatility market. From here place 1-2 trades using your most conservation type of profit target / stop loss. (I.E don't stay in the trade for ever, take profits and run.). Don't count on the trend to stay in tack for ever, get in take one or two trades and get out. A shift is certainly coming and you don't want to be on the wrong side of it.

2. Medium Volatility Markets: When the market shifts down to medium volatility from high you will be glad you were on the side lines. And you should remain on the sidelines. Don't trade this market at all. Just wait.

3. Low Volatility Market: When the market moves from Medium to Low volatility, put a few indicators to use to confirm, look both ways first, then place 1 -2 counter trend type of trades. Maybe go with a mean reversal, or or run some support / resistance type of system. After 1-2 trades just stop. This will likely ramp back up to Medium Volatility again shortly and you will get hit on the wrong side of it.

So that's it, just get in for a short 1-2 trades on High and Low Volatility markets, and wait for the rest. The good opportunities only typically come at the begging, and if you get too greedy and stay in for two long with the wrong type of system the market will change before you knew what hit you.

Happy Trading!


Ian

Attached Thumbnails
Click image for larger version

Name:	HighV.png
Views:	152
Size:	64.5 KB
ID:	243219   Click image for larger version

Name:	MediumV.png
Views:	152
Size:	68.1 KB
ID:	243220   Click image for larger version

Name:	LowV.png
Views:	148
Size:	96.6 KB
ID:	243221   Click image for larger version

Name:	MT1.jpg
Views:	98
Size:	13.3 KB
ID:	243224  
Attached Images
  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 10 users say Thank You to iantg for this post:
 
(login for full post details)
  #35 (permalink)
SF, CA/USA
 
Experience: Beginner
Platform: SC
Broker: Stage 5
Trading: NQ....that's what it boils down to
 
jackbravo's Avatar
 
Posts: 1,338 since Jun 2014
Thanks: 4,340 given, 2,389 received

Excellent post!

"It does not matter how slowly you go, as long as you do not stop." Confucius
Reply With Quote
 
(login for full post details)
  #36 (permalink)
Webinar Host
Indianoplace, IN
 
Experience: Intermediate
Platform: SierraChart
Broker: CQG
Trading: All Micros, especially the ones that move
 
Rrrracer's Avatar
 
Posts: 2,379 since Feb 2017
Thanks: 16,364 given, 8,974 received

Fascinating thread with great information. And I loved playing that game back in the day

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #37 (permalink)
New Orleans, La (Mardi Gras City)
 
Experience: Advanced
Platform: NinjaTrader
Broker: Ninjatrader / Optimus Futures / AmpFutures
Trading: ES / 6E / 6B / CL
 
DavidHP's Avatar
 
Posts: 1,347 since Aug 2009
Thanks: 9,425 given, 2,360 received

I'm not sure how I missed seeing this thread but I've just added it to my follow list.

Very good observation and the reason over the years that I have learned that automating the trades is great but blindly letting the system trade without my input is not.

Looking forward to further insights from you.

Thanks


iantg View Post
When I first started designing my algorithmic trading systems I was always perplexed by the fact that every time I built a new system I always hit the same issue. I would take a given day to back-test and almost curve fit a system that could beat that particular day very well. Then as soon as I started testing it on other days it would immediately blow up. Sometimes I would make it 2 or 3 days, sometimes a week, but ultimately they would all fail. And it took me a while to figure out why?

So while I won't give too much away about my own personal system, I will offer this small bit of advice.

1. High Volatility Markets: Use whatever signals you prefer to confirm you just recently entered into a high volatility market. From here place 1-2 trades using your most conservation type of profit target / stop loss. (I.E don't stay in the trade for ever, take profits and run.). Don't count on the trend to stay in tack for ever, get in take one or two trades and get out. A shift is certainly coming and you don't want to be on the wrong side of it.

2. Medium Volatility Markets: When the market shifts down to medium volatility from high you will be glad you were on the side lines. And you should remain on the sidelines. Don't trade this market at all. Just wait.

3. Low Volatility Market: When the market moves from Medium to Low volatility, put a few indicators to use to confirm, look both ways first, then place 1 -2 counter trend type of trades. Maybe go with a mean reversal, or or run some support / resistance type of system. After 1-2 trades just stop. This will likely ramp back up to Medium Volatility again shortly and you will get hit on the wrong side of it.

So that's it, just get in for a short 1-2 trades on High and Low Volatility markets, and wait for the rest. The good opportunities only typically come at the begging, and if you get too greedy and stay in for two long with the wrong type of system the market will change before you knew what hit you.


Rejoice in the Thunderstorms of Life . . .
Knowing it's not about Clouds or Wind. . .
But Learning to Dance in the Rain ! ! !
Follow me on Twitter Reply With Quote
The following user says Thank You to DavidHP for this post:
 
(login for full post details)
  #38 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Thanks for all the interest from the Futures IO community. I am going to keep posting my random observations, some trading stats from different systems i have built, and a few out there ideas that I am working on. But if there are any topics that anyone would like me to cover relating to programming, algorithm trading, statistics, or anything else, let me know and I may post some more of those topics specifically.

Mostly I am just trying offer some insights that I have picked up on that may help other traders.

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #39 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

One of the first concepts that I can point to that helped me become an effective trader was incorporating metadata into my research. By that I mean understanding why something worked or failed not based on intuition or a guess but raw statistics. In NinjaTrader programmers have the ability to print output for just about anything relating to their trades.

For example if I wanted to print out data around a set of indicators or highs and lows 10 bars before I entered a trade, and then the same data while I was in the trade I could. From this you can correlate your winners and losers back to the data associated with each. When I started doing this I found some incredible things. In trading there is one key concept that is often overlooked but it basically what every trader is trying to do.... and that is predict something. Discretionary traders are at a huge disadvantage in analyzing their prediction accuracy because they will not have a program that can provide them with all the relevant statistics, they would have to arrive at this manually. Even most algo traders are likely not taking advantage of this concept fully. But here is the general idea:


Trading Idea: I believe that when factors X,Y,Z are in place the market will be ripe for my trade and I can expect ...... to occur. This is called a hypothesis. It may be right or it may be wrong, but for sure we can test it and quantify it. So what you should do is build a test simulation to test exclusively this condition and this alone. This is the critical part, you need to isolate all other variables as much as possible so you just test the one variable in your prediction independently.

Test Simulation: (Here is an example you can use): When Volatility in the ES is high trading conditions are good for a specific type of system to work. How you do quantify this? Build two arrays like this to quantify volatility:

double[] DataArrayMax = {High[0],High[1],High[2],High[3],High[4],High[5],High[6],High[7],High[8],High[9]};
double[] DataArrayMin = {Low[0],Low[1],Low[2],Low[3],Low[4],Low[5],Low[6],Low[7],Low[8],Low[9],Low[10]};
// Finding max and min
RMax = DataArrayMax.Max();
RMin = DataArrayMin.Min();


Volatility =((RMax-RMin) *4);

This is measuring the total range of ticks over the last 10 bars. In a time-frame such as 50 ticks to 300 ticks, If the Volatility is > 15 I consider this high. If it is over 20 or 30 this is just stupid high. So now you can print the output of this variable along with each bar update, or each long or short position, or anything you want really. So now you can quantify what the exact market Volatility was during a given trade.

So let's say your criteria for entering a trade had nothing to do with volatility, but rather (this line crossing that line) or something simple. Now you can take your winners and losers and correlate them with the volatility experienced both right before the trade and through the trade, and from this gain knowledge of if different volatility levels helped or hurt your performance.

You can print it off as simple as this:

Print(tradennumber +"/"+
Time[0] +"/"+
Close[0] +"/"+
Open[0] +"/"+
High[0] +"/"+
Low[0] +"/"+
Volatility (At the entry of the trade) +"/"+
Volatility (N Bars ago) +"/"+
BarNO +"/"+
EntryPrice +"/"+
Position.AvgPrice+"/"+
Position.MarketPosition+"/"+
PT +"/"+
SL );

This type of output tells me a lot of things.

1. I can take the trade number here and match it to my overall P&L and correlate trades this way.
2. The time field will give me a measuring stick for all the bars printed from right before the trade to during the trade so I can see the whole picture.
3. You can print the volatility at the first bar of the actual trade, several bars prior to entering the trade where you likely got your signal to enter the trade, also throughout the trade you can check to see if there was a significant change.
4. Printing your Position will tell you when you went flat and when your in the trade (Long or Short). This will help you split your prediction logic from your actual trades.
5. I have a custom variable to hold my profit target and stop loss. This will let you know what your bet was on each trade. You may find that playing with these two settings will product different results


But overall, when you have this type of power at your fingertips you can start to correlate your losers and winners to different things and ultimately trace the root cause. I personally have built models similar to this to test 10-20 different types of variables independently over 4 years in the ES. From this I have learned how things are correlated and which variables are relevant and which are not. This type of scientific approach is real and does work. Once you have this type of a feedback loop, your likely hood of cracking the whole thing approaches a near certainty. But with out these types of tools and scientific method, you will likely be at a crippling disadvantage.

Hopefully sharing some of my process will help other traders follow the scientific approach.

Happy Trading

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 10 users say Thank You to iantg for this post:
 
(login for full post details)
  #40 (permalink)
Sydney Australia
 
Experience: Intermediate
Platform: Rithmic API
Trading: ES CL
 
Posts: 63 since Jan 2014
Thanks: 11 given, 24 received

iantg, your post on resting limit orders to get ahead of the queue is interesting.

With everything, testing is key. So how do you test the effect of placing limit orders earlier? Unless you have a complete order book recreation, the only way to know if your order is filled is if price goes past your order. Do you have some sort of approximation to avoid the seemingly impossible (astronomically expensive from what i can see) task of recreating the entire order book in simulations?

Or do we just assume that increasing fill rate is on the whole better? I have a soft opinion that a filled order can be good or bad depending on the circumstance. ie, missing an order you may have in fact dodged a bullet. So I am a bit hesitant to build this in to my algorithms with no way to test! Please enlighten us some more on the theory in this area.

Also, if you make the approximation of only filling your order if price goes past your limit, is this a pessimistic simulation of returns? In your posts you seem to say that these circumstances are usually a losing trade (because it continues past by a few pips at minimum). This may be true, but it can be mitigated by moving the limit forward slightly and take a slightly higher cost to still pick up those winning trades you would have otherwise missed.

Reply With Quote
 
(login for full post details)
  #41 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Hi Luke,

Thanks for reading my blog here and this is a great question. As you mentioned this would be incredibly complex to dig into if you were trying to use level 2 data. So I built something a little more basic using level 1 data. Here is an example of the basic logic to quantify this in NT:

(Code first then explanation)


if(Position.MarketPosition == MarketPosition.Long)
{
PositionStatus = 1;
LowB = Open[0];


EntryPrice = Position.AvgPrice;

if(BarNO == 0)
{
BarNO = 1;
}
//else
{
BarNO = BarNO + 1;
}


if(BarNO <3)

{

directionvalue = EntryPrice - LowB ;

Print("directionvalue");



So this example is to measure a Long Position (You can run the logic backwards for Shorts). Here we are taking the first bar printed after the order was filled. In NT the status will flip from Flat to Long. I wrote a little code logic to identify the first bar of the long position. In this bar we can quantify if the price significantly changed from the Entry Price or not. There are a couple ways of looking at this. If you go with the conservative approach you can use the Low in the case of long positions or the High in the case of Short positions, or if you want to be less pessimistic you could flip this to the Open. But the action on this bar can tell us the following:

1. If you come out of this bar flat or better vs the Entry price this was just a touch and not a pass through.
2. If you come out of this bar upside down, this was one of two things: (A. A pass through that had to occur to fill your order, or B. A touch fill, but price went against you almost immediately.)

The finer the time frame you run this on the more you could make assumptions about touch vs. pass-through, the larger the time frame the more this just becomes quantifying the action on the first bar. But run this on 1 tick and you have it.

From this I have been able to extrapolate the following:

1. How well different price points for entries work to hit a touch vs. only getting filled on pass through. For example enter at current price, current price +- .25, .5,.75,1,1.25, etc
2. How well different rests work to hit to a touch vs. only getting filled on pass through: For example place an order 1 full point from current price, then wait X number of bars before canceling and resubmitting.
3. And finally how well a trade performs overall if it starts from a touch vs. a pass through.

The correlation studies I have done on all 3 of these are in line with what you might expect. The further away you are from the action the more likely you will get filled on touch, the longer you rest, the more likely you will get filled on touch, and the touch orders tend to be far more profitable whereas the pass through orders can be a mixed bag since these often have strength going against you from the start.

Hope this helps. Let me know if you have other questions and I can try to elaborate more.

Thanks,

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to iantg for this post:
 
(login for full post details)
  #42 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received


iantg View Post
One of the first concepts that I can point to that helped me become an effective trader was incorporating metadata into my research....

Excellent Post. (Just clicking Thanks didn't give it enough credit)

Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #43 (permalink)
New Orleans, La (Mardi Gras City)
 
Experience: Advanced
Platform: NinjaTrader
Broker: Ninjatrader / Optimus Futures / AmpFutures
Trading: ES / 6E / 6B / CL
 
DavidHP's Avatar
 
Posts: 1,347 since Aug 2009
Thanks: 9,425 given, 2,360 received

Do you know the settings for CL for this Test Simulation?

Also, it may be helpful to post a .cs of an example of this technique.

Thanks


iantg View Post
Test Simulation: (Here is an example you can use): When Volatility in the ES is high trading conditions are good for a specific type of system to work. How you do quantify this? Build two arrays like this to quantify volatility:

This is measuring the total range of ticks over the last 10 bars. In a time-frame such as 50 ticks to 300 ticks, If the Volatility is > 15 I consider this high. If it is over 20 or 30 this is just stupid high. So now you can print the output of this variable along with each bar update, or each long or short position, or anything you want really. So now you can quantify what the exact market Volatility was during a given trade.
Ian


Rejoice in the Thunderstorms of Life . . .
Knowing it's not about Clouds or Wind. . .
But Learning to Dance in the Rain ! ! !
Follow me on Twitter Reply With Quote
 
(login for full post details)
  #44 (permalink)
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
 
Posts: 170 since Jul 2017
Thanks: 326 given, 348 received

Hi, Ian and thanks for posting.

Your post is very helpful. I know you are hinting a lot of good stuff to trade from but I can not use them since I only can use ninjatrader strategy builder. It took long time to come up with my current strategy. Nothing near your result but mine seems to be profitable. I have been testing it for a few month and begun the real trading on ES.

Just curious. What kind of back up plan do you have in case of power outage or internet goes down?

Thanks,
Kane.

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to kanepa for this post:
 
(login for full post details)
  #45 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Hi Kanepa,

Thanks for reading my blog and for inquiring. You raise a great question about power, and internet. For this particular reason I recommend renting a server and running your NT script from there.

There are lots of great ones that out there that are fairly cheap and reliable.

Here is one that I have used in the past that specializes in working with traders. They have servers near most of the exchanges, so depending on what you are trading and where you are located you can reduce your latency as well.

MyTradeHost ? MyTradeHost Dedicated Servers, VPS, Trading, Monitoring, Failover, Backup, Recovery, TradeStation, MultiCharts, NinjaTrader, MetaTrader, Sierra Charts, Smart Quant, Cunningham, CTS, Amibroker, TT

Thanks,

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #46 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

One of the keys to building a strong system is to constantly look for flaws, to embrace failure as a teacher and to learn from any and all mistakes. When I first started out, if I got a system even close to being profitable, I almost refused to believe that anything could be wrong with it. I was afraid to kick the tires, because this could expose me. So mediocre systems stayed mediocre.

Fast forward to today, and my attitude has done a 360 degree change. I am constantly looking for weakness, or failure or
any flaw I can find, because I know that this is the only true way to learn. Without finding weakness in a system there is no need to improve it. And every system always has a degree of weakness to it, even it you break it down to nothing more than the house edge of the market.

The reason I thought of this is because I recently started migrated my code base to NT 8 and started testing and redefining some of my strategies that have beat NT 7's market replay handily. The differences between 8 and 7 regarding the SIM engine are considerable. What works in 7 may not always work in 8 and vice versa. This obviously presenting me a whole new host of opportunities to optimize, refine and tweak my already decent strategies to prepare for the real market (Which I will likely be rolling the dice with soon).

I am enclosing a couple of screen grabs from today to illustrate the point of how you need to constantly kick your tires and look for flaws. The first pic is of a daily net profit chart running Jan 2016 to Mar 2016. At first glance this looks pretty solid, mostly all winners, decent profits, etc. But I started kicking the tires, and if you look at the second picture you can see the weakness. This system is very optimized to crush it during the normal US trading hours, but during the overnight hours the system needs a new set of rules bad! If i just never traded anything but 9-5 I would increase my profits significantly because this is a HFT system and the commissions kill me taking trades that don't pan out overnight. So now I get to bifurcate my system into 2 sets of rules.

1. The existing set of rules that work great during 9-5.
2. The new set of rules that target the low volume overnight session. I just simply need to add new variables and optimize for the typical volumes that are found overnight.

When I run this as a two step smart system, I will likely seen an increase by 20%- 30%. It was already a good system, but now I can make it even better by looking for any and every opportunity to find weakness.

Happy trading!

Ian

Attached Thumbnails
Click image for larger version

Name:	2016 Q1 NT8 Daily.png
Views:	114
Size:	250.7 KB
ID:	244454   Click image for larger version

Name:	2016 Q1 NT8 Hourly.png
Views:	114
Size:	246.4 KB
ID:	244455  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 7 users say Thank You to iantg for this post:
 
(login for full post details)
  #47 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

It's been a while.... Approximately 7 years ago I had a live trading account. I only traded a few days, and only lost around $300 bucks total. Looking back, I realized that at the time, I had no real rules, no discipline, had no concept of money management, and no real business doing it. About the only thing I got right was knowing when to call it a day and stop.

Fast forward a few years later, I got introduced to NinjaTrader and the concept of Automated trading and I have been hooked ever since. Over the last 3 years I have developed hundreds of strategies, and I have found 50 or so that generate some decent profit in SIM / Market Replay. From this pool of 50 strategies I narrowed it down to just a handful maybe 10 or so to focus on. Over the last year I have really polished these up to the point that they are nearly too good to be true in SIM / Market Replay. I have noticed that some work extremely well in NT 8 and others work extremely well in NT 7. Some work on both, but I have learned the subtle differences of the SIM engine on both 7 and 8 and they can be considerable at times. I am inclined to believe that the live market will behave more like the SIM engine on 8, so I am taking my top strategies I have developed and I am plugging these into the Live market in the coming days.

I have posted quite a bit of trade stats on here from SIM / Market Replay over the last year, and for anyone wondering... "Yeah but, can he hit these figures in a live market?" I have been wondering this too. It's time to put my money where my mouth is and jump in the deep end of the pool.

I'll post back next week sometime once I have some statistics!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 5 users say Thank You to iantg for this post:
 
(login for full post details)
  #48 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

So today was a big day for me. I finally got to test out some of my ideas that I have been working on for a long time on the real market. While I can't share everything obviously, I will share some of the statistics and some interesting observations about the day.

1. First the bad.... I totally expected to logon, and turn on my strategy and see it run through around 100 trades. Regardless of how it performed, I at least expected to get off the ground. Everything started fine for about 20 minutes, and then I noticed that the live data feed seemed kinda slow, like suspiciously slow. After checking a few modules, I confirmed that NinjaTrader had frozen and become unresponsive. I was able to close my strategy and exit any position I was currently in, but that was a bit scary and unforeseen. i reached out to NinjaTrader and asked them about it, and they couldn't find any smoking gun. I got some tech support from the guy that runs the server hosting and he confirmed that NinjaTrader showed as unresponsive on the server logs but it didn't do a hard crash, instead it just kinda stopped working. After I closed it, I tried to fire it back up to check my account and make sure my positions were closed but I was instead met with the attached error message. I couldn't get it to restart from there. The guy that runs the server helped me eventually resolve it and get it back up and running, but I decided to spend the rest of the day just doing Live SIM instead of cash trading just to be safe.

2. Statistics: I had 14 trades overall from 9:30 to about 9:50. I have often wondered how some of my entry and exit systems would do in a fast moving market, so this was a good chance to see how they held up. Here are some figures.

Entries:

1. 10 of the 14 trades got filled with the bid or ask price on the signal bar prior, while for of the trades got filled on from the ask or bid price on the next bar.
2. 9 of the trades stayed flat on the first entry bar meaning they were filled on touch, while 4 of the trades had the price pass through it to get filled, so I ended up down 1 tick at some point on the first bar of the trade.
3. Attempts to get filled: Here are the number of attempts it took to get filled on each trade:

T# Attempts to get filled
2 2
3 4
4 2
5 1
6 1
7 1
8 3
9 2
10 1
11 1
12 1
13 1
14 1

I counted the number of times an entry would be attempted but not filled. Overall I was averaging around 1.5 attempts per trade. Most of these attempts were milliseconds or seconds apart. So I don't anticipate that the few times I had multiple attempts to enter would be material.

Exits:

1. Touches / Getting Out: I averaged 3.5 touches on exits to get out of my trades. This was in line with about what I expected, though the very first trade right at 9:30 on the dot, took 9 touches to get out. I expected this though, since it was the open of the day.

2. Slippage: Because I only got to trade from 9:30 to 9:50 I was in the largest rush of the day, and therefore experienced mad slippage. 4 winning trades each lost 1 tick, and 1 losing trade had 1 tick of positive slippage in my favor. The rest were in line with my original targets. I replayed the day later in market replay for the same 15-20 minute window using the same strategy and got roughly the same slippage. I typically see both positive and negative slippage during fast times in my testing, so I wasn't surprised by this.


Obviously I don't have enough trades to tell me much at this point, but so far things seem in line with my market replay / SIM results. I will take another stab at it Monday and see how it goes.... Hopefully I can get in a full day next time!

Attached Thumbnails
Click image for larger version

Name:	Error Message.png
Views:	107
Size:	82.6 KB
ID:	244779  
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #49 (permalink)
Webinar Host
Indianoplace, IN
 
Experience: Intermediate
Platform: SierraChart
Broker: CQG
Trading: All Micros, especially the ones that move
 
Rrrracer's Avatar
 
Posts: 2,379 since Feb 2017
Thanks: 16,364 given, 8,974 received

Cool to see this progress. I'm strictly discretionary (and not very at that LOL) but it's quite interesting to see this sort of systemic process at work... gives a little insight into what I and other manual traders are up against

Looking forward to seeing how your first full day goes. Good luck!

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to Rrrracer for this post:
 
(login for full post details)
  #50 (permalink)
SF, CA/USA
 
Experience: Beginner
Platform: SC
Broker: Stage 5
Trading: NQ....that's what it boils down to
 
jackbravo's Avatar
 
Posts: 1,338 since Jun 2014
Thanks: 4,340 given, 2,389 received

I'm excited for you, primarily because it looks like live results are mirroring SIM. Hopefully you'll kill it next week and next year!

"It does not matter how slowly you go, as long as you do not stop." Confucius
Reply With Quote
The following 2 users say Thank You to jackbravo for this post:
 
(login for full post details)
  #51 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

I am hoping to get back to trading soon, but in the meantime here is a thread covering some of the technical issues I ran into recently. This is only something that I have ever seen in V 8.11. Prior to this I never had this issue.

For anyone else out there on the new 8.11 version. If you are running a memory intensive strategy and you go to refresh the trade performance, it might crash, or freeze and become unresponsive.

I have been in contact with NinjaTrader support and they are working on addressing the bug. Hopefully they will have a fix sometime soon, but in the meantime I am going to be working off of an older version < 8.11 where I do not experience this particular issue.

Here is the thread for anyone interested in learning more.

https://ninjatrader.com/support/forum/showthread.php?p=528491

Happy Holidays!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to iantg for this post:
 
(login for full post details)
  #52 (permalink)
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
 
Posts: 170 since Jul 2017
Thanks: 326 given, 348 received

Hi, Ian.

Sorry to hear about the ninjatrader. It has happened to me as well. What I have done was break strategies into pieces. Fist I had two buy signals and one sell signal in one strategy. After freezing and not firing orders, I broke it into separate strategy. So I am now running three different strategies and so far, not a single problem.

Hope they will resolve this issue soon.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #53 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


kanepa View Post
Hi, Ian.

Sorry to hear about the ninjatrader. It has happened to me as well. What I have done was break strategies into pieces. Fist I had two buy signals and one sell signal in one strategy. After freezing and not firing orders, I broke it into separate strategy. So I am now running three different strategies and so far, not a single problem.

Hope they will resolve this issue soon.


I don't know if it helps but my work around was to just not use the newest version. 8.11. I am currently running 8.6 and have not had any issues with it. I know that 8.10 is still on their site, that one may be more stable. I am not sure what version you are on, but in my case the issue was only with 8.11.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #54 (permalink)
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
 
Posts: 170 since Jul 2017
Thanks: 326 given, 348 received


iantg View Post
I don't know if it helps but my work around was to just not use the newest version. 8.11. I am currently running 8.6 and have not had any issues with it. I know that 8.10 is still on their site, that one may be more stable. I am not sure what version you are on, but in my case the issue was only with 8.11.

Thanks, Ian. I use 8.11 So far, no problem at all.

Visit my futures io Trade Journal Reply With Quote
 
(login for full post details)
  #55 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Going into 2018, I just wanted to throw out a few ideas that really helped me connect the dots in 2017. This may help others find their own edge, or look at things a different way.

I used to always look at price changes as a very binary very simple view. Price goes up 1 tick, Price goes down 2 ticks, Price goes up 4 ticks, Price goes down 6 ticks, etc. I characterized each of these events evenly and treated all changes in price as the same. Even though I have been aware of the exchange mechanics for sometime now, I didn't really start to bifurcate price changes in tandem with the way the exchange clears levels until recently. Here is an example.

The Bid and Ask both change levels up from lets say a whole number ending on .25 and now it moves to .5. So the whole level moves up 1 tick. A number of things will occur next.

1. Since the price level that cleared moved up, the first side to get served will typically be the bid volumes, when this occurs the last price will = the bid price.
2. Once the bid volumes get fed, then the guys in the ask line are next. So the last price will tick the other way to the ask price.

3. This game will continue until one side wins and one side loses. As the last price oscillates between the bid and ask, you will see the price tick up .25 and down .25. Eventually the volume from one side will drop to 0 and then the level will clear and the entire Bid / Ask price will change up or down.

So with this simple example in mind, it is important to differentiate between a price change that occurred as a function of a level change vs. a price change that occurred as a function of ticking between serving the bid / ask volumes on the same level.

* The level change is a significant event
* The ticking between bid and ask volumes within the same level is noise.
* If looking at a chart, or just price history, or anything liner, all you will see is just (Price ticked up, price ticked down) It is not obvious that these mechanics are even in place, But if you adapt your strategy (in my case code) to look for the level change events, it makes all the difference in the world in terms of truly seeing what is going on in the market.

I don't concern myself with speculative volumes (level 2) at all. I only use actual transaction data (level 1). This area of research has been on modeling the sequencing of how a level actually clears. I think most traders miss this or don't have a deep enough knowledge of the mechanics here, because they may be spending too much time looking at the level 2 speculative volumes while they are missing the most important part. Truth be told, you will never be able to see this type of action on a tool like the DOM because it occurs insanely fast, and there is no running visual history. I had to build a script to output a few hundred thousand rows of data just to get one days worth of data to analyze, and only then did I find the patterns, but they are there!

In the past 3-4 months I have developed at least 5 different solid entry systems that get me into trades flat or better in most cases based on doing some nano-structure work in the field of research concerning the level changes, bid / ask volume oscillations, sequencing events, and millisecond precision timing using the fabulous on market data event handler.

While I can't give too much away, I will say that this area of research has been quite fruitful for me in 2017, and hopefully others will research it as well and find their own edge with it.

Happy Trading!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 11 users say Thank You to iantg for this post:
 
(login for full post details)
  #56 (permalink)
Pretoria Gauteng
 
Experience: Intermediate
Platform: Sierra Charts
Trading: NQ
 
Popsicle's Avatar
 
Posts: 249 since May 2016
Thanks: 2,426 given, 545 received

Hi iantg,

Just want to say thank you for all that you share. I went through the thread from the beginning and the information that you share is absolutely awesome!

Has Ninjatrader been able to resolve your live trading issue?

Favourable Trading,
Popsicle

Follow me on Twitter Reply With Quote
 
(login for full post details)
  #57 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Thanks for reaching out and thanks for the kind words. I have been working with NinjaTrader on the technical issues. They are aware of the bug I found in 8.11 and will likely have a fix in 8.12. In the meantime I am testing different past versions to try to find one that is stable. Unfortunately most of last week I was experiencing different but equally as serious bugs with 8.6. Page 2 on this thread outlines what occurred:

https://ninjatrader.com/support/forum/showthread.php?t=106562&page=2

I have had no issues at all on 8.6 with my laptops but on my VM it crashed 4 days in a row!. I am thinking that it has something to do with this: https://ninjatrader.com/support/forum/showthread.php?t=102060

So after my issues with 8.11 and 8.6, Thursday I put 8.10 on the VM and ran SIM and it has not crashed yet, so I am very optimistic that this version might be stable for me. I plan to test in SIM for most of next week and if I don't hit any more technical issues then I will likely go back to my real trading account either late next week or the following week.

I am so ready to do this you have no idea!

Ian



Popsicle View Post
Hi iantg,

Just want to say thank you for all that you share. I went through the thread from the beginning and the information that you share is absolutely awesome!

Has Ninjatrader been able to resolve your live trading issue?

Favourable Trading,
Popsicle


Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #58 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

I don't typically give too much away about my personal system, but in this case I will share because this method is widely known and widely used by all HFT systems. I am punting this out there also because I have a few questions, and I am hoping that someone out there has an answer to my question at the end, we will see...

But here is the concept in a Nutshell.

When flat, you submit both buy and sell limit orders 4 levels above and 4 levels below the current level, so you have 8 orders resting at one time. My goal is simple, I just want to get one of these 8 orders filled within the first two sequences of action when the market moves to that level. A typical sequence once the price moves up looks like this:

Seq 1. Resting Bids gets served first. Usually 50% to 75% of the resting amounts get served.
Seq 2. Resting Asks get served next. Usually 50% to 75% of the resting amounts get served.
Seq 3. Resting Bids get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 4. Resting Asks get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 5. The final Bids that were resting that did not yet get filled + any new bids that entered are being served
Seq 6. The final Asks that were resting that did not yet get filled + any new bids that entered are being served
Etc.....

With the ES, 55% to 65% of all sequences that start with price up level change, and start by serving the Bid queue first, will end up depleting the bids before the asks are depleted, so the next level will be down. Vice Versa for levels that move down and start with the asks first. This is due to the fact that by sequence 3 you will have 2 attempts to clear the side that went first and only one attempt on the side that went second. By level 5 this will be 3 to 2 and by level 7 this will be 5 to 4. So this is why there seems to be so much noise in price movement in the ES, but typically one level that clears is the signal, and the next opposite level that clears is the noise. Determining which level is noise and which level is the signal can come as a function the count of levels cleared prior to the change, or duration of time on a level, but this is beyond the scope of this topic. (I may cover this in depth in later posts if there is adequate interest). Now in my example I show 6 sequences with the possibility of more behind it, and on average I see 5-9, but no one really knows when the shoe is going to drop and the level will suddenly change. So for any entry system to work optimally the primary goal above anything else is to get filled on the current level and to not get filled on one of the last sequences only to have your order be the very last shoe to drop before the level change goes against you and you lose 1 tick.

So if you have size and you are resting a few levels out you have a good shot at getting filled in sequence 2 or sequence 4, if you are not filled by sequence 2 then you have a 50% / 50% shot of having your fill come at the expense of losing 1 tick. If you are not filled by sequence 4 then your odds increase to 75%, and anyone getting filled on sequence 6 or later should expect 90% of the time to lose the spread and be down 1 tick for getting filled. I have studied this part enough to just free ball the stats, but generally this is what I have observed over hundreds of thousands of simulations.

Obviously no one really has a good shot at being in the front 50% of the queue coming in just one tick off of the current level, but when you put 8 resting orders (4 per level) your odds increase. The logic is fairly obvious from here, but I will just lay it out there for everyone on how this works.

1. The market finally reaches the level of your first resting order which you put 1 tick away from the current action. After filling sequence 2 and moving to sequence 3 (which is now serving the opposite side) you find you did not get filled, so you cancel your order, and resubmit a new order 4 ticks away using the same long or short position as the one you just canceled.

2. From here the market either continues moving the same direction and meets your next resting order that was for the same direction, or it moves backwards and finds your first resting order for the opposite direction which was as far away resting as 2 levels. Now we just repeat the process. If you don't get filled by sequence 2 cancel and resubmit 4 levels away.

Now this is just covering the basic mechanics, but in addition to this method I also have other things I look at to determine strength in direction, (Alpha Signal), and if I am not satisfied when my number is called, I can cancel for lack of a coherent alpha signal as well. But for obvious reasons I will not cover this part.

But generally speaking the idea is to only accept fills that have a very high probability of getting filled on the current level. Using 8 resting orders 4 levels out on both sides will increase the likely-hood of achieving the time priority, but it will sill suffer from orders that cut in line because they hold larger sizes. So to gain advantage here, and because there is a prevailing opinion that HFT algos are doing this... Is it possible to gain advancement in the queue by holding larger sizes and ultimately canceling down to a smaller size and not just lose the queue positioned gained immediately?

Does anyone have any insight on if or how often the matching engines sort based on size / time priority? This could be one of two scenarios.

1. The Matching Engine sorts and shuffles orders ever X interval to account for changes in size. This would ultimately move people up and down in priority, but not instantly, as the sorting only occurs ever X intervals. In this scenario one could gain advantage by faking larger sizes than they intent to trade then canceling at a point prior to the matching engine resorting and demoting them.

2. There is no sorting in place at all, every order's status is virtually instantly updated in real time. In which case the idea of holding 5 contracts to gain priority if there is only intent to trade 1 won't work because as soon as you cancel 4 you are instantly in the place in the queue you belong just holding 1. I am guess it is most likely this scenario, but there is definitely a prevailing opinion that algorithmic traders show size then cancel to both trick order flow traders and to try to gain advantage in the queues but I do not know to what extent their attempts to gain advantages in the queues are successful.

For anyone that can give me some insight into these mechanics I would be glad to share some of my code for this system, or potentially help in other ways as a reward.

Thanks in advance.

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 3 users say Thank You to iantg for this post:
 
(login for full post details)
  #59 (permalink)
Bay Area California
 
Experience: None
Platform: TT T4
Broker: Phillip Capital
Trading: Futures
 
Posts: 800 since Nov 2011
Thanks: 873 given, 881 received


iantg View Post
I don't typically give too much away about my personal system, but in this case I will share because this method is widely known and widely used by all HFT systems. I am punting this out there also because I have a few questions, and I am hoping that someone out there has an answer to my question at the end, we will see...

But here is the concept in a Nutshell.

When flat, you submit both buy and sell limit orders 4 levels above and 4 levels below the current level, so you have 8 orders resting at one time. My goal is simple, I just want to get one of these 8 orders filled within the first two sequences of action when the market moves to that level. A typical sequence once the price moves up looks like this:

Seq 1. Resting Bids gets served first. Usually 50% to 75% of the resting amounts get served.
Seq 2. Resting Asks get served next. Usually 50% to 75% of the resting amounts get served.
Seq 3. Resting Bids get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 4. Resting Asks get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 5. The final Bids that were resting that did not yet get filled + any new bids that entered are being served
Seq 6. The final Asks that were resting that did not yet get filled + any new bids that entered are being served
Etc.....

With the ES, 55% to 65% of all sequences that start with price up level change, and start by serving the Bid queue first, will end up depleting the bids before the asks are depleted, so the next level will be down. Vice Versa for levels that move down and start with the asks first. This is due to the fact that by sequence 3 you will have 2 attempts to clear the side that went first and only one attempt on the side that went second. By level 5 this will be 3 to 2 and by level 7 this will be 5 to 4. So this is why there seems to be so much noise in price movement in the ES, but typically one level that clears is the signal, and the next opposite level that clears is the noise. Determining which level is noise and which level is the signal can come as a function the count of levels cleared prior to the change, or duration of time on a level, but this is beyond the scope of this topic. (I may cover this in depth in later posts if there is adequate interest). Now in my example I show 6 sequences with the possibility of more behind it, and on average I see 5-9, but no one really knows when the shoe is going to drop and the level will suddenly change. So for any entry system to work optimally the primary goal above anything else is to get filled on the current level and to not get filled on one of the last sequences only to have your order be the very last shoe to drop before the level change goes against you and you lose 1 tick.

So if you have size and you are resting a few levels out you have a good shot at getting filled in sequence 2 or sequence 4, if you are not filled by sequence 2 then you have a 50% / 50% shot of having your fill come at the expense of losing 1 tick. If you are not filled by sequence 4 then your odds increase to 75%, and anyone getting filled on sequence 6 or later should expect 90% of the time to lose the spread and be down 1 tick for getting filled. I have studied this part enough to just free ball the stats, but generally this is what I have observed over hundreds of thousands of simulations.

Obviously no one really has a good shot at being in the front 50% of the queue coming in just one tick off of the current level, but when you put 8 resting orders (4 per level) your odds increase. The logic is fairly obvious from here, but I will just lay it out there for everyone on how this works.

1. The market finally reaches the level of your first resting order which you put 1 tick away from the current action. After filling sequence 2 and moving to sequence 3 (which is now serving the opposite side) you find you did not get filled, so you cancel your order, and resubmit a new order 4 ticks away using the same long or short position as the one you just canceled.

2. From here the market either continues moving the same direction and meets your next resting order that was for the same direction, or it moves backwards and finds your first resting order for the opposite direction which was as far away resting as 2 levels. Now we just repeat the process. If you don't get filled by sequence 2 cancel and resubmit 4 levels away.

Now this is just covering the basic mechanics, but in addition to this method I also have other things I look at to determine strength in direction, (Alpha Signal), and if I am not satisfied when my number is called, I can cancel for lack of a coherent alpha signal as well. But for obvious reasons I will not cover this part.

But generally speaking the idea is to only accept fills that have a very high probability of getting filled on the current level. Using 8 resting orders 4 levels out on both sides will increase the likely-hood of achieving the time priority, but it will sill suffer from orders that cut in line because they hold larger sizes. So to gain advantage here, and because there is a prevailing opinion that HFT algos are doing this... Is it possible to gain advancement in the queue by holding larger sizes and ultimately canceling down to a smaller size and not just lose the queue positioned gained immediately?

Does anyone have any insight on if or how often the matching engines sort based on size / time priority? This could be one of two scenarios.

1. The Matching Engine sorts and shuffles orders ever X interval to account for changes in size. This would ultimately move people up and down in priority, but not instantly, as the sorting only occurs ever X intervals. In this scenario one could gain advantage by faking larger sizes than they intent to trade then canceling at a point prior to the matching engine resorting and demoting them.

2. There is no sorting in place at all, every order's status is virtually instantly updated in real time. In which case the idea of holding 5 contracts to gain priority if there is only intent to trade 1 won't work because as soon as you cancel 4 you are instantly in the place in the queue you belong just holding 1. I am guess it is most likely this scenario, but there is definitely a prevailing opinion that algorithmic traders show size then cancel to both trick order flow traders and to try to gain advantage in the queues but I do not know to what extent their attempts to gain advantages in the queues are successful.

For anyone that can give me some insight into these mechanics I would be glad to share some of my code for this system, or potentially help in other ways as a reward.

Thanks in advance.

Ian


The matching algorithm for ES is solely based on time priority. There is no size component. Though other products due have size components.

There is also no ďintervalĒ. All modern orderbooks are event driven. Meaning they are updated in real time on every submit/cancel/modify.

https://www.cmegroup.com/education/matching-algorithm-overview.html



Sent from my iPhone using Tapatalk

.
Reply With Quote
The following 4 users say Thank You to addchild for this post:
 
(login for full post details)
  #60 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Thanks for the quick reply. That pretty much confirms what I thought. I thought ES was FIFO, but I know others are size / time priority and I just wondered about the general concept and how it may apply. I would guess that any half decent exchange would synchronize their books in real time thus preventing any possible manipulation to gain rank. But with all the smoke out there about HFT systems gaming the exchanges in every possible way, I just figured I would punt this out there.

Thanks!

Ian



addchild View Post
The matching algorithm for ES is solely based on time priority. There is no size component. Though other products due have size components.

There is also no ďintervalĒ. All modern orderbooks are event driven. Meaning they are updated in real time on every submit/cancel/modify.

Matching Algorithm Overview - CME Group



Sent from my iPhone using Tapatalk


Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #61 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received

https://www.cmegroup.com/confluence/display/EPICSANDBOX/Matching+Algorithms

combined with

https://www.cmegroup.com/globex/files/globex-product-reference-sheet.xls

well let you see exactly which product uses which matching Algo

Reply With Quote
The following 3 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #62 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


iantg View Post
I don't typically give too much away about my personal system, but in this case I will share because this method is widely known and widely used by all HFT systems. I am punting this out there also because I have a few questions, and I am hoping that someone out there has an answer to my question at the end, we will see...

But here is the concept in a Nutshell.

When flat, you submit both buy and sell limit orders 4 levels above and 4 levels below the current level, so you have 8 orders resting at one time. My goal is simple, I just want to get one of these 8 orders filled within the first two sequences of action when the market moves to that level. A typical sequence once the price moves up looks like this:

Seq 1. Resting Bids gets served first. Usually 50% to 75% of the resting amounts get served.
Seq 2. Resting Asks get served next. Usually 50% to 75% of the resting amounts get served.
Seq 3. Resting Bids get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 4. Resting Asks get served again, the remaining 25% to 50% are worked on here, but not completely exhausted.
Seq 5. The final Bids that were resting that did not yet get filled + any new bids that entered are being served
Seq 6. The final Asks that were resting that did not yet get filled + any new bids that entered are being served
Etc.....

With the ES, 55% to 65% of all sequences that start with price up level change, and start by serving the Bid queue first, will end up depleting the bids before the asks are depleted, so the next level will be down. Vice Versa for levels that move down and start with the asks first. This is due to the fact that by sequence 3 you will have 2 attempts to clear the side that went first and only one attempt on the side that went second. By level 5 this will be 3 to 2 and by level 7 this will be 5 to 4. So this is why there seems to be so much noise in price movement in the ES, but typically one level that clears is the signal, and the next opposite level that clears is the noise. Determining which level is noise and which level is the signal can come as a function the count of levels cleared prior to the change, or duration of time on a level, but this is beyond the scope of this topic. (I may cover this in depth in later posts if there is adequate interest).



I punted this topic out here a couple days ago, and it has a lot of interesting points to it. I can tell you that the mechanics of how the bids and asks fill typically follow what I have described above. I only know this, because I have observed this by extracting millisecond level data (Millions of rows) and analyzing them. I have some theories on this behavior, but honestly I don't quite understand the exact drivers of the following: I know how it kind of works but I would love to know more about it.

At any point in time the Fill engine has two options.

1. Fill all the resting volumes on a single side and drive the price up or down to the next level accordingly.
2. Fill only a portion of the volumes on one side, and the stop and switch and fill from the other side keeping the level in tact.

I figure there are a few possible reasons for why it moves the way that it does, but none of these quite make sense. Here are a few theories that I think would be easy to debunk:

Theory A: The FIFO may apply to both sides evenly. So it uses time stamps on both the bids and asks indiscriminately. Whichever got there first gets worked on, this is why it flips the last all the time.

Obvious Issues: If this was the case the last would jump back and forth hundreds of times on each level, but I only typically observe 10 or so moves back and forth and each with a a significantly clearing of the level, so this doesn't add up.

Theory B: There is a special ratio that the fill engine uses to hit a predetermined portion on pass 1, pass 2, pass 3, etc. This is all ratio driven and based on the volumes that are out there X amount will be cleared before switching sides.

Obvious Issue: If such a targeted method was in place the patterns would be more consistent. I don't observe this.

Theory C: This could be pricing related in terms of average price movements: If a certain portion of bids or asks clear then the weighted average of the spread relative to recently cleared volumes drives the last price up or down. When it gets passed the median threshold it changes which side gets served. This is all just a function of pricing.

Obvious Issue: The line that divides this would get crossed every few trades, and the logic to find a price related balancing point would be like sprinkling salt on two sides of an gram scale.. It would just swing back and forth with every few flakes of salt you added.


So for bonus points, does anyone have any clue exactly the reasoning for why the fill switches sides between the bid and ask, as in what is the key trigger for this event. I have tried to look this up but I have never found anything on this topic anywhere. I can observe the data and figure up statistics about it, but I can't find any information governing the logic in the mechanics. I would appreciate any insight anyone has on this topic. And if anyone wants any of my research on this, just let me know.


Thanks!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #63 (permalink)
Pretoria Gauteng
 
Experience: Intermediate
Platform: Sierra Charts
Trading: NQ
 
Popsicle's Avatar
 
Posts: 249 since May 2016
Thanks: 2,426 given, 545 received

My guess is that it might be something like this:

Price is moved by market orders (also called aggressors) as we know. Market orders on both sides are also put into one FIFO queue and serviced accordingly. So as an example for a fictional instrument that uses the FIFO algorithm:

Best Bid : $105.00 - 100 limit orders resting
Best Offer : $106.00 - 250 limit orders resting

Market orders arrive in the following order:

Sell Aggressor : 3 contracts
Sell Aggressor : 5 contracts
Buy Aggressor : 9 contracts
Buy Aggressor : 10 contracts
Buy Aggressor : 6 contracts
Sell Aggressor : 1 contract
....

The way this will play out in the matching engine is: Bid price->Bid Price->Offer Price->Offer Price->Offer Price->Bid Price.

This one queue of market orders will be serviced on a first in first out basis until all the resting limit orders on a level are exhausted and then move on to the next level.

Once again, this is my guess.

What do you think?

Follow me on Twitter Reply With Quote
The following user says Thank You to Popsicle for this post:
 
(login for full post details)
  #64 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


Popsicle View Post
My guess is that it might be something like this:

Price is moved by market orders (also called aggressors) as we know. Market orders on both sides are also put into one FIFO queue and serviced accordingly. So as an example for a fictional instrument that uses the FIFO algorithm:

Best Bid : $105.00 - 100 limit orders resting
Best Offer : $106.00 - 250 limit orders resting

Market orders arrive in the following order:

Sell Aggressor : 3 contracts
Sell Aggressor : 5 contracts
Buy Aggressor : 9 contracts
Buy Aggressor : 10 contracts
Buy Aggressor : 6 contracts
Sell Aggressor : 1 contract
....

The way this will play out in the matching engine is: Bid price->Bid Price->Offer Price->Offer Price->Offer Price->Bid Price.

This one queue of market orders will be serviced on a first in first out basis until all the resting limit orders on a level are exhausted and then move on to the next level.

Once again, this is my guess.

What do you think?

Thanks for taking a swing at this!

You know this was another scenario that I thought about but I left it off my list of guesses because market orders are typically small sized. While everyone including big players use them for stop losses, this is the most common order type used for entries by the small retailers.... So this would imply that you would see tons and tons of 1-3 contract market orders. If the fill logic was to just flip long and short entries from retail traders against the resting bid and ask limit orders, then I would expect to see the market flash like a strobe light alternating the last price almost every millisecond. So every level that clears would have thousands of flips between the bid and ask. What I typically observe is 3- 10 flips of the last price as the bid volumes and ask volumes are being served on each level. So I would be surprised if this alone was it. But it is possible that the timing of the market orders has something to do with it overall. I just don't think time alone is the factor for the reason I outlined.

I would love to know the answer to this query though. I think that a ton of successful trading techniques could be derived from uncovering this.

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
 
(login for full post details)
  #65 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received

With a FIFO engine what @Popsicle said is I believe correct.

Remember the matching engine is only processing orders when they come in. Without new/change orders nothing happens. At time it seems like you are implying the matching engine decides whether to fill the bid or ask next. That is decided by whether the next aggressor order is a buy or a sell and nothing to do with the engine itself.

Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #66 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received

Here's another theory. What percentage of the ES market do you think is HFT or other automated trading systems? 50%? If so half of all those trades are those systems taking position, exiting positions, re-positioning themselves, etc etc.

Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #67 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


SMCJB View Post
With a FIFO engine what @Popsicle said is I believe correct.

Remember the matching engine is only processing orders when they come in. Without new/change orders nothing happens. At time it seems like you are implying the matching engine decides whether to fill the bid or ask next. That is decided by whether the next aggressor order is a buy or a sell and nothing to do with the engine itself.

I am inclined to say that I think this theory makes the most sense and I would believe this to be the case except the data I have collected doesn't quite line up.

The enclosed spreadsheet captures every change in level one position when the last price flips between the bid and ask. I have this sequenced up to 7 sequences. (beyond this was unnecessary because the level usually broke prior to this.) Here is how the sequence works:

If the level moves up:
Seq 1 = Last = Bid and bid volumes are cleared
Seq 2 = Last = Ask and ask volumes are cleared
Seq 3 = Last = Bid and bid volumes are cleared
etc.

If the level moves down:
Seq 1 = Last = Ask and ask volumes are cleared
Seq 2 = Last = Bid and bid volumes are cleared
Seq 3 = Last = Ask and ask volumes are clared
etc.

There are a few exceptions, but like 90% + of the time if the move is down, the ask volumes go first, and if the move is up the asks go first. Considering the ES typically has a few hundred resting bids and asks on both sides and considering that most market orders are smallish < 10 contracts, to take down an entire level I would expect to see hundreds of flips between bid and ask if this was just FIFO on market orders.

I have my code built to run an update on the OnMarketData event handler every time the last price flips, and I have it sequenced so it always knows the count up to level 7. Everything is color coded so it's pretty easy to follow. This is just a cross section of an hour or so from the ES last month, but this is how I am seeing the market move and levels cleared.

I would love to understand the mechanics of this more, and it is possible that market orders are driving this (Flipping of the last) but I just don't quite get how.

Anyway, I am punting this out there with some actual data, to try to spark a lively conversation. Hopefully we can crack this.

Thanks to everyone pitching in throwing in ideas!

Ian

Attached Files
Register to download File Type: xlsx NanoStructure Research.xlsx (275.1 KB, 28 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #68 (permalink)
ANCONA - ITALY
 
 
Posts: 5 since Feb 2014
Thanks: 1 given, 0 received

before do the backtest you have to do the double check that the ticks who you have are correct,
you can try with tickstory

Reply With Quote
 
(login for full post details)
  #69 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received

Are you sure your data is complete?
Row 2 is 2654.75/2655 341x155
Row 3 is 2654.75/2655 154x322 (ie bid volume down 187, ask volume up 167)
Obviously one order can not both decrease the bid volume and increase the ask volume without changing price levels.
I suspect your not getting the raw tick data but something where updates are conjugated before being distributed. If so, then what appears to be one update to you, could be made up of dozens of updates.

Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #70 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


SMCJB View Post
Are you sure your data is complete?
Row 2 is 2654.75/2655 341x155
Row 3 is 2654.75/2655 154x322 (ie bid volume down 187, ask volume up 167)
Obviously one order can not both decrease the bid volume and increase the ask volume without changing price levels.
I suspect your not getting the raw tick data but something where updates are conjugated before being distributed. If so, then what appears to be one update to you, could be made up of dozens of updates.


I have thought this also. I have ran this type of script on 1 tick, 1 second and the OnMarketData event handler in NT. https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdata.htm And tried to get it as granular as possible.
The issue that I keep hitting is that the resting volumes seem to have obvious gaps, even though the time stamps seem to have millisecond precision. The data is supposed to be unfiltered, it's just the standard replay data from Ninjatrader / Kinetic. I might record some live data from my Continuum data feed and play it back and see if there is any more granularity there.

The only possible explanation I can think of is that 25% to 50% of all the volumes we see are being canceled before they actually execute. This might be the case on the ES with all the HFT. So this might explain why level changes occur when the last published volumes are north of 0 by a large margin. Or... I just have serious gaps in my data somehow.

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #71 (permalink)
Choctaw
 
 
Posts: 10 since Sep 2017
Thanks: 12 given, 2 received

Ian, Thanks for this thread! I've gained a lot of insight from your candidness.

I took your idea on "metadata" and I've started printing "volatility" (based on your 10 bar example), VROC and other other metrics to my output tab. I just got the coding done and haven't thought much about how I'll go about optimizing. I ran a backtest with a simple EMA strategy over a year for some data to test. Right now I'm manually filtering columns in excel (volatility > n for example) just poking around seeing how it affects the overall P/L. So here's my question: What is your method to optimize a strategy once you have this metadata?

Reply With Quote
 
(login for full post details)
  #72 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


pen15 View Post
Ian, Thanks for this thread! I've gained a lot of insight from your candidness.

I took your idea on "metadata" and I've started printing "volatility" (based on your 10 bar example), VROC and other other metrics to my output tab. I just got the coding done and haven't thought much about how I'll go about optimizing. I ran a backtest with a simple EMA strategy over a year for some data to test. Right now I'm manually filtering columns in excel (volatility > n for example) just poking around seeing how it affects the overall P/L. So here's my question: What is your method to optimize a strategy once you have this metadata?


Hi Pen15,

Thanks for reading this thread and thanks for asking. This is a great question. I have a pretty thorough process to my optimizations and I will likely post on this in more detail in the near future, but generally what you are trying to do with any type of optimization once you have metadata is isolate your variables to try to see if there is any sort of correlation that drives performance up or down. I will give you an idea pertaining to volatility to try. Test this and see what you think, but it should work for you. Using your metadata (Output print windows) you need to collect the following variables:

1. Trade Number
2. Volatility level: (There are a ton of ways to quantify this, and I provided some examples in prior posts, and there may even be some indicators for this, but really what you are trying to do is just get the numbers. 1-10, 3-7, (Low, - High) how ever you draw the line in the sand.
3. Profit Target (In terms of number of ticks
4. Stop loss in terms of number of ticks

Technically you don't have to even collect the last two you just have to keep these constant for the test. So don't change these at all, pick a setting, say for example a PT of 5 vs. an SL of 10, and don't touch it for the entire simulation.

Now for the testing: You will need to run at least 3 simulations over a period of time. If you can do this via a strategy then this will be ideal as you can collect more data much faster but if you are doing it discretionary then at least get a day per sample. But here is the general idea for the tests.

Test 1: Set a Tight PT with a High SL. Let's say 3 ticks of profit and 6 ticks of loss, or some other 1x2 ratio that seems appropriate for the instrument you are testing. Then run the simulation collect all your statistics and correlate the following.

A: For each trade # what was the volatility number: 1-10 for example
B: For each trade # what was the outcome. Did you win or lose?
C: We already know the PT and SL settings because these were constant through out the whole simulation.

Now here is the analysis, take your results in a spreadsheet format and put these into a pivot table and then aggregate the results by the volatility levels 1-10 and you will see the following:
1. At low levels of volatility 1-3 for example, was your net profit (All trades combined with volatility at this level) positive or negative, what was your average trade value for example.
2. At medium levels of volatility (4-6) for example what was your average trade value, and total net profit
3. At high levels of volatility (7-10 for example) what was your average trade value and total net profit

You will quickly find that setting a low PT against a high SL is very favorable in a low volatility type of trading environment, but typically performs poorly in a high volatility trading environment. In low volatility the market stays in a tight range, so by placing your stop loss just outside of this range, you will almost never hit it, so even though it may have 2x or 3x the value of your PT, you can beat the expectancy.

Next simulation: set your PT at 1.5 x or even 2x your SL. Lets say PT = 8 and SL = 5 (just throwing it out there).
Now run the exact same test and collect the statistics the same way. Here you will see the following.

In low volatility you hit your SL way more than your PT and lose.... Big time. But in times of high volatility, the odds of a big move occurring are equally as likely as a small move, so your # of winners may be close to even your number of losers but the value of your winner is 1.5 or 2X the value of your loser. You will crush it with this populations.

And you will run the same type of test for medium volatility levels, etc, and find the sweet spot there too.

I can't tell you how easy this is conceptually, but it takes time and is quite boring to do, and finding exactly where the lines are can be tedious and will take lots of repetition, but eventually these will be crystal clear and you will start crushing it more than you can ever believe.

The volatility measurement itself takes some work to optimize because you can use a 5 period, 10 period, 20 period or greater population to sample the data to find the absolute delta between the high and low, but depending on your time series this may be to tight or too lose. For example running a 5 period measurement on tick data of like 15 ticks would produce next to nothing, everything would show as low volatility because you are too zoomed in. By contrast if you back this up to a 50 period measurement over a 30 minute time period the volatility looks crazy high all the time. You need to find that sweet spot where you can see good delineation. So this is your first step.

Your second step is finding the right PT / SL settings to get good throughput in low volatility and high volatility within your chosen time frame. This will take a lot of tweaking.

Finally your third step, is to find the exact lines. It's easy to see low and high volatility but medium is way more subtle and there may be 2 or 3 levels of delineation here you may want to add within this population. Ultimately by testing over and over in different time frames, with different volatility settings and different lines in this sand here is what you are hoping to come up with.

1. For low volatility quantified by values of X or less, trade with a PT of Y against an SL of X. I told you generically what this is, but you will need to find the sweet spot with your given instrument.
2. For high volatility quantified by values of X or >, trade with a PT of Y against an SL of X. Here you want to shoot for home runs and eat more losers than winners but with lower value losers and higher value winners. Something in the 2x1 range or >.
3. Medium volatility: This is the trickiest part, because you will need to find the exact volatility level matched with the exact PT / SL setting to switch between low and medium and medium and high. Within medium you may want to bifurcate this further into medium low / medium high, etc. But generally you are trying to keep your PT / SL fairly even, so that as these play out you have 50%/ 50% winner / losers and the payout is equal. Truthfully there is a lot less of a chance to exploit these type of market conditions. The best play here is to set the bet line even and just wait for a change back to low V or high V where you have a definitive betting edge.

Anyway, I can cover more about the metadata and process you will need but this should help you get setup and started. Good luck. I hope you crush it!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #73 (permalink)
Choctaw
 
 
Posts: 10 since Sep 2017
Thanks: 12 given, 2 received

Thatís super helpful to know how you change stops and targets in different volatility environments. I like the idea of pivot tables to optimize, but I think there could be a more automated and complete way to find the very best combination. Iím trying to use excelís solver add-in to run through various iterations of the volatility level to maximize profit over my 1-year data set. Still working out some bugs though. I think Iíll also try to make volatility a strategy input so I can test it with ninjatraderís optimizer, though Iím not sure how accurate the optimizer results are.

Reply With Quote
 
(login for full post details)
  #74 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

One of the more powerful but less understood optimization tools out there is the MAE / MFE stat. For the uninitiated:

MAE: Maximum Adverse excursion: This shows you the most ticks you were upside down in a given trade. (Your worst position) Sometimes this is = to your stop loss, sometimes it is worse than your stop loss, and other times (For winning trades) it shows the closest towards your stop loss that you got.

MFE: Maximum Favorable excursion: This shows you the most ticks you gained positive in a given trade. (The highest value positive you ever hit). This may be = to your PT, or even higher if you don't have a hard PT for example. For losing trades this shows you how close you got to hitting your PT before your trade turned into a loser.

There are a number of trading platforms that collect this data for you automatically. NinjaTrader has this statistic built into backtesting / simulations and market replay. So getting the data is quite easy, but analyzing it and turning this into actionable insights can be a bit tricky. It took me over a year of backtesting before I even looked at it and understood what it was and starting figuring out how to use it to my advantage. So I am going to throw out a few ways it helped me, in the hopes that others may learn from this.

You will be running a few test simulations to try to optimize your exits using the results of the MAE / MFE to fine tune your exit system.

For each of these tests you will use the exact same entry system. So whatever alpha signal you use for your entries (If this occurs go long, if that occurs go short) do not change this throughout the course of your tests.

Test 1: Set a fairly high profit target against a fairly low stop loss. Something in the 2x1 range. Run your simulation on a decent population of data. Do at least 1,000 trades (Ideally over different time frames with different seasonality, volatility, etc). Then analyze your results by running the following what if scenarios:

MFE Analysis 1. If the profit target was shorter by 1 tick, 2 tics, 3 ticks, etc, how many more winners would I have had? You can find this out by looking at all your losers and seeing the MFE: For each of these, how close were they to your profit target? Your goal in this "what if analysis" is to try to turn as many of your losers into winners as possible. But in order to do this and keep the results honest you will also have to decrease all of your winners by the same number of ticks. So for example if your original profit target was 10 ticks and your original stop loss was 5 ticks, and you only won 20% of the time, but if after analyzing your MFE you fine that at 8 ticks you would have won 35% of the time, you have to take all of your winners at 10 ticks and make them 8 ticks as well. This gets you more winners, but you lose a couple of ticks on your existing winners. Now the comparison... Does this "what if scenario" perform better or worse for you? You will have to do this type of "what if analysis" on every single tick level from the current PT down all the way to 1 tick to see every possible outcome, but ultimately you will find that by reducing the PT down, you increase your winning percentage and at a certain threshold you will find the sweet spot that optimizes your profit.

Test 2 analyzing the MAE: Now for test 2 you will need to run the test slightly backward. Set your stop loss higher and your PT lower. Maybe run it with the opposite 1X2 PT = 5, SL = 10. After 1,000 trades or so, you will be analyzing your stop losses and running a series of "what if analysis" on them. Using the MAE on your winning trades, if you move the stop loss up from 10 ticks, to 8 ticks, or 5 ticks and essentially turning winning trades into losing trades will this help your overall performance. You may at first think this is a crazy idea, why would you want more losing trades? But the obvious benefit is that you can decrease all your actual real trades at 10 ticks down to this new level. So lets say that if instead of running your stop loss at 10 ticks, you move it up to 8 ticks. If your MAE only shows that 5%-10% of your winners hit this level, then you eat 5%- 10% more losses but all of your 10 tick losers now get to hurt less and become 8 tick losers. So with this approach in mind, you will need to run this sort of "what if analysis" on ever tick level from 10 down to 1 to see where that sweet spot is at.

Naturally this process can be quite tedious and take a lot of work to find the sweet spot on each side, and what comes next is a bit like whack a mole, but you get to take the optimized result of each side and then rerun the test again! The full series that I recommend follows this pattern:

Level 1:
Series 1: High PT / Low SL (Target Optimize the PT)
Series 2: High SL / Low PT (Target Optimize the SL)

Level 2: (Using the results of series 1 as the input)
Series 1a : Series 1 optimized PT vs High SL
Series 1b: Series 2 optimized PT vs Low SL

Level 3: (Using the results of series 2 as the input)
Series 2a: Series 2 optimized SL vs High PT
Series 2b: Series 2 optimized SL vs Low PT

In series 2 and 3 your goal should be to optimize the opposite side testing both high and low to find opportunities.

Finally once you have a full analysis from series 2 and 3, you run the final optimized outputs together in the last simulation. Eventually you will find the best combination.

This process in case you are thinking it already.... is known as curve fitting. And if done on a small sample size this serves little to no purpose other than showing that you can find the answer to beat a specific sample of data by optimizing your strategy. But if you do the work (And it is a ton of work) over a very large population, say for example 3 years and hundreds of thousands of trades, then your results will be not curve fitting for a small population but for the actual instrument you are trading. Obviously most people have a life outside of trading, and if your entry system is only mediocre this type of analysis and time spend may not be enough to save you, but if you already have a decent system, this may be a nice approach to give you even more edge.

Happy Trading!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #75 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

I recently punted out a topic a few posts ago concerning the nano-structures the ES market and the way the levels were being cleared by alternating between the bid and ask volumes. I got some good feedback and there were some gaps in the data I was seeing and what theories we all came up with to describe the behavior of the market. Based on the ideas of @Popsicle and @SMCJB (That the queue for market orders would match with the queue for limit orders and work in tandem based on FIFO) I knew that I had some gaps in my data but I couldn't quite get my head wrapped around how to see the market any more granular.

So I took my granularity level from 50 ticks down to 1 tick and tried to see if I could build a full sequence of trades from x contracts down to the last 1 contract before each level broke. After doing some research I have come to understand that this will never be achievable based on just the GetCurrentBidVolume() / GetCurrentAskVolume() alone. There are always going to be gaps in the data between each update as new contracts get added to the queue and transactions occur.

So I have been working on a few ideas to try obtain as much granularity as possible and I have come up with something pretty neat that I wanted to share. Using the https://ninjatrader.com/support/helpGuides/nt8/en-us/?marketdataeventargs.htm methods I have been able to build a full history of all transacted volumes at the bid and ask between each update from GetCurrentBidVolume() and GetCurrentAskVolume(). I still have no way to quantify new volumes added to the queue and canceled orders, but I can at least gain visibility to transacted volumes between updates. So this should at least improve my visibility to the worst possible scenarios such as.

1. Between updates: There is a massive series of large transactions on one side, moving the volumes down to near 0.
2. Between update: One side shifts and the volume power switches significantly to the other side between updates.

In addition to this, I have implemented a queue position tracker to quantify where I am at in the queue. This works like this.

1. When an order gets placed, grab the applicable volume (bid or ask) and use this as a starting point. This is the number of contracts that have to clear before you will get filled.
2. Keep a running tally of all the transacted volume that occurs after your entry. Once this running total exceeds the original volume of your entry you can safely assume a fill. In practice this will likely occur even faster due to orders that cancel. Unless you are not measuring FIFO instruments this approach will work.

So I built this queue tracker in connection with live updates to the running bid / ask volumes between gaps to try to give me insight into the following.

1. How close am I to getting filled relative to the level breaking against me? For example if there are 20 contracts ahead of me that need to fill before I get filled, but only 30 contracts left total, then waiting in line to get filled would be bad. Because as soon as I get filled, the price level will change and I will be down 1 tick.

2. Is there a specific ratio of bid / ask volumes present when entering a trade that will make it more likely or less likely to get filled in the following ways.
A: You get filled safely on a level, but neither close to the level breaking in your favor or against you.
B: You get filled because you are in the front of the queue and the opposite side is week, so therefore it will likely break and you will move up 1 level with an instant positive position of 1 tick.
C: You are in the back of the queue and your fill will come at the expense of the level immediate dropping against you and you will be down by 1 tick.

So I am in the process of building some models to test this over millions of simulations of different entry points with different starting conditions. I am assuming entry attempts take place on every price level and I am running this on 1 tick data. So for context a few hours ends up being > 100K rows with very rich data points. I haven't quite started working on aggregating this up to less granular time frames but I imagine it can be done fairly easily. For now I am just trying to find some sweet spots.

I am enclosing a small sample of this data output just to see if anyone wants to take a whack at analyzing it and throwing in their two cents on some possible good / bad entry points. For this sample data I assumed the entry points in all cases would be the weak side entry on a price level change. So for example: If the price moved up a level, the entry point would be to buy the bid, by contrast if the price moved down a new level the entry point would be to sell the ask. This was just easy to program to start off with. Eventually I will find ratio specific entry points based on volumes and resting on various levels away from the entry level, but this is just a starting point.

If anyone wants to take a whack at analyzing this and giving me some ideas, I am game.

Thanks,

Ian

Attached Files
Register to download File Type: xlsx NanoStructure + Queue Positioning Research.xlsx (3.55 MB, 19 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 4 users say Thank You to iantg for this post:
 
(login for full post details)
  #76 (permalink)
Market Wizard
Chicago Illinois USA
 
Experience: Advanced
Broker: IB, ToS
Trading: /ES, US Equities/Options
 
wldman's Avatar
 
Posts: 3,341 since Aug 2011
Thanks: 1,978 given, 8,827 received

I appreciate this post and I respect the work, for sure. So not to discourage, but rather to understand your focus...What will completion of this project mean for making money? How will you monetize this effort?

Dan

Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to wldman for this post:
 
(login for full post details)
  #77 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received


wldman View Post
I appreciate this post and I respect the work, for sure. So not to discourage, but rather to understand your focus...What will completion of this project mean for making money? How will you monetize this effort?

Dan

Hi wldman, Thanks for inquiring. I suppose some of my ideas can sound a bit like rambling nonsense at times... So I imagine there are still plenty of dots I need to connect to correlate how such an undertaking can assist with improving my trading.

Basically I am working on this type of research to help me improve my entries. Like so many others I suffer from entering on the wrong side at times, or put another way my timing might be off so getting filled comes at the expense of immediately being down 1 tick. This may sound insignificant, and for traders that hold longer positions this wouldn't amount to too much, but I am in the HF trading spectrum so losing a tick at entry really costs me.

To your question about how it will translate into making money: If I can tighten up my entries by filtering out trades that are going against me (I.E my place in the queue is too close to the end of the line and the level is close to breaking) then I would have far more profitable entries.

There are 3 distinct types of entries that can be observed from this research.

1. Your position in the queue relative to your side's volume is stable, but the opposite side is also stable. This implies you will likely get filled in the middle of the current price level but you are not sure whether the level will break up or down.

2. Your position in the queue relative to your sides volume is very favorable, and the opposite sides volume is fairly week. For example you are position 10 in the queue for bids and there are a total of 200 bids, on the opposite side there are only 50 asks. So the most likely outcome here will be to fill your order prior to the 50 asks getting filled, then shortly after you get your fill, the 50 asks clear and you immediately see the price level move up and you are up by 1 tick.

3. Your position in the queue relative to your sides volume is very poor, and the opposite side is very strong. For example you are # 20 in line and there are only 30 contracts on your side, by contrast there are 300 contracts on the opposite side. In this case you will get your fill, and immediately the level will break and you will be upside down 1 tick.


So the goal of my research at a minimum is to identify type 3 above and cancel my order before the shoe drops. This would end up producing more type 1 and type 2 scenarios. Without this I am flying blind.

In type 2 scenarios I would immediately be profitable, and in type 1 I would at least not immediately suffer a level change against me. I would likely ride this one out and cancel if it starts too look bad, or let it fill me if it looks good.

If I can get around 70% type 1 and type 2 entries and only 30% type 3 entries then this would be a considerable success. Hope it makes sense. If you would like to know any other specifics let me know and I will be glad to share!

Ian

Visit my futures io Trade Journal Started this thread Reply With Quote
The following 6 users say Thank You to iantg for this post:
 
(login for full post details)
  #78 (permalink)
Market Wizard
Chicago Illinois USA
 
Experience: Advanced
Broker: IB, ToS
Trading: /ES, US Equities/Options
 
wldman's Avatar
 
Posts: 3,341 since Aug 2011
Thanks: 1,978 given, 8,827 received

That makes a lot of sense. There are a few guys on here, one notable being @artemiso , that might chime in. I just have one friend, a floor holdover that is active in that space. He know nothing about the mechanics of how their systems work. He simply manages a position that the computer gives him.

If the names Getco or Steve Schuler are familiar to you he is a local neighbor. When the time is right I might have enough rapport to make an introduction if you promise to focus on his advise for your plan.

Dan

Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to wldman for this post:
 
(login for full post details)
  #79 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Dan,

Wow! Thanks for reaching out, and of course I would welcome any introduction from anyone that has expertise in this space.

Thanks,

Ian


wldman View Post
That makes a lot of sense. There are a few guys on here, one notable being @artemiso , that might chime in. I just have one friend, a floor holdover that is active in that space. He know nothing about the mechanics of how their systems work. He simply manages a position that the computer gives him.

If the names Getco or Steve Schuler are familiar to you he is a local neighbor. When the time is right I might have enough rapport to make an introduction if you promise to focus on his advise for your plan.

Dan


Visit my futures io Trade Journal Started this thread Reply With Quote
The following user says Thank You to iantg for this post:
 
(login for full post details)
  #80 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received


wldman View Post
There are a few guys on here, one notable being @artemiso , that might chime in.

I agree this is definitely his area, but he's not using NT! He has posted a lot in the High Frequency Thread and has his own Ask Me Anything Thread, but his participation here is irregular.

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #81 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


iantg View Post
So I took my granularity level from 50 ticks down to 1 tick and tried to see if I could build a full sequence of trades from x contracts down to the last 1 contract before each level broke. After doing some research I have come to understand that this will never be achievable based on just the GetCurrentBidVolume() / GetCurrentAskVolume() alone. There are always going to be gaps in the data between each update as new contracts get added to the queue and transactions occur.

For starters, are you sampling on every trade ("1 tick" in NT lingo) or every market depth update? I'm not familiar with latest NT but a quick search up the API docs suggests this is what you need: https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdepth.htm

If you still see reconstruction issues while using the OnMarketDepth callback, my best guess is a common design pattern in these platforms where they aggregate all book state changes before dispatching the callback that your application logic (strategy) is enclosed in.

The reason for this design pattern is that it's the most sensible thing to do if you anticipate any queuing bottlenecks. Examples of these bottlenecks include the:
- expectation that the application logic is slow, often the case among retail users
- limited bandwidth between market data source and client

As a result, you might find gaps even inbetween calls of your OnMarketDepth handler.

This design pattern is even more common on the GUI side because platform developers safely assume that no screen is supposed to keep up with the messaging traffic. You can try this out yourself and call up your broker or use a separate application to click-trade a unique lot size for you on ES, and chances are that it will not print on your GUI. I've done this dozens of times and I've never been able to match my fill to a trade print on any retail platform's book viewer.



iantg View Post
In addition to this, I have implemented a queue position tracker to quantify where I am at in the queue. This works like this.

Sounds about right, no comment.



iantg View Post
2. Is there a specific ratio of bid / ask volumes present when entering a trade that will make it more likely or less likely to get filled in the following ways.

Yes, we know this without even needing to resort to data analysis. Just read the dozens of cases where people get caught spoofing and it's always the classic technique where they place a large bid on the other side of a small offer they're actually trying to fill, and then they cancel the bid once they're filled. If you trade in less-regulated markets, this effect is even more rampant.

So the obvious answer is that you're more likely to get filled on the thinner side of an asymmetric book. However this is also offset by worse PnL on that fill.



iantg View Post
1. How close am I to getting filled relative to the level breaking against me? For example if there are 20 contracts ahead of me that need to fill before I get filled, but only 30 contracts left total, then waiting in line to get filled would be bad. Because as soon as I get filled, the price level will change and I will be down 1 tick.

Not an obvious conclusion. Back in the day like 2008-2010 this would be a useful heuristic. Nowadays market makers are more competitive and a wiped level is more likely to replenish on the same side of the book than to be replenished on the other side of the book.

-----

My best advice for you is to collect your own raw data and build your own platform. It's near impossible to do analysis at this granularity without an open source peek into how the data is touched before your application logic. I know this sounds like a huge distraction, because building any meaningful platform is a 1+ year endeavor for a first timer, but that's generally the upfront cost you have to pay if you want to beat 1 tick of slippage on your round trip. No free lunch.

Reply With Quote
The following 5 users say Thank You to artemiso for this post:
 
(login for full post details)
  #82 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received

Also you mentioned some way to estimate your fills by assuming the volume needs to accumulate to your initial position. There's a couple of edge cases that you're probably already aware of, e.g. that you need to short circuit that logic if there's a price trade-through, but do test this thoroughly. I still get the code subtly wrong every now and then even after having done it many times over for different exchanges.

I'll save you a few months of work and tell you that the pessimistic case is very close to the actual case. So if you don't think you can make (save?) money using limit orders with your current analysis, you're probably not going to be able to do it with more accurate queue simulation.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #83 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Hi artemiso,

Thanks for providing me with some feedback on everything. I appreciate you taking the time to provide me with your insights. So here is where I am at with everything.

1. Regarding your comments below about the marketdepth event handler and the potential gaps. I have seen the gaps and have built a pretty neat workaround. NT support recently confirmed that the GetCurrentBidVolume() and GetCurrentAskVolume() methods are bar size specific. So you only get these updates on the bar update regardless of which event handler you call them from. When I moved my strategy from a 50 tick time series down to 1 tick, I did see more updates, but there were still gaps. So between updates, I am running back the actual transacted volumes of the bid and ask via the OnMarketData event handler which gives every change to the level 1 books independent of everything else. So I am deducting any transacted volume against my last GetCurrentBidVolume() and GetCurrentAskVoulme() update. Once I get a refreshed update, I reset my cumulative transaction count back to 0 and start again. I am able to capture the levels breaking fairly accurately in between updates and this accounts for all but the canceled volumes, which unfortunately I can't see between updates.

2. I have observed that you have a higher lightly-hood of getting on the right side of the action if you wait in the queue a few levels out, by the time your level comes up you may be in the top 25% to 50% of the queue, so you increase your likelihood of not getting filled at the end of the line and the level immediately breaking against you. The unfortunate downside to this approach is you end up with far less trades because you spend too much time waiting in line a few levels out. My proposed workaround, and the point of this recent project is to try to place orders to the weak side (Side where there was not a queue from other levels already working),and then just cancel when my estimated place in the queue relative to my side of the book and the opposite side of the book looked poor. When my queue position relative to my side of the book and the opposite side is attractive I would let it play out. I have already gotten this kind of working with around 50% / 50% split between positive and negative outcomes (Level break in my favor, level breaks against me). And typically submitting to the weak side at least with the ES you typically see the weak side break around 60% or higher, so I think I am beating the odds currently, but I imagine with further optimization I can get this figure even higher.

3. I think you are right with the recommendation of going to an independent platform for testing. I have an expertise in SQL and have built trading simulators in the past with this route, and will likely pursue this again for this endeavor. With this approach I can test almost every possible permutation of different entry volume ratios and throughout the various life cycles of trades see where canceling or riding out will play in my favor or run through my level against me.

This is definitely a field of research that I think will prove extremely fruitful but as you can imagine it is no small undertaking.

Thanks,

Ian


artemiso View Post
For starters, are you sampling on every trade ("1 tick" in NT lingo) or every market depth update? I'm not familiar with latest NT but a quick search up the API docs suggests this is what you need: https://ninjatrader.com/support/helpGuides/nt8/en-us/?onmarketdepth.htm

If you still see reconstruction issues while using the OnMarketDepth callback, my best guess is a common design pattern in these platforms where they aggregate all book state changes before dispatching the callback that your application logic (strategy) is enclosed in.

The reason for this design pattern is that it's the most sensible thing to do if you anticipate any queuing bottlenecks. Examples of these bottlenecks include the:
- expectation that the application logic is slow, often the case among retail users
- limited bandwidth between market data source and client

As a result, you might find gaps even inbetween calls of your OnMarketDepth handler.

This design pattern is even more common on the GUI side because platform developers safely assume that no screen is supposed to keep up with the messaging traffic. You can try this out yourself and call up your broker or use a separate application to click-trade a unique lot size for you on ES, and chances are that it will not print on your GUI. I've done this dozens of times and I've never been able to match my fill to a trade print on any retail platform's book viewer.




Sounds about right, no comment.




Yes, we know this without even needing to resort to data analysis. Just read the dozens of cases where people get caught spoofing and it's always the classic technique where they place a large bid on the other side of a small offer they're actually trying to fill, and then they cancel the bid once they're filled. If you trade in less-regulated markets, this effect is even more rampant.

So the obvious answer is that you're more likely to get filled on the thinner side of an asymmetric book. However this is also offset by worse PnL on that fill.




Not an obvious conclusion. Back in the day like 2008-2010 this would be a useful heuristic. Nowadays market makers are more competitive and a wiped level is more likely to replenish on the same side of the book than to be replenished on the other side of the book.

-----

My best advice for you is to collect your own raw data and build your own platform. It's near impossible to do analysis at this granularity without an open source peek into how the data is touched before your application logic. I know this sounds like a huge distraction, because building any meaningful platform is a 1+ year endeavor for a first timer, but that's generally the upfront cost you have to pay if you want to beat 1 tick of slippage on your round trip. No free lunch.


Visit my futures io Trade Journal Started this thread Reply With Quote
The following 5 users say Thank You to iantg for this post:
 
(login for full post details)
  #84 (permalink)
SF, CA/USA
 
Experience: Beginner
Platform: SC
Broker: Stage 5
Trading: NQ....that's what it boils down to
 
jackbravo's Avatar
 
Posts: 1,338 since Jun 2014
Thanks: 4,340 given, 2,389 received

*mind blown* hope you find much success with this effort

Sent using the https://futures.io%20mobile%20app

"It does not matter how slowly you go, as long as you do not stop." Confucius
Reply With Quote
The following user says Thank You to jackbravo for this post:
 
(login for full post details)
  #85 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


iantg View Post
Hi artemiso,

Thanks for providing me with some feedback on everything. I appreciate you taking the time to provide me with your insights. So here is where I am at with everything.

1. Regarding your comments below about the marketdepth event handler and the potential gaps. I have seen the gaps and have built a pretty neat workaround. NT support recently confirmed that the GetCurrentBidVolume() and GetCurrentAskVolume() methods are bar size specific. So you only get these updates on the bar update regardless of which event handler you call them from. When I moved my strategy from a 50 tick time series down to 1 tick, I did see more updates, but there were still gaps. So between updates, I am running back the actual transacted volumes of the bid and ask via the OnMarketData event handler which gives every change to the level 1 books independent of everything else. So I am deducting any transacted volume against my last GetCurrentBidVolume() and GetCurrentAskVoulme() update. Once I get a refreshed update, I reset my cumulative transaction count back to 0 and start again. I am able to capture the levels breaking fairly accurately in between updates and this accounts for all but the canceled volumes, which unfortunately I can't see between updates.

2. I have observed that you have a higher lightly-hood of getting on the right side of the action if you wait in the queue a few levels out, by the time your level comes up you may be in the top 25% to 50% of the queue, so you increase your likelihood of not getting filled at the end of the line and the level immediately breaking against you. The unfortunate downside to this approach is you end up with far less trades because you spend too much time waiting in line a few levels out. My proposed workaround, and the point of this recent project is to try to place orders to the weak side (Side where there was not a queue from other levels already working),and then just cancel when my estimated place in the queue relative to my side of the book and the opposite side of the book looked poor. When my queue position relative to my side of the book and the opposite side is attractive I would let it play out. I have already gotten this kind of working with around 50% / 50% split between positive and negative outcomes (Level break in my favor, level breaks against me). And typically submitting to the weak side at least with the ES you typically see the weak side break around 60% or higher, so I think I am beating the odds currently, but I imagine with further optimization I can get this figure even higher.

3. I think you are right with the recommendation of going to an independent platform for testing. I have an expertise in SQL and have built trading simulators in the past with this route, and will likely pursue this again for this endeavor. With this approach I can test almost every possible permutation of different entry volume ratios and throughout the various life cycles of trades see where canceling or riding out will play in my favor or run through my level against me.

This is definitely a field of research that I think will prove extremely fruitful but as you can imagine it is no small undertaking.

Thanks,

Ian

For most part it seems that you are going about this sensibly. A few remaining simple things:

1. It seems that you're thinking of this as a way to reduce transaction costs ("improve entries") for some more important, overarching strategy. This seems weird because there's lower hanging fruit ways to reduce transaction costs. If your goal is to make market making your core strategy, then it makes more sense.

2. Yeah, you have practically zero hope of getting a good queue position from interacting with the top of the book with NT. Also your cancels will be awful. There's a pretty sharp cutoff in round trip latencies at which your cancels matter or not and I can tell you that you're not close enough with NT.

3. PnL targets and stop losses should worsen your PnL but improve your cash flow, don't see the reason for you to use them here.

Reply With Quote
The following 3 users say Thank You to artemiso for this post:
 
(login for full post details)
  #86 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Hi artemiso,

Thank you for taking the time to follow up with me. I appreciate your insight.

As you can no doubt guess I am applying this to a HF trading strategy, so you can see why the 1 tick matters as I am more on the market making spectrum.


I have fairly decent speed with NT as I am co-locating in close proximity to my data provider. So while I am not in the league of direct market access, I would say I am at the higher end of retail. My testing with canceling live trades has been surprising fairly good thus far, so I think outside of the 9:30- 10:00 AM rush, I shouldn't hit too many snags. From your experience where do you see the lines in the sand around latency and NinjaTrader? I think most people run NT off of at least 10,20,50 ticks or more, so this creates a lag in and of itself. I run all of my execution related code off of the OnMarketData event handler, which captures every change in level 1 events. So this will execute faster than even running a 1 tick time frame for example. I see that you use think or swim, have you found this to have speed advantages over ninjatrader? If you have any ideas that would help improve latency that won't break the proverbial bank I am all ears.

But for sure, I could be on a fools errand and putting in the work all for not, but I got to give it a shot right?

Thanks,

Ian



artemiso View Post
For most part it seems that you are going about this sensibly. A few remaining simple things:

1. It seems that you're thinking of this as a way to reduce transaction costs ("improve entries") for some more important, overarching strategy. This seems weird because there's lower hanging fruit ways to reduce transaction costs. If your goal is to make market making your core strategy, then it makes more sense.

2. Yeah, you have practically zero hope of getting a good queue position from interacting with the top of the book with NT. Also your cancels will be awful. There's a pretty sharp cutoff in round trip latencies at which your cancels matter or not and I can tell you that you're not close enough with NT.

3. PnL targets and stop losses should worsen your PnL but improve your cash flow, don't see the reason for you to use them here.


Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #87 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


iantg View Post
As you can no doubt guess I am applying this to a HF trading strategy, so you can see why the 1 tick matters as I am more on the market making spectrum.


iantg View Post
I have fairly decent speed with NT as I am co-locating in close proximity to my data provider.

That really doesn't matter. You're likely still taking a handful of BGP hops and paying garbage collection on your application. The people who're looking to lift your trades 90+% of the time (I'm not throwing this number out randomly) aren't the guys in your same proximity hosting facility, they're the ones optimized to evaluate the value of your order and lift it a few thousand times between every 2 OnMarketData calls.


iantg View Post
From your experience where do you see the lines in the sand around latency and NinjaTrader?

The line in the sand is around 2-3 orders of magnitude faster than what you can achieve with the setup you've described.

It's good however that you have the sensible judgment to call it a "line in the sand", because many people don't grasp that it's an all-or-nothing scenario - if you're not across the line, you're practically competing with everyone else on equal latency grounds whether it's grandma from her iPhone app or someone in the same rack as you.

This isn't necessarily a bad thing, it's just that you are probably overspending on the wrong vendor services when you could've spent it on something with a more meaningful impact at your time scale, e.g. an alternate data source.

You'll have no problem canceling orders no doubt even during US cash open, it's just that you won't land the cancellations that matter. Almost always. Even during US T+1 session.



iantg View Post
I think most people run NT off of at least 10,20,50 ticks or more, so this creates a lag in and of itself. I run all of my execution related code off of the OnMarketData event handler, which captures every change in level 1 events. So this will execute faster than even running a 1 tick time frame for example.

That's not a very meaningful optimization. Deallocation in .NET garbage collection likely takes more cycles than most things you can cram into 10 passes of your main event loop.



iantg View Post
I see that you use think or swim, have you found this to have speed advantages over ninjatrader?

That's just a reference to an old joke that before my team had a working GUI, I needed to confirm our new platform was working properly, so someone fired up thinkorswim and compared our feed against that.

Reply With Quote
The following 4 users say Thank You to artemiso for this post:
 
(login for full post details)
  #88 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received

By the way, one cheap way you can know the marginal latency benefit of your proximity hosting and/or application is to submit orders in some benchmark pattern, e.g. replace->acknowledged->cancel.

Then ask your broker to look up your orders in their risk platform and ask for the exact timestamps of your orders. They'll be able to see actually how fast your orders are from the frame of reference of the exchange, if they use CME's front end for risk, or from the execution vendor's gateway application.

Now you can actually modify your proximity hosting decisions and see the impact, i.e. run the same benchmark side-by-side on your local workstation.

CME's front end risk unfortunately only has millisecond granularity but it's good enough if you're just proximity hosting. You can get your broker to generate a report like this in CSV (see below) through Firmsoft. In this example, I'm canceling my order shortly after its modify has been acknowledged and some market data event arrived to trigger the cancel, and from CME's perspective, it takes place within the same millisecond. It actually takes much less but CME's clock isn't precise enough.



With NT and proximity hosting at 350 E Cermak I estimate you're going to take several milliseconds. I doubt it's going to be meaningfully faster than running it on your own desktop. And I doubt 1 tick or 50 tick resolution is going to make a difference at all.

Reply With Quote
The following 6 users say Thank You to artemiso for this post:
 
(login for full post details)
  #89 (permalink)
Market Wizard
Chicago Illinois USA
 
Experience: Advanced
Broker: IB, ToS
Trading: /ES, US Equities/Options
 
wldman's Avatar
 
Posts: 3,341 since Aug 2011
Thanks: 1,978 given, 8,827 received

This is why I stay connected to FIO. @artemiso @iantg

Inspiring to see the effort and the willingness to share. Those qualities and the spirit they reflect represent values that I wish where more prevalent in the world.

Kudos and respect to both of you guys.

Please leave some crumbs for the old school guys that still put their hands in the air and shout at the monitor for execution.

artemiso, are guys like you gonna swallow guys like me? Do I need get my application in at Walmart?

Dan

Visit my futures io Trade Journal Reply With Quote
The following 7 users say Thank You to wldman for this post:
 
(login for full post details)
  #90 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


wldman View Post
artemiso, are guys like you gonna swallow guys like me? Do I need get my application in at Walmart?

You're fine. I'm probably more of the one in need of a new job at the rate that our industry is consolidating and volatility keeps shrinking.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #91 (permalink)
Bay Area California
 
Experience: None
Platform: TT T4
Broker: Phillip Capital
Trading: Futures
 
Posts: 800 since Nov 2011
Thanks: 873 given, 881 received


artemiso View Post
You're fine. I'm probably more of the one in need of a new job at the rate that our industry is consolidating and volatility keeps shrinking.



Iím glad Iím not the only one who feels this way.


Sent from my iPhone using Tapatalk

.
Reply With Quote
The following user says Thank You to addchild for this post:
 
(login for full post details)
  #92 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

artemiso,

I appreciate your sharing your insight. I definitely don't have super speed at this point in the game, but my lo-location (even being still on the retail level) is giving me a decent improvement over what I I tested from my local machine by a decent margin. Here is a snapshot of some of the time stamps of my live trading.





By comparison, I tested canceling from my house (I don't have the greatest internet speed anyway) and it took around 100 to 300 milliseconds on average from Canceled Submitted to Cancelled. Co-located I am getting this down to 10 milliseconds. So I would characterize 100- 200 millisecond latency as nearly unworkable, but 10 milliseconds I can work with.... though I have to run my cancellation logic more conservative and look further out. In my world, if the volume on my side ever gets below 50, I should cancel. In your world, you could likely wait until this got near 1 and then pull the trigger.

So just out of curiosity, and I respect if you don't want to revile anything:

Are you using something like R | Diamond API and co-locating with Rythmic, or are you at the direct market access level doing it full custom? Any details you are willing to share in this regard I would love to take notes on.

Obviously I can't afford to scale too heavy at this point, but if there is a different route that you would recommend for someone in the < $500 per month co-location range I would love to hear your thoughts. I am open to building my own trading platform via any number of APIs, I am a programmer by trade, so I wouldn't be too intimidated, but I haven't really seen a need for it yet. If you have any thoughts on this topic I am all ears.


Thanks,

Ian

artemiso View Post
By the way, one cheap way you can know the marginal latency benefit of your proximity hosting and/or application is to submit orders in some benchmark pattern, e.g. replace->acknowledged->cancel.

Then ask your broker to look up your orders in their risk platform and ask for the exact timestamps of your orders. They'll be able to see actually how fast your orders are from the frame of reference of the exchange, if they use CME's front end for risk, or from the execution vendor's gateway application.

Now you can actually modify your proximity hosting decisions and see the impact, i.e. run the same benchmark side-by-side on your local workstation.

CME's front end risk unfortunately only has millisecond granularity but it's good enough if you're just proximity hosting. You can get your broker to generate a report like this in CSV (see below) through Firmsoft. In this example, I'm canceling my order shortly after its modify has been acknowledged and some market data event arrived to trigger the cancel, and from CME's perspective, it takes place within the same millisecond. It actually takes much less but CME's clock isn't precise enough.



With NT and proximity hosting at 350 E Cermak I estimate you're going to take several milliseconds. I doubt it's going to be meaningfully faster than running it on your own desktop. And I doubt 1 tick or 50 tick resolution is going to make a difference at all.


Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #93 (permalink)
Estonia
 
Experience: None
Platform: windows nt
Broker: windows nt
Trading: everything what moves
 
Posts: 10 since Mar 2014
Thanks: 170 given, 12 received

Sam has written little tester strat for NT, hope it helps:
https://futures.io/ninjatrader/30662-speed-test.html#post390008

Reply With Quote
The following user says Thank You to soultrader for this post:
 
(login for full post details)
  #94 (permalink)
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 3,966 since Dec 2013
Thanks: 3,258 given, 7,769 received


iantg View Post
By comparison, I tested canceling from my house (I don't have the greatest internet speed anyway) and it took around 100 to 300 milliseconds on average from Canceled Submitted to Cancelled. Co-located I am getting this down to 10 milliseconds.

Just another 'timing' data point to put your 10 milliseconds in perspective. I use XTrader Pro and have servers at both Aurora and Cermak. I haven't looked at my my actual message logs (will try tonite), but looking at my fill logs (for Aurora trading CL) I can see that the Autospreader takes between 1 and 2 milliseconds to leg a spread. Meaning I receive a fill message for one leg of the spread, and XTrader reacts, sends the other leg of the spread, which in turn is filled within 1-2 milliseconds of the original fill. I think you'll find what I'm using is a pretty standard setup for professional/institutional traders, but to @artemiso I'm a dinosaur.

Also - less relevant to the conversation, but since you mentioned home desktop latency - I monitor my connection to my servers continually and for the last 240 pings/1 hour, round trip latency between my desktop (Houston) and Cermak (Chicago) is 31.2 milliseconds.

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #95 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

First of all I just wanted to thank everyone that has recently contributed and helped me with this recent project.

@artemiso, @SMCJB @wldman, @soultrader, @jackbravo and anyone else that has chimed in recently.

After considering a number of ways to go about this research I determined that running this through any sort of retail trading platform doing countless SIM trades to model outcomes simply wouldn't cut it. So I took a different rout.

I used ninjatrader and wrote a script to extract the most granular output I could concerning level 1 events. I collected every trade with all resting volumes and all transacted volumes around each side. I did this over a few months worth of data just to confirm the patterns and validate what I was seeing. I kept it simple and just used a 1 tick time frame and pulled all the resting volume data on the OnBarUpdate event handler. For transacted volume I pulled this through the OneMarketData event handler. I could have technically gotten more granular with the resting volumes using OnMarketData but as I was mostly only concerned with the starting volumes I figured the OnBarUpdate would be good enough.

Let me back up and explain what exactly I am trying to do first because this gets fairly technical fairly quick and just to lay the ground work you have to have a basic understanding of the following.



Strong Side / Weak Side: This may be common knowledge to some and new to others. Also maybe my use of the terminology could be throwing people off a bit (Strong / Weak... I just came up with this, I am not sure anyone thinks of it this way)

The image above describes it though. You have one side with resting orders 10-20 levels out and the other side with nothing. As you can't submit limit long orders above the market, or limit short orders below the market, your only option for this (Weak side) is to just wait until the new level is created and then submit. So the weak side will typically spontaneous pick up new limit orders, whereas the strong side will just bring down orders from levels further out that move. So what I have observed is the following 2 points that everyone should know.

1. The weak side will have thinner volume naturally, which improves your chances of getting filled considerably.
2. Also by having thin volume, you have a much higher chance of this side breaking against you after you get filled.

So this double edge sword creates a paradox. Do you want to get the most possible fills / trades, or do you want to protect your P&L by not being on the side most likely to break. So my quest in my recent project was to find a way to thread the proverbial needle of doom and have my cake and eat it to. Ambitious? Sure, Stupid? Probably. Advisable? No. I wouldn't recommend anyone try this. It is like playing Tetris on level 120 while drunk.

Now to the results of my research (Excel file enclosed)

I consolidated over 2,000 price level changes down to just a few specific variables that I could run simulations on in excel.

1. Starting level change: (Was the newly created level up or down from the previous level) This identifies the weak side / strong side. If the new level moved up it would be considered a weak side to submit a buy order, but strong side to submit a sell order. The image above along with my previous explanation will hopefully give this context.

2. Starting volume for both the bid and ask
3. Total transacted volume at the bid and ask separated out for each level.
4. Would I have gotten filled or not. With points 2-3 I can test the assumption that I submit my order immediately as the level change first occurs, to test if I would have gotten filled, I just measure the total transacted volume - the starting volume. If TV > SV then I assume a fill. I work this for both the bid and ask independently.
5. The Ending Level. This is the second data point we need to understand what occurred. In this example we are always betting on the weak side. So if Staring level = Up and Ending Level = Up, this produces a Win. Whereas Starting Level = Down and ending level = Up produces a loss. So any two "like" events is a win. Up, Up, Down, Down, but any alternating events is a loss. Up, Down, or Down, Up.
6. The actual bet. The win loss criteria just determines if you get an entry up 1 tick or down 1 tick due to how the first level clears after your starting level. This in and of itself doesn't translate to a real trading outcome but simply measures how effective the first sequence of your trade goes. If you were only aiming for 1 tick profit or 1 tick loss, then this may be a 75% good proxy for it, but this is only meant to quantify if you get a decent entry or not.

So from these thousands of data points, I was able to easily run every possible permutation of different entry logic to see if I could find anyway to beat the built in bias against me. By the way, the build in strong side / weak side bias in the ES is 75% favorable to the strong side. Which means on every single price level 75% of the time the weak side will break. This is something that I think is critical for every trader to know. The primary reason for this is just due to the nature of the strong side / weak side explanation I have provided above. The other explanation that I can think of is the presence of spoofing the weak side. (I am doing a separate statistical analysis to quantify this in a future post.) But for now my initial hypothesis is that this occurs at least twice as much on the weak side.

Now the conclusions from my testing: After testing every significant possible volume ratio, and absolute volume setting, I have found several combinations that beat the natural odds. They don't do this be a huge margin and truthfully at best I can only cut this down to close to a 50%/ 50% win loss ratio. But considering with no filtering you would typically have a 25%/ 75% Win Loss ratio, I would characterize this as a significant break though.

Now this isn't good enough in and of itself to constitute a true edge, but with proper canceling logic in place throughout the life of each trade moving from Accepted > Working > Filled, you have a shot at getting the win loss ratio up higher. I would say 55% on the low side, up to as high as 75% if you nail every cancellation you can spot and you have perfect latency. I don't think you will ever do better than that because most spoofing moves drop the entire level with no warning, so these will typically get you with 0 time to react.

As you can imagine, you can get significantly more fills on the weak-side, but this is clearly a land mind, so that is why the idea of threading this needle seems so intriguing. If one could pull this off, their prize would likely be at least 2x the number of fills they would get if only submitted from the strong side multiple levels out to gain optimal queue placement.

In the coming days I am going to code this and post my market replay / Live SIM results.

Happy Trading!

Ian

Attached Files
Register to download File Type: xlsx Threading the Needle.xlsx (559.1 KB, 21 views)
Visit my futures io Trade Journal Started this thread Reply With Quote
The following 5 users say Thank You to iantg for this post:
 
(login for full post details)
  #96 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


iantg View Post
By comparison, I tested canceling from my house (I don't have the greatest internet speed anyway) and it took around 100 to 300 milliseconds on average from Canceled Submitted to Cancelled. Co-located I am getting this down to 10 milliseconds. So I would characterize 100- 200 millisecond latency as nearly unworkable, but 10 milliseconds I can work with....

Have you tried calculating the marginal value of those additional 190-290 milliseconds?



iantg View Post
Are you using something like R | Diamond API and co-locating with Rythmic, or are you at the direct market access level doing it full custom? Any details you are willing to share in this regard I would love to take notes on.

Everything is custom. On CME everyone half-serious is bypassing broker risk because the exchange has a good risk platform. I use a mix of FPGA and x86 for the hardware.



iantg View Post
Obviously I can't afford to scale too heavy at this point, but if there is a different route that you would recommend for someone in the < $500 per month co-location range I would love to hear your thoughts. I am open to building my own trading platform via any number of APIs, I am a programmer by trade, so I wouldn't be too intimidated, but I haven't really seen a need for it yet. If you have any thoughts on this topic I am all ears.

>=10 ms is low tier even for proximity hosting. The line that most people use between Equinix and CME is about 0.293 ms one-way host-to-host. If you're experiencing a lot more than that, it's because something is wrong at the ends. If for example you know the bandwidth provider for your data/execution provider is fixed, you could ask your hosting provider to see if they can patch you only onto that provider's network. If they can't do it, the worst case cost should be $250-300 per month on top of your existing costs.

If you really want the mental satisfaction of having nicer round trips, you can still get actual colo for <$500 per month but it just takes careful snooping. Some vendors stock up an entire rack with blades and just need a few remaining people to take out the slots that are unused so they can recoup their hosting costs.

Of course you can get the cheapest gains by replacing your application, but it's also the most time consuming.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
(login for full post details)
  #97 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received

And you don't have to thank me, I could be bullsh*tting completely for all you know. Test things out yourselves and be very skeptical of people offering advice online.

Good luck.

Reply With Quote
The following 2 users say Thank You to artemiso for this post:
 
(login for full post details)
  #98 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received

Another piece of advice: which method call are you using for your timing measurements? C# has a few timer calls but the trivial ones have poor accuracy.

Reply With Quote
The following user says Thank You to artemiso for this post:
 
(login for full post details)
  #99 (permalink)
charlotte nc
 
Experience: Master
Platform: ninjatrader
Broker: NinjaTrader
Trading: Emini (ES, YM, NQ, ect.)
 
Posts: 398 since Jan 2015
Thanks: 76 given, 1,081 received

Hi artemiso,

I do feel compelled to thank you, because I have done my research on you, and you came voluntarily to this thread because you were well recommended by several members on here with great reputations. I honestly do appreciate your insight because you are much further in the game than I am, and the feedback you share will be taken seriously, though I am a lifetime skeptic and will also take your advise to test everything myself and continue to do my own research.


artemiso View Post
And you don't have to thank me, I could be bullsh*tting completely for all you know. Test things out yourselves and be very skeptical of people offering advice online.

Good luck.


I have done some testing on my latency and how they impact my strategies. I am going the VPS route at the moment, but I may move to a dedicated server next. I think this is likely why I am a little slow at the moment.... that and obviously I am using a platform that primarily caters to chart traders and moves slower than something that was just executing my strategy and nothing else. I might would consider eventually building my own system, I know that the CME has a number of APIs that contain standard market events and some of the event handlers that I would need, but I would still have to go the retail route of either going with CQG / Continuum or Rithmic for data. So I would end up going Application > Data Provider > Broker Risk > Exchange, whereas DMA guys can go Application > Exchange. Maybe I am wrong... but I always assumed that any savings I picked up from a quicker application would be offset by the additional steps I am going through presently.

Below you mentioned something that very much interests me. When you mention that I won't be able to land the cancellations that matter I am envisioning a specific type of cancellation scenario and I was hoping that you could give me some verification of my thinking. I know that right now I can cancel based on volume related criteria provided that it is looking far enough out... But the scenario I am very curious about is a price level change. Let's say the market moves the bid price down from 2746 to 2745.75 and I had a buy order at 2746. If you have an event handler catching this price change at the most granular level would you ever have any chance at all of pulling off a cancel? I have never been able to nail one myself, but again I am on the slow side. I am just wondering if this type of cancellation is possible to achieve at all. I mean obviously if the market clears a level it is assumed that all volume traded at that level, but with spoofing, flipping, and not knowing the exact sequence of events at the exchange, I just figured I would ask, as this was the first scenario that came to mind, and I would love to know if this is even possible and if so where the line is on latency for this.

Thank you for your time and input.

Ian



artemiso View Post

It's good however that you have the sensible judgment to call it a "line in the sand", because many people don't grasp that it's an all-or-nothing scenario - if you're not across the line, you're practically competing with everyone else on equal latency grounds whether it's grandma from her iPhone app or someone in the same rack as you.

This isn't necessarily a bad thing, it's just that you are probably overspending on the wrong vendor services when you could've spent it on something with a more meaningful impact at your time scale, e.g. an alternate data source.

You'll have no problem canceling orders no doubt even during US cash open, it's just that you won't land the cancellations that matter. Almost always. Even during US T+1 session.




artemiso View Post
Have you tried calculating the marginal value of those additional 190-290 milliseconds?


Everything is custom. On CME everyone half-serious is bypassing broker risk because the exchange has a good risk platform. I use a mix of FPGA and x86 for the hardware.


>=10 ms is low tier even for proximity hosting. The line that most people use between Equinix and CME is about 0.293 ms one-way host-to-host. If you're experiencing a lot more than that, it's because something is wrong at the ends. If for example you know the bandwidth provider for your data/execution provider is fixed, you could ask your hosting provider to see if they can patch you only onto that provider's network. If they can't do it, the worst case cost should be $250-300 per month on top of your existing costs.

If you really want the mental satisfaction of having nicer round trips, you can still get actual colo for <$500 per month but it just takes careful snooping. Some vendors stock up an entire rack with blades and just need a few remaining people to take out the slots that are unused so they can recoup their hosting costs.

Of course you can get the cheapest gains by replacing your application, but it's also the most time consuming.


Regarding the timing measurements, I am just using the standard NT logs. I am not programming anything myself to pick up the time stamps. They are .Net, so they are going with C#.



artemiso View Post
Another piece of advice: which method call are you using for your timing measurements? C# has a few timer calls but the trivial ones have poor accuracy.


Visit my futures io Trade Journal Started this thread Reply With Quote
The following 2 users say Thank You to iantg for this post:
 
(login for full post details)
  #100 (permalink)
Legendary Bought TSLA at 880
New York, NY
 
Experience: Beginner
Platform: Vanguard 401k
Broker: Yahoo Finance
Trading: Mutual funds
 
Posts: 1,127 since Jul 2012
Thanks: 765 given, 2,534 received


iantg View Post
I might would consider eventually building my own system, I know that the CME has a number of APIs that contain standard market events and some of the event handlers that I would need, but I would still have to go the retail route of either going with CQG / Continuum or Rithmic for data. So I would end up going Application > Data Provider > Broker Risk > Exchange, whereas DMA guys can go Application > Exchange. Maybe I am wrong... but I always assumed that any savings I picked up from a quicker application would be offset by the additional steps I am going through presently.

You do need enough margin capital (YMMV but say 250-500k) and volume for your broker to let you bypass risk, because it takes up some attention span of their trading desk, risk and compliance.



iantg View Post
Let's say the market moves the bid price down from 2746 to 2745.75 and I had a buy order at 2746. If you have an event handler catching this price change at the most granular level would you ever have any chance at all of pulling off a cancel? I have never been able to nail one myself, but again I am on the slow side. I am just wondering if this type of cancellation is possible to achieve at all.

There are 2 separate matters of staleness of your application and quality of your cancellation here.

1. A properly designed application should, by construction, in a pure FIFO market, never allow you to successfully cancel a bid that was resting at 2746 and show you that the best bid has gone below your order. By the time this scenario shows up on the feed, it's already too late.

The only edge case where you can successfully cancel in this scenario is that you placed that bid close enough to the price move (which is quantifiable, but that requires a longer post) that the feed has yet to disseminate that you and/or other parties have improved the best bid, however that doesn't seem like the scenario you're describing to me.

I said a pure FIFO market because there are some markets that allow you to retroactively cancel that bid but that's another issue.

2. The second issue that I'm talking about takes place in a different scenario, which is, your feed tells you that the BB is @ 2746 and your order has been accepted for long enough @ 2746, is there some way to predict that the market will move against you in the next dt time and pull that order in a way that improves your PnL substantially enough to pay off your marginal expenditure on latency improvements? It's possible, but you need to be faster than what's possible for proximity hosting + NT.


iantg View Post
Regarding the timing measurements, I am just using the standard NT logs. I am not programming anything myself to pick up the time stamps. They are .Net, so they are going with C#.

I've dealt with external platforms, data and libraries before. One of the less productive, but necessary evils when you use external components is that you need to be able to trust your timestamps.

What's 12:34:56.789 to NT? Exchange timestamp for the event? Vendor timestamp? PC local time? PC local time when it reaches which part of user space? You could, for example, timestamp it yourself in the strategy layer and estimate the duration between that and your own timestamp, which is useful for knowing how fast the application itself is (e.g. in book construction).

For .NET I believe you need to read the literature around the Win32 API (QueryPerformanceCounter) or home brew your own from unmanaged C++ and assembly.

Reply With Quote
The following 4 users say Thank You to artemiso for this post:


futures io Trading Community Trading Journals > Outside the Box and then some....


Last Updated on June 23, 2018


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1500 prizes from Topstep!

February
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March