NexusFi: Find Your Edge


Home Menu

 





How do you decide when an automated strategy is good enough to take live?


Discussion in Emini and Emicro Index

Updated
      Top Posters
    1. looks_one 303Trader with 7 posts (7 thanks)
    2. looks_two SydAlgoTrader with 4 posts (4 thanks)
    3. looks_3 sixtyseven with 3 posts (6 thanks)
    4. looks_4 OldGerman with 2 posts (2 thanks)
      Best Posters
    1. looks_one TradingOgre with 2 thanks per post
    2. looks_two sixtyseven with 2 thanks per post
    3. looks_3 303Trader with 1 thanks per post
    4. looks_4 SydAlgoTrader with 1 thanks per post
    1. trending_up 5,294 views
    2. thumb_up 42 thanks given
    3. group 12 followers
    1. forum 29 posts
    2. attach_file 4 attachments




 
Search this Thread

How do you decide when an automated strategy is good enough to take live?

  #11 (permalink)
 kanepa 
philadelphia pa
 
Experience: Intermediate
Platform: ninja
Broker: NinjaTrader Brokerage
Trading: es
Posts: 202 since Jul 2017
Thanks Given: 348
Thanks Received: 409

Hi. You can check out my journal. I started system trading from summer of 2017 and you could see up and downs of my system.

1. Commission. You need to add them into calculation.
2. Add 1 slippage for every trade to see if it survives.
3. Need to go back at least 08 and 09 to see if system survived and profitable in bear market.


-How do you decide when a strategy is good enough to take live, and what data do you use to determine this?

at least 100% return annually on historic max DD then forwarding test for a month.

-Is this simple method of backtesting a reliable qualifier for a strategy? (I'm worried I've simply optimized my strategy for this data set and this won't be representative of how it performs live.)

No. I would not trade those numbers at this time. but please feel free to forward test it with SIM.

Hope you find a good strategy. Good luck!

Visit my NexusFi Trade Journal Reply With Quote
Thanked by:

Can you help answer these questions
from other members on NexusFi?
ZombieSqueeze
Platforms and Indicators
Better Renko Gaps
The Elite Circle
NexusFi Journal Challenge - April 2024
Feedback and Announcements
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
Are there any eval firms that allow you to sink to your …
Traders Hideout
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
61 thanks
Funded Trader platforms
39 thanks
NexusFi site changelog and issues/problem reporting
26 thanks
GFIs1 1 DAX trade per day journal
18 thanks
The Program
18 thanks
  #12 (permalink)
303Trader
Denver CO
 
Posts: 7 since Dec 2018
Thanks Given: 3
Thanks Received: 7

Thanks Ian. See my responses in red.


iantg View Post
1. It looks like you have data going from 2011 to 2018. So does this imply that over the life of 8 years the system made roughly $15K after commissions? This would be evenly divided equal to roughly $2k per year. Is this what your results are showing or am I missing something? I think you can imagine the answer if the question, "Should I trade a system live that produces $2k per year in back testing." But maybe I am missing how your results are displayed.
No, you're not missing anything. That's what the system is showing. What do you consider to be a reasonable and obtainable goal for yearly profit, trading one contract?


2. It looks like you are almost always only taking long positions. Is this by design or is something in your code misfiring?
Nothing misfiring. After running the Strategy Analyzer it looks like my entry criteria are only effective for long entries.

3. What exactly was your testing on? Strategy Analyzer, Market Replay, Live SIM, etc.? Each of these have levels of accuracy that can impact your results.
I have only tested using Strategy Analyzer. Clearly this isn't enough.

4. What is the latency sensitivity of your alpha signal? Some strategies test well, but in live trading speed crushes them. Some signals you can catch from home on a laptop over the internet (Market Orders with a long term trade), Some you would need a VPN / VPS with: Limit order / speed sensitive strategies. And some signals you can't reach in time with NinjaTrader no matter what you do.
I'm hoping this won't be a problem for my strategy. I'm trading based on a five minute chart.

In order to test a realistic live trading scenario with NT you should do the following.

Run your test with Market Replay, and put your code in the level 1 data event handler. (OnMarketData). The OnBarUpdate front runs the Level 1 and Level 2 feed and can give you a glimpse into the future if your alpha signals are derived from the level 1 feed. No that Market Replay assumes 0 latency. If you want to really simulate live trading types of latency, you can build in some latency in your code yourself before you release your order.

If you can get passed market replay, then move onto Live SIM. The simulate around 200 MS of latency and the live feed is the live feed. Here you can use the OnBarUpdate to get back some of the lag from your interenet + broker to exchange latency.

Also put in your commissions.... You might be okay based on what I saw, but once you have realistic latency to measure against, who knows.

If you can pass these benchmarks and have a positive expectancy you should be fine.

This is excellent info and gives me something to shoot for. Thank you!!

Ian



OldGerman View Post
If I look at your equity, it looks like the trades will be held for a longer period of time. This requires a VPS. See Speedytradingservers.com in this forum or look at Google.

Could you elaborate on this? I would think this would be something that's necessary for very short time frame trading (my avg. time in market is 93minutes). I must be missing something.



kanepa View Post
Hi. You can check out my journal. I started system trading from summer of 2017 and you could see up and downs of my system.

1. Commission. You need to add them into calculation.
2. Add 1 slippage for every trade to see if it survives.
3. Need to go back at least 08 and 09 to see if system survived and profitable in bear market.

I'll definitely take a look at your journal. FYI, If I add 1 slippage my net profit decreases by ~15%. Yikes.

Thank you everyone for your input. I was hoping to come to this forum and be humbled. It seems to be working.

Reply With Quote
Thanked by:
  #13 (permalink)
 
OldGerman's Avatar
 OldGerman 
Münster Germany
Legendary Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader, ProRealtime
Broker: NinjaTrader Brokerage
Trading: FDXM, FDAX, ES, YM, 6B and 6E
Posts: 588 since Mar 2015
Thanks Given: 656
Thanks Received: 1,290



303Trader View Post
Could you elaborate on this? I would think this would be something that's necessary for very short time frame trading (my avg. time in market is 93minutes). I must be missing something.

I use my VPS because I can not secure the internet connection for a long time.
VPS also offer a very low latency. But that should not be so important on most systems.

The biggest risk you can ever take is not betting on yourself!
(Bill Williams +)
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #14 (permalink)
 centaurer 
south africa
 
Posts: 169 since Dec 2018

A strategy doesn't have to go from $0 to 2X your networth on margin.

I am interested in getting a strategy to real money ASAP because it is like a bug, it is not going to have that long of a life anyway.

As it performs better it gets more money to trade and as it performs worse it gets less until it dies.

I do have a real problem with optimization in this context though. All that optimization is doing is more finely tuning the model to conditions in the past that we know will not hold tomorrow. You are really optimizing the brittleness of the model most likely.

What I want to see from my models is that the strategy does not completely fall apart when used on another highly correlated time series.

If something blows the doors off ES but gets killed on SPY it is far more likely your model is simply over fitted to a random realization of the time series for ES than capturing and exploiting a structure of ES that is not part of SPY when we know the time series has the same generating process. Then I want to see SPY vs QQQ, NQ vs ES, YM vs SPY, etc..

You also need to look at as a baseline at the profit/loss if you had just gone long SPY and held since 2011 then normalize the leverage in ES. If you are not beating that then what is the point of trading. A system is not good because it is profitable, a trading system is good because it beats by and hold over the same period.

Don't forget too a purely random bad strategy is going to be between 40%-60% win/loss rate. I mean I would love to find a strategy that loses 95% of the time so I can just flip things. I agree with Ozquant 65% rule of thumb although I am ok with 60%. You need to give yourself a model error/noise cushion to stay above pure randomness.

Reply With Quote
  #15 (permalink)
 bchip 
Torino, Italy
 
Experience: Advanced
Platform: TradeStation
Trading: ES,YM,CL,GC
Posts: 132 since Sep 2017
Thanks Given: 160
Thanks Received: 116


303Trader View Post
Hi all. I'm new here and this is my first post. I'm super excited to have found this community!

I've been working on building an automated ES trading strategy for the last several months. I've optimized and backtested and I believe this strategy to be profitable. My goal is to take this strategy live starting in January but I'm getting cold feet. I'd like to gain some confidence from more experienced traders that I'm not fooling myself and I'm making a well informed decision to take this strategy live.

I've optimized my strategy using data from 01/2011 - 12/18. Backtesting using the optimized parameters over the same period I get the results shown in the attached pictures.

Something I noticed is that you used the full historical data set for optimization?
That would just be curve fitting the rules.

My 2c is that when you test a system, you must always test against out-of-sample data.

Maybe read Kevin Davey's book on "Building an algo", definitely worth a read.

Reply With Quote
Thanked by:
  #16 (permalink)
 
FlyingMonkey's Avatar
 FlyingMonkey 
Los Angeles CA
Legendary Market Wizard
 
Experience: Intermediate
Platform: NT8 & TOS
Trading: Futures & Stocks
Posts: 654 since Jan 2016
Thanks Given: 2,970
Thanks Received: 2,289


bchip View Post
My 2c is that when you test a system, you must always test against out-of-sample data.

This. Building a strategy that optimizes well and gives you a great back test is easy (and I agree with previous posters, your result is not yet a great backtest). Building an edge that tests well on out of sample data using a walk-forward test is difficult. And a test that performs well on live data (in sim) is the final step before unleashing on real money.

Give him the stick.

Don't give him the stick!
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #17 (permalink)
303Trader
Denver CO
 
Posts: 7 since Dec 2018
Thanks Given: 3
Thanks Received: 7


bchip View Post
Maybe read Kevin Davey's book on "Building an algo", definitely worth a read.

I just finished reading Kevin's book a few weeks ago. It definitely put things in perspective and answered many of my questions.


FlyingMonkey View Post
This. Building a strategy that optimizes well and gives you a great back test is easy (and I agree with previous posters, your result is not yet a great back-test). Building an edge that tests well on out of sample data using a walk-forward test is difficult. And a test that performs well on live data (in sim) is the final step before unleashing on real money.

I'm finding this to be very, very true. Since my original post I've increased back-test profitability and win rate substantially. However, as soon as I move to out of sample data it falls apart. Furthermore, I've been live sim trading this strategy since the beginning of the year (I figured why not?) and it's doing extremely well! It's infuriating and fascinating all at the same time.

I'm still convinced that my entry signal can be reliable. The two things I'm really struggling with is how to scale my entry parameters to match market volatility and how to let my winning trades run. I'm also still baffled on why this strategy seems to hate short trades. If I can get these things figured out maybe I'll have something.



Reply With Quote
Thanked by:
  #18 (permalink)
 sixtyseven 
Golden Bay, New Zealand
 
Experience: Beginner
Platform: Sierra Chart
Trading: ES, NQ
Posts: 186 since May 2012
Thanks Given: 851
Thanks Received: 337


303Trader View Post
Since my original post I've increased back-test profitability and win rate substantially. However, as soon as I move to out of sample data it falls apart.

It wasn't clear to me if you are still working on the same basic strategy, or whether you have moved on.

But just in case - if you test the in-sample against the OOS, modify and re-test then the OOS effectively becomes in-sample. You should only peak at the OOS once. If it doesn't work, then start again from scratch.

The advantage of 'algos', is that you can have a lot of them, which keeps the individual bet size low, which lowers max drawdown. Since you need a lot of them, you need to develop a process so you can churn them out. You'll be relatively indifferent if it works or not. It should be expected that most won't work. Keep in mind, you can get decent OOS results, which in reality is still random noise. That's another reason you want multiple systems.

Reply With Quote
  #19 (permalink)
 centaurer 
south africa
 
Posts: 169 since Dec 2018

That PNL is obviously over fitted. It is not reasonable that a system 20 bags and has a life span of that long. Of course falling apart out of sample is a classic symptom of over fitting.

IMO testing back to 2011 just makes no sense in general.

At the most I would use something like whatever the data providers back fill on ticks is, this is your DB. View it as a rolling window that the stale data falls out of. To me that kills two birds with one stone as far as data curation and dumping stale data.

If you start going back longer IMO you need some kind of regime switching classifier.

Reply With Quote
  #20 (permalink)
 SidewalkAerobics 
Los Angels
 
Experience: Intermediate
Platform: MultiChart
Trading: Emini ES
Posts: 115 since Aug 2018
Thanks Given: 173
Thanks Received: 71



centaurer View Post
That PNL is obviously over fitted. It is not reasonable that a system 20 bags and has a life span of that long. Of course falling apart out of sample is a classic symptom of over fitting.

IMO testing back to 2011 just makes no sense in general.

At the most I would use something like whatever the data providers back fill on ticks is, this is your DB. View it as a rolling window that the stale data falls out of. To me that kills two birds with one stone as far as data curation and dumping stale data.

If you start going back longer IMO you need some kind of regime switching classifier.



Centaurer, what is "out of sample data"? I use TradeStation and don't see it there. Any help on this would be appreciated.

Reply With Quote
Thanked by:




Last Updated on February 18, 2019


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts