Taking a Trading System Live - futures io
futures io futures trading



Taking a Trading System Live


Discussion in Trading Journals

Updated by tturner86
      Top Posters
    1. looks_one kevinkdog with 260 posts (714 thanks)
    2. looks_two Big Mike with 61 posts (97 thanks)
    3. looks_3 deaddog with 25 posts (24 thanks)
    4. looks_4 swz168 with 20 posts (40 thanks)
      Best Posters
    1. looks_one kevinkdog with 2.7 thanks per post
    2. looks_two swz168 with 2.0 thanks per post
    3. looks_3 rk142 with 2.0 thanks per post
    4. looks_4 Big Mike with 1.6 thanks per post
    1. trending_up 60,442 views
    2. thumb_up 1,128 thanks given
    3. group 97 followers
    1. forum 529 replies
    2. attach_file 189 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Thread Tools Search this Thread
 

Taking a Trading System Live

  #301 (permalink)
Cincinnati Ohio
 
Experience: Intermediate
Platform: NinjaTrader
Broker: Futures Broker
Trading: FX, Stocks, Options
 
Silver Dragon's Avatar
 
Posts: 1,783 since Feb 2011
Thanks: 5,047 given, 4,100 received


kevinkdog View Post
I am curious what people out there think. If this was your trading system, would you still be trading it?

Please answer a 1 question survey here: https://www.surveymonkey.com/s/XS6X7DN
(If this is against forum rules, I apologize and ask Big mike to delete the post).


Here are some performance curves to help you make a decision. For the sake of the survey, assume you started trading with real money on the green portion of the curve.



Attachment 126958


Attachment 126959

@kevinkdog

Sorry if some of these thoughts and observations are repeats... I have not read your entire thread but thought it was important to give some insight from a data perspective.

First and foremost; There is no way you should quit or even think about quitting unless you have data to back it up your decision. You posted some guidelines for quitting here:
Have these changed? Have you hit one of these conditions?

From a data perspective you need more data points before you start asking if it is working or not. A data set needs a minimum of 30 data points to be statically significant. For trading you need more and from what I can see you are not there yet. Think of your entire equity curve as your arm. The live testing is about size of your fingernail. Your not comparing apples to apples . Give it time to develop and dont let your emotions get involved when determining if it is a good system or bad system. Emotions cloud your judgement and cause irrational decisions. If the system is truly bad, the data will bare it out in the in the end.

If you gave this system to me to manage here is the one question I would ask you:

What is max draw down for the system. In other words- What dollar amount are you willing to risk to properly test the system? Think of it as a longer term investment.. You have looked at the prospectus now you have to determine how much you want to risk and for how long you want to invest. I believe you have defined this in the post above as 10%

After that I would just let it run and DONT MESS WITH IT!!! Doing so nullifies your testing and any future comparisons... IMO you need to have at least 6 months to year worth of data for a proper comparison. This will capture different market conditions and news cycles. After this time is up then I would compare it to the past. (This assumes the max draw down has not been hit.)

My other suggestion would be to run the same strategy in SIM alongside live account. Maybe you are already doing this?? This will give you invaluable information. If there is a major difference in the day to day trades between the live and sim then I would stop immediately to find out whats going on. If it is taking similar trades, taking into account slippage and market conditions then you know it is working as designed. Finally I would run another back test after the 6 months is up to compare.

At this point you will have 3 data sets. In theory the two SIM back tests should be nearly, if not identical. The live set should be within acceptable levels of the SIM sets taking into account slippage and unforeseen market conditions which cause bad entries etc. If you do not want to wait the full six months to run the 3rd data set then run it monthly. Again, this will tell you if there is a problem in the code or the data. If it is significantly off then run a second month to see if the results are the similar. Stop the system if it is. Otherwise let it continue to play out.

The thing to remember is proper analysis takes time. Its easy to back test because of the 5 minute instant gratification factor. It is harder to stay the course and let the system play out for a year for better or for worse.

At this point I would not worry about $$ performance. Thats not important from an analysis standpoint. All that is going to do is cloud your judgement. Instead focus on the nuts and bolts of your operation. Make sure everything is running as expected. Do a weekly and monthly comparison of trades between SIM and Live and define the differences in a journal. In fact if you have time do it daily. Define the market conditions and what caused the difference. Once you get done with all that then you will have a detailed document of what went right and what went wrong. Then you can make a determination on if it was successful or not.

Finally, something to put things in perspective; Its possible to do everything right and still lose the game. So, you can have a losing system $$ wise but it still be a successful system from a execution stand point. In this case the detailed notes of the SIM vs Live executions will tell you where the adjustment needs to be made. Fact is systems have been known to fail $$ wise on as little as 1 tick slippage per trade.

BTW- I have had the same thing happen to me when I went live with my automated system; Results varied from the back test. I determined the issue had to do with the fills and the exotic bar type I was using. So your not alone.

Robert

nosce te ipsum

You make your own opportunities in life.
Visit my futures io Trade Journal Reply With Quote
The following 6 users say Thank You to Silver Dragon for this post:

Can you help answer these questions
from other members on futures io?
Convert NinjaTrader NT7 indicator to Tradestation EasyLanguage and RADARSCREEN
Platforms and Indicators
Issue meeting order entry conditions
Elite Automated NinjaTrader Trading
Last 5 minutes Label
ThinkOrSwim
Moving average with Lag reduction
NinjaTrader
Linux?
ThinkOrSwim
 
 
  #302 (permalink)
detroit, mi
 
Experience: Beginner
Platform: TradeStation
Trading: SPY
 
Posts: 66 since Nov 2011
Thanks: 1,399 given, 71 received

Accelerated Testing?

kevin,

thanks for the thread.

just looking at your hypothetical curve diagrams and assuming diagrams had significance then you are nowhere near quitting point.

1) would you consider the curve as representation (or sum total) of various statistics?
2) what about the curve during the backtesting and forward testing portion made you decided to use the system?
3) what about the curve during the backtesting and forward testing portion would make you decide to not pursue the system?

since you used period of about 4 years
5) you would have to wait perhaps at least that long to decide whether system is worth it?
6) or if you have drawdown much longer in time duration (1.5x or 2x) and/or in much greater dollar loss (1.5x or 2x)

in my dayjob we do accelerated testing to validate the life of the product. the test has duration, predefined stresses and reliability with confidence attribute in the event sample test parts successfully pass the test.

if the backtest period ( 4 years in this case) is the life of the product, do you have to wait for 4 years to find out whether your product survives the life or is there an accelerated way to stress your product in much shorter period of time ( this would not be a walkforward test and period)? i could further expand on the analogy. perhaps it could be a useful analogy to what you are trying to do or it is just misguided rambling....

regards,
tihfa

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to tihfa for this post:
 
  #303 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,421 given, 5,325 received



Silver Dragon View Post
@kevinkdog

Sorry if some of these thoughts and observations are repeats... I have not read your entire thread but thought it was important to give some insight from a data perspective.

First and foremost; There is no way you should quit or even think about quitting unless you have data to back it up your decision. You posted some guidelines for quitting here:
Have these changed? Have you hit one of these conditions?

From a data perspective you need more data points before you start asking if it is working or not. A data set needs a minimum of 30 data points to be statically significant. For trading you need more and from what I can see you are not there yet. Think of your entire equity curve as your arm. The live testing is about size of your fingernail. Your not comparing apples to apples . Give it time to develop and dont let your emotions get involved when determining if it is a good system or bad system. Emotions cloud your judgement and cause irrational decisions. If the system is truly bad, the data will bare it out in the in the end.

If you gave this system to me to manage here is the one question I would ask you:

What is max draw down for the system. In other words- What dollar amount are you willing to risk to properly test the system? Think of it as a longer term investment.. You have looked at the prospectus now you have to determine how much you want to risk and for how long you want to invest. I believe you have defined this in the post above as 10%

After that I would just let it run and DONT MESS WITH IT!!! Doing so nullifies your testing and any future comparisons... IMO you need to have at least 6 months to year worth of data for a proper comparison. This will capture different market conditions and news cycles. After this time is up then I would compare it to the past. (This assumes the max draw down has not been hit.)

My other suggestion would be to run the same strategy in SIM alongside live account. Maybe you are already doing this?? This will give you invaluable information. If there is a major difference in the day to day trades between the live and sim then I would stop immediately to find out whats going on. If it is taking similar trades, taking into account slippage and market conditions then you know it is working as designed. Finally I would run another back test after the 6 months is up to compare.

At this point you will have 3 data sets. In theory the two SIM back tests should be nearly, if not identical. The live set should be within acceptable levels of the SIM sets taking into account slippage and unforeseen market conditions which cause bad entries etc. If you do not want to wait the full six months to run the 3rd data set then run it monthly. Again, this will tell you if there is a problem in the code or the data. If it is significantly off then run a second month to see if the results are the similar. Stop the system if it is. Otherwise let it continue to play out.

The thing to remember is proper analysis takes time. Its easy to back test because of the 5 minute instant gratification factor. It is harder to stay the course and let the system play out for a year for better or for worse.

At this point I would not worry about $$ performance. Thats not important from an analysis standpoint. All that is going to do is cloud your judgement. Instead focus on the nuts and bolts of your operation. Make sure everything is running as expected. Do a weekly and monthly comparison of trades between SIM and Live and define the differences in a journal. In fact if you have time do it daily. Define the market conditions and what caused the difference. Once you get done with all that then you will have a detailed document of what went right and what went wrong. Then you can make a determination on if it was successful or not.

Finally, something to put things in perspective; Its possible to do everything right and still lose the game. So, you can have a losing system $$ wise but it still be a successful system from a execution stand point. In this case the detailed notes of the SIM vs Live executions will tell you where the adjustment needs to be made. Fact is systems have been known to fail $$ wise on as little as 1 tick slippage per trade.

BTW- I have had the same thing happen to me when I went live with my automated system; Results varied from the back test. I determined the issue had to do with the fills and the exotic bar type I was using. So your not alone.

Robert


Thanks for the comments. I agree with pretty much everything you said.


One important part you mention is comparing SIM (or strategy results) vs. live. This is critical, especially for exotic bar types or strategies where slippage is a factor (and initial estimates of it were low).


I appreciate your excellent reply, even if you are a Bengals fan. But, they are doing much better than my Browns this year!

Follow me on Twitter Reply With Quote
The following 2 users say Thank You to kevinkdog for this post:
 
  #304 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,421 given, 5,325 received


tihfa View Post
kevin,



1) would you consider the curve as representation (or sum total) of various statistics?

Not sure I understand this question. Can you explain?


tihfa View Post
2) what about the curve during the backtesting and forward testing portion made you decided to use the system?

3) what about the curve during the backtesting and forward testing portion would make you decide to not pursue the system?

I set out my goals and objectives before developing the system. Since it met those goals, I then considered it for live trading. For me, the primary number I look at is Return/DD ratio.

There are a lot of reasons why I might not trade the system, even if the number look good. Let's say, for example, that the numbers are great, but the equity has been down the past 2 years. That might be a deal killer.



tihfa View Post
since you used period of about 4 years
5) you would have to wait perhaps at least that long to decide whether system is worth it?
6) or if you have drawdown much longer in time duration (1.5x or 2x) and/or in much greater dollar loss (1.5x or 2x)

I would use the 1.5-2.0 x drawdown before I'd wait 4 years. Your point is good though: the longer the real time out of sample period, the more confidence you'd have in the system.


tihfa View Post
in my dayjob we do accelerated testing to validate the life of the product. the test has duration, predefined stresses and reliability with confidence attribute in the event sample test parts successfully pass the test.

if the backtest period ( 4 years in this case) is the life of the product, do you have to wait for 4 years to find out whether your product survives the life or is there an accelerated way to stress your product in much shorter period of time ( this would not be a walkforward test and period)? i could further expand on the analogy. perhaps it could be a useful analogy to what you are trying to do or it is just misguided rambling....

regards,
tihfa

I understand what you are getting at, but I'm not sure how you could do this. A couple of thoughts come to mind:

1. Run the system on simulated data - data that isn't real, but has same general characteristics as real data. I think Tushar Chande's book describes this method.

2. You could arbitrarily reduce all trades by a certain percentage. Knock 25% off wins, and add 25% to losses. Does the system still work? This might simulates future performance.


Thanks for the comments. You've put forth some good ideas to think about.

Follow me on Twitter Reply With Quote
The following user says Thank You to kevinkdog for this post:
 
  #305 (permalink)
detroit, mi
 
Experience: Beginner
Platform: TradeStation
Trading: SPY
 
Posts: 66 since Nov 2011
Thanks: 1,399 given, 71 received

Equity Curve Use vs Stats Use


kevinkdog View Post
Not sure I understand this question. Can you explain?

kevin,

you have yourself pointed to the curves as basis for opinion whether to continue trading or not. yet you have statistics calculations which are calculated independently of the shape or form of the curve?

could you visually judge/compare Return/DD ratio of live trading period to that of backtested/walkforward period. If the answer is no, then you should not have offered the curves as the basis of opinion, but rather stats only. Anyhow a minor point...

regards,
tihfa

Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to tihfa for this post:
 
  #306 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,421 given, 5,325 received


tihfa View Post
kevin,

you have yourself pointed to the curves as basis for opinion whether to continue trading or not. yet you have statistics calculations which are calculated independently of the shape or form of the curve?

could you visually judge/compare Return/DD ratio of live trading period to that of backtested/walkforward period. If the answer is no, then you should not have offered the curves as the basis of opinion, but rather stats only. Anyhow a minor point...

regards,
tihfa

Thanks for clarifying. I would not recommend looking at Return/DD for such a short period of time as I have so far in live testing. Maybe in a few years that will be appropriate. Hopefully I will still be trading it at that time!

Thanks for your comment!

Follow me on Twitter Reply With Quote
The following user says Thank You to kevinkdog for this post:
 
  #307 (permalink)
Site Administrator
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 48,761 since Jun 2009
Thanks: 31,472 given, 94,081 received


kevinkdog View Post
1. Run the system on simulated data - data that isn't real, but has same general characteristics as real data. I think Tushar Chande's book describes this method.

BTW, if you are interested in this, there is a data generation thread here that creates data based on samples you provide:



Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my Facebook Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to Big Mike for this post:
 
  #308 (permalink)
Nashville, TN
 
Experience: Intermediate
Platform: NinjaTrader
Broker: IB
Trading: NQ ES
 
Posts: 468 since Feb 2011
Thanks: 323 given, 543 received

I took the equity curve chart (most of it) and windowed it (cutting it in half and then cutting the pieces in half) until I reached a point where there were sections that significantly underperformed the "average performance" of the total equity curve. Place smallest oval possible around each section that contains the start, finish, and all equity curve action in between. Make copies of oval or ovals and place in such a manner that would be inherently pessimistic to the "live trade section". Exits outside the bottom side of the risk ovals would indicate problems.

Going forward, move the ovals to new start point as new data points come in at intervals the size of the windowed region the ovals originated from. Looks like a few weeks ago there should a new start point for the small oval if you keep my windowing points.

The large oval appears to be about $7,500 profit top to bottom. Since the oval won't allow for placement that takes the full range, probably 2/3rds of that would be enough to get the equity curve outside the range...your $5k drawdown power-off button seems right.

Can't say I know of a proof or theory that legitimizes this as a means to test an equity curve going forward. I'll let someone else do the math. Fun with Ovals....yay!!!

If nothing else, it shows that you have plenty of time to consider performance.


Reply With Quote
The following 5 users say Thank You to Luger for this post:
 
  #309 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,421 given, 5,325 received

Very interesting analysis. Thanks for taking the time to do it, and to share it. Looks like I need to avoid the ovals!

Follow me on Twitter Reply With Quote
 
  #310 (permalink)
Nashville, TN
 
Experience: Intermediate
Platform: NinjaTrader
Broker: IB
Trading: NQ ES
 
Posts: 468 since Feb 2011
Thanks: 323 given, 543 received


I should have given a bit more background about my analysis but it was late. Your survey about continuing or quitting the system got me thinking.

The statistics of how much data you need to say that your live sample is different from the in-sample period seemed to be the most important factor. That is what got me slicing up the data into windows to see how much time constituted a period of underperformance in the in-sample data.

Also I've known traders that have used ovals on price/time squared charts. The basic shape is interesting when you think about how positive and negative volatility works over time.

Sure the analysis is subjective, but it adds a visual component that is representative of underperforming periods in the in-sample data. In this case it shows that you need a good bit more time before trashing the system. It also frames the volatility of in-sample underperforming periods which should helpful in determining drawdown potential going forward.

You may laugh, but I am going to be adding this as a subjective component to my system tracking. I'll be doing it in excel instead of paint though...lol

Reply With Quote



futures io Trading Community Trading Journals > Taking a Trading System Live




Upcoming Webinars and Events
 

Free BloodHound Licenses to everyone!

June
 

Every journal equals ten meals for the hungry

June
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts