Taking a Trading System Live - futures io
futures io futures trading



Taking a Trading System Live


Discussion in Trading Journals

Updated by tturner86
      Top Posters
    1. looks_one kevinkdog with 260 posts (714 thanks)
    2. looks_two Big Mike with 61 posts (97 thanks)
    3. looks_3 deaddog with 25 posts (24 thanks)
    4. looks_4 swz168 with 20 posts (40 thanks)
      Best Posters
    1. looks_one kevinkdog with 2.7 thanks per post
    2. looks_two swz168 with 2.0 thanks per post
    3. looks_3 rk142 with 2.0 thanks per post
    4. looks_4 Big Mike with 1.6 thanks per post
    1. trending_up 60,479 views
    2. thumb_up 1,128 thanks given
    3. group 97 followers
    1. forum 529 replies
    2. attach_file 189 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 100,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Thread Tools Search this Thread
 

Taking a Trading System Live

  #151 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,422 given, 5,326 received


Big Mike View Post
Kevin,

One advantage I think NinjaTrader has over TradeStation is the custom fitness freedom.

Check this thread:



Tons and tons and tons of different fitness and can all be tweaked to exactly what is important to you, and you can create your own formulas if the metric you wan't doesn't already exist.

Mike


I need sleep too (I drove home in middle of night last night after watching my Michigan Wolverines beat Notre Dame - biggest crowd in NCAA football history), but I'll try to keep up with you.


Tradestation I know is trying to work on better fitness functions. Most of the standard ones are weak. I usually use Net Profit, for simplicity, and typically, the good net profit cases have low drawdowns.

One other big issue is that using drawdown as part of a fitness function with walkforward can give very misleading results, if the walkforward window moves with time. I will try to explain when I answer an earlier post on walkforward.

Follow me on Twitter Reply With Quote

Can you help answer these questions
from other members on futures io?
Convert NinjaTrader NT7 indicator to Tradestation EasyLanguage and RADARSCREEN
Platforms and Indicators
Issue meeting order entry conditions
Elite Automated NinjaTrader Trading
Last 5 minutes Label
ThinkOrSwim
Moving average with Lag reduction
NinjaTrader
Linux?
ThinkOrSwim
 
Best Threads (Most Thanked)
in the last 7 days on futures io
Spoo-nalysis ES e-mini futures S&P 500
72 thanks
June Charity Fundraiser - Feeding hungry families
48 thanks
Fixed vs trailing stops
31 thanks
FIO Journal Challenge - June 2020 edition w/SharkIndicators
30 thanks
AMP Futures / AMP Global Review
26 thanks
 
  #152 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,422 given, 5,326 received


Big Mike View Post
Kevin, out of curiosity have you watched @NJAMC's webinar "An Intro to Machine Learning"?



NJAMC has been working on his project for a while now, he and @Luger were also involved in the Artificial Bee Colony thread, the predecessor to Greg's ML thread:



I tried to help where I could using my Wave59 license, but in the end those guys were just too far ahead of me. That's what I get for being a high school drop out, it sucks to be a trader that is poor at math (if only I had known...)

I bring it up because I was thinking of maybe getting together a consortium of sorts, like maybe John Ehlers, Suri Duddella, Manesh Patel, and @NJAMC, @Luger, @Fat Tails and some other fine folks who I can't think of right now, and see if maybe some sort of collaborative project could be started to further the work that Greg (NJAMC) is doing on the ML front. I figured maybe a few 1-hour conference calls or skype sessions might be enough to get some really great brainstorming done to make some real headway in this area.

Perhaps we should take this off-line... I've cluttered your thread enough.

Mike


That's OK, this is good stuff. I have the webinar on my to do list.

Follow me on Twitter Reply With Quote
 
  #153 (permalink)
Site Administrator
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 48,775 since Jun 2009
Thanks: 31,485 given, 94,162 received



kevinkdog View Post
Without originally intending to, I think I did just tear your system apart - not by analyzing your results, but by asking you how you got those results. Based on your answers (no walkforward, no out of sample, optimized over all data set), I would have no need to look further at your results. They might be spectacular results, but optimized results almost always are. Doesn;t mean jack going forward.

Where it would get tricky is if you lied and said it wasn't optimized, or that it had x% out of sample results at end. How would I be able to tell you were lying? One way I know is the "if it looks too good to be true, it probably is" test. Another way would be watching it in real time (incubation). Maybe someone else has a better approach they would like to share.


I think this is why it is so hard for people who buy strategies or systems. How do they know if the system results are "real," or if they are just some optimized BS? Most times, a vendor who sells optimized systems would probably lie to you about how he got the results anyway. I know, because sometimes for fun I converse with vendors of these super duper systems, and 9 times out of 10 it is clear to me that
a) the results are garbage
b) the vendor doesn't know what he is talking about
c) the vendor's personality is some kind of combination of unethical, immoral, irresponsible and feeble minded.


I propose you keep tracking the system, and let's see how it does a few months from now. That should be very revealing...

You make good points and I agree completely.

I see the same stuff vendors are selling, and the worse thing is -- people are buying it. The other real problem with NinjaTrader's summary page is there is no position sizing anywhere. So you can make a system look great by just adding a few zero's to the position size, and it shows more trades, more profit. People then think it is not curve fitted because it has so many more trades.

The other big one is when people use limit orders. That's an easy way to make NT reports inaccurate.

I generally prefer systems that trade 1,000 times a year or more on 1 contract. If I start dipping below 500 trades a year, I get worried about over fitting. Obviously you take a very different approach, and this is just one of many ways we make a market together

I prefer systems that have very few input parameters, like 1 or 2 ideally, where everything else is dynamic and based on volatility and such in the market. That way I can optimize for my best score, and that golden looking equity curve we are all after, by combining that system along with many more into a portfolio and looking at how it benefits or balances the portfolio as a whole.

As for keeping track of the system, I wish I had fixed the code I was using to automatically post charts and trade results into a futures.io (formerly BMT) thread via the futures.io (formerly BMT) API. I just never had time to fix it, and certainly don't want to start now (I am preparing for vacation mode...). So I can't do any kind of live test. Closest I could do would be to shelf it and come back in two months and just run a new report to see how that out of sample data looks, but it would still be historical sim not live.

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my Facebook Visit my futures io Trade Journal Reply With Quote
 
  #154 (permalink)
Site Administrator
Manta, Ecuador
 
Experience: Advanced
Platform: My own custom solution
Trading: Emini Futures
 
Big Mike's Avatar
 
Posts: 48,775 since Jun 2009
Thanks: 31,485 given, 94,162 received


kevinkdog View Post
Tradestation I know is trying to work on better fitness functions.

What ever came of their acquisition of Grail Computer? I thought they would be killing it by now with all the work Grail was doing. Have they integrated it into a production product yet?

Mike

We're here to help -- just ask

For the best trading education, watch our webinars
Searching for trading reviews? Review this list

Follow us on Twitter, YouTube, and Facebook

Support our community as an Elite Member:
https://futures.io/elite/
Follow me on Twitter Visit my Facebook Visit my futures io Trade Journal Reply With Quote
 
  #155 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,422 given, 5,326 received


Big Mike View Post
What ever came of their acquisition of Grail Computer? I thought they would be killing it by now with all the work Grail was doing. Have they integrated it into a production product yet?

Mike

They incorporated at least some of Grail's capabilities into version 9.0. I think they are continuously more features.

Follow me on Twitter Reply With Quote
 
  #156 (permalink)
Atkinson, NH USA
 
Experience: Intermediate
Platform: NinjaTrader 8/TensorFlow
Broker: NinjaTrader Brokerage
Trading: Futures, CL, ES, ZB
 
NJAMC's Avatar
 
Posts: 1,963 since Dec 2010
Thanks: 3,029 given, 2,375 received


Big Mike View Post
You make good points and I agree completely.

I see the same stuff vendors are selling, and the worse thing is -- people are buying it. The other real problem with NinjaTrader's summary page is there is no position sizing anywhere. So you can make a system look great by just adding a few zero's to the position size, and it shows more trades, more profit. People then think it is not curve fitted because it has so many more trades.

The other big one is when people use limit orders. That's an easy way to make NT reports inaccurate.

I generally prefer systems that trade 1,000 times a year or more on 1 contract. If I start dipping below 500 trades a year, I get worried about over fitting. Obviously you take a very different approach, and this is just one of many ways we make a market together

Mike

@Big Mike,

Unfortunately, it is easy to over-fit even a large data-set. It is a matter of having enough degrees of freedom in your "solution" or what I am less familiar with, but have seen none the less is a simple MA crossing that gets lucky. (Likely large drawdown ,MAE, or ETDs)

From the "black magic" side, all you really need to do is give an algorithm enough time and degrees of freedom (input values, products/sums of terms, square's, square roots, etc.) and you can match to the max of degrees (ultimately more good trades) of freedom. Sometime it is difficult to see all the degrees of freedom, but there is likely more than you can see.

This is why Forward or Out of Sample testing is critical. The more "over-fit" a function is, the fast it will fall apart as you test values that were not used to train the system. What you are looking for is a "fit" function. One that does okay, but generalizes the solution into the future. This is the goal of my Genetic Programming investigation.

Nil per os
-NJAMC [Generic Programmer]

LOM WIKI: NT-Local-Order-Manager-LOM-Guide
Artificial Bee Colony Optimization
Visit my futures io Trade Journal Reply With Quote
The following 3 users say Thank You to NJAMC for this post:
 
  #157 (permalink)
Atkinson, NH USA
 
Experience: Intermediate
Platform: NinjaTrader 8/TensorFlow
Broker: NinjaTrader Brokerage
Trading: Futures, CL, ES, ZB
 
NJAMC's Avatar
 
Posts: 1,963 since Dec 2010
Thanks: 3,029 given, 2,375 received


Big Mike View Post
I looked at his Adap Trade stuff. What turned me off is it's for TradeStation (EasyLanguage) only, and given my falling out with MultiCharts I don't want to go back down that path.

I also could not tell what type of machine learning he was really using. Maybe @NJAMC could view his youtube videos and make a guess. My best guess was it is not the type of ML that @NJAMC is working on, or that Wave59 has, but instead just a Genetic Optimization type formula.

I was also a bit turned off by the "brute force" method of the app, basically trying all kinds of crazy indicators to find a good fit. This clearly is a concern for overfitting....

Anyway, his MSA software is another matter, and I think is quite good for analysis.

Mike

@Big Mike,

I do believe the Adaptrade Software is using Genetic Programming based upon configuration and function. I certainly cannot attest to the ability it has to solve the problem at hand.

As discussed above, it is very easy to "over-fit" a solution using any Machine Learning approach or (manually this is trickier but possible). I can tell you that I have many systems that will over-fit a solution to the training set. This is quite easy. The difficult part is to figure out how to get a ML system to create a Generic solution without access to the Out of Sample data.

This is a bit of a paradox for me right now. I want to create a solution that fits my training set (in sample data), but the second requirement is that it needs to also fit data I have not seen yet (Out of sample data). I can test the OOS data, but then it is touched and now should be considered "In Sample" data. It is unclear in systems like the Adaptrade system when it shows the OOS data on its charts, has it reviewed this data. It should be okay to calculate the answer (Profit or EC), but that data cannot be brought back into the Engine to affect the genetics of the system without it now being considered In Sample data again.

My current approach is to simply store anything that looks profitable. I then use a 2nd analysis which is manual for me to accept the strategy as generic. What I do is run all the "looks profitable" through NT (I use the optimizer, but only to load the different strategies for analysis) and check for nice EC curves and profitability OOS.

In the future, I may add more logic to truly analyse the "looks profitable" before storing. What may not be overly obvious is the longer you run a ML algorithm on a training set, the more likely it will specialize (over-fit) the training dataset. So the "looks profitable" strategies need to be pealed off as soon as they start to look good before over-fitting starts.

Nil per os
-NJAMC [Generic Programmer]

LOM WIKI: NT-Local-Order-Manager-LOM-Guide
Artificial Bee Colony Optimization
Visit my futures io Trade Journal Reply With Quote
The following 2 users say Thank You to NJAMC for this post:
 
  #158 (permalink)
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
 
treydog999's Avatar
 
Posts: 896 since Jul 2012
Thanks: 291 given, 1,023 received


NJAMC View Post
@Big Mike,

I do believe the Adaptrade Software is using Genetic Programming based upon configuration and function. I certainly cannot attest to the ability it has to solve the problem at hand.

As discussed above, it is very easy to "over-fit" a solution using any Machine Learning approach or (manually this is trickier but possible). I can tell you that I have many systems that will over-fit a solution to the training set. This is quite easy. The difficult part is to figure out how to get a ML system to create a Generic solution without access to the Out of Sample data.

This is a bit of a paradox for me right now. I want to create a solution that fits my training set (in sample data), but the second requirement is that it needs to also fit data I have not seen yet (Out of sample data). I can test the OOS data, but then it is touched and now should be considered "In Sample" data. It is unclear in systems like the Adaptrade system when it shows the OOS data on its charts, has it reviewed this data. It should be okay to calculate the answer (Profit or EC), but that data cannot be brought back into the Engine to affect the genetics of the system without it now being considered In Sample data again.

My current approach is to simply store anything that looks profitable. I then use a 2nd analysis which is manual for me to accept the strategy as generic. What I do is run all the "looks profitable" through NT (I use the optimizer, but only to load the different strategies for analysis) and check for nice EC curves and profitability OOS.

In the future, I may add more logic to truly analyse the "looks profitable" before storing. What may not be overly obvious is the longer you run a ML algorithm on a training set, the more likely it will specialize (over-fit) the training dataset. So the "looks profitable" strategies need to be pealed off as soon as they start to look good before over-fitting starts.

I have played with the Adaptrade software for a while. It was interesting, but not my cup of tea. In regards to the contamination of data set of OOS since it plots that in the chart. There is a thread in the googlegroup for them. As far as I understand it, it remains untouched as the results are not recycled. Basically all results that are used for the GP/GA optimization and next generation are based on the in sample. Then that goes to seed the next generation. The OOS results are shown but are not used to generate the next generation. But as a user you can see whats going on. You can go check the google group to be sure or ask questions that are answered by the programmer/founder of adaptrade himself.

Reply With Quote
The following 2 users say Thank You to treydog999 for this post:
 
  #159 (permalink)
Austin, TX
 
Experience: Master
Platform: ninjatrader, r-trader
Trading: CL, NG, TF, NQ, YM, GC, ES
 
liquidcci's Avatar
 
Posts: 862 since Jun 2011
Thanks: 609 given, 1,064 received


kevinkdog View Post
I have never used the product above, but Mike Bryant has some other good tools, for free (through his newsletters) and for pay. He has some very good ideas on trading system development.

@kevinkdog I find Adaptrades MSA is an excellent piece of software. If you have a solid system on 1 contract you can plug your test results in and really see how position size using different methods affects your outcome. Very powerful tool. Also has a nice portfolio feature and some other features that are good for analyzing a system. There a few things I wish it would do better but overall it is some of the best money I have ever spent. It is really inexpensive for what it does.

"The day I became a winning trader was the day it became boring. Daily losses no longer bother me and daily wins no longer excited me. Took years of pain and busting a few accounts before finally got my mind right. I survived the darkness within and now just chillax and let my black box do the work."

Last edited by liquidcci; September 9th, 2013 at 10:36 AM.
Reply With Quote
The following user says Thank You to liquidcci for this post:
 
  #160 (permalink)
Legendary Market Wizard
Cleveland Ohio/United States
 
Experience: Advanced
Platform: Tradestation
Broker: Tradestation, DeCarley, others
Trading: futures
 
Posts: 2,768 since Jul 2012
Thanks: 1,422 given, 5,326 received

Monitoring a Strategy


In an earlier post, I shared with you one way I track a trading strategy. Here is another tracking tool I use...

When I used to work in aerospace (or the "real world" as I sometimes refer to it), our small company ($250 million annual sales) would have a weekly meeting called "How We Doin." Incorrect grammar aside, it was an excellent way for the managers of the company to quickly see how sales were for the month and quarter, what quality and production problems were occurring, and just a general sense of where the company currently stood.

Now, fast forward a few years. I am trading full time, working alone. But, I still want to see at a glance "how I'm doin" with my strategies and trading. Obviously, my account statements and equity curve tell the overall story, but that is not enough detail for me. What strategies are doing good? Which are underperforming? Of strategies I am incubating, how do they look? Should I make some changes in what I am trading? This "How I'm Doin" report can help me answer all of these questions.

I developed a little spreadsheet (sorry, I am not sharing it, but it is easy enough to do yourself) to help me with this task. It tells me at a glance how my strategies are doing, and I can easily drill down and see detail if I need to.

First, there is a summary page. I include every strategy I am trading live. I also include in another section the strategies I currently am incubating. This summary sheet collects all the data I am interested in (of course, if you did this yourself, you'd likely pick different metrics than I did). This summary sheet gets the data from the individual sheets, which I will describe a bit later.

To keep things simple, I base everything on one contract being traded, even though that is usually not what I am actually trading. Why? My goal with this spreadsheet is to see how my strategies are doing compared to how I thought (calculated) they'd be doing. If I included position sizing, it would muddy up the view for me.

Of all the numbers on this sheet, I am primarily interested in two columns:

1. Return Efficiency - How am I doing, compared to my expectations? That is how I define return efficiency, and it is simply my actual return divided by my expected return. If my strategy is performing exactly as I had calculated, it will be 100%. Obviously, I want this to be close to or above 100%. Typically, when I take all the strategies together, I find my efficiency is somewhere between 70-100%. So, this says that if my historical testing says I should make $10 a year, I am actually making somewhere between $7-10.

2. Drawdown Efficiency - This is how I am doing with regards to drawdown. Just like with return efficiency, I calculate this as my actual drawdown divided by my expected drawdown. I then subtract the result from 1, to make the number 100% the ideal value. It is a bit backwards to do this, but I do it that way so that both efficiency numbers have 100% as their ideal value. Then, the closer to efficiencies get to zero, the worse off things are.


Once a month, I go through and update each of the individual system sheets with performance data, and that automatically updates the main sheet.

In the next post I'll show an individual sheet.


Follow me on Twitter Reply With Quote
The following 8 users say Thank You to kevinkdog for this post:



futures io Trading Community Trading Journals > Taking a Trading System Live




Upcoming Webinars and Events
 

Free BloodHound Licenses to everyone!

June
 

Every journal equals ten meals for the hungry

June
     



Copyright © 2020 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts