NexusFi: Find Your Edge


Home Menu

 





Taking a Trading System Live


Discussion in Trading Journals

Updated
      Top Posters
    1. looks_one kevinkdog with 260 posts (717 thanks)
    2. looks_two Big Mike with 61 posts (97 thanks)
    3. looks_3 deaddog with 25 posts (24 thanks)
    4. looks_4 swz168 with 20 posts (40 thanks)
      Best Posters
    1. looks_one kevinkdog with 2.8 thanks per post
    2. looks_two swz168 with 2 thanks per post
    3. looks_3 rk142 with 2 thanks per post
    4. looks_4 Big Mike with 1.6 thanks per post
    1. trending_up 96,352 views
    2. thumb_up 1,131 thanks given
    3. group 99 followers
    1. forum 529 posts
    2. attach_file 189 attachments




 
Search this Thread

Taking a Trading System Live

  #151 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,647 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


Big Mike View Post
Kevin,

One advantage I think NinjaTrader has over TradeStation is the custom fitness freedom.

Check this thread:



Tons and tons and tons of different fitness and can all be tweaked to exactly what is important to you, and you can create your own formulas if the metric you wan't doesn't already exist.

Mike


I need sleep too (I drove home in middle of night last night after watching my Michigan Wolverines beat Notre Dame - biggest crowd in NCAA football history), but I'll try to keep up with you.


Tradestation I know is trying to work on better fitness functions. Most of the standard ones are weak. I usually use Net Profit, for simplicity, and typically, the good net profit cases have low drawdowns.

One other big issue is that using drawdown as part of a fitness function with walkforward can give very misleading results, if the walkforward window moves with time. I will try to explain when I answer an earlier post on walkforward.

Follow me on Twitter Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
NT7 Indicator Script Troubleshooting - Camarilla Pivots
NinjaTrader
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
The space time continuum and the dynamics of a financial …
Emini and Emicro Index
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
Better Renko Gaps
The Elite Circle
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
56 thanks
Funded Trader platforms
44 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #152 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,647 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


Big Mike View Post
Kevin, out of curiosity have you watched @NJAMC's webinar "An Intro to Machine Learning"?



NJAMC has been working on his project for a while now, he and @Luger were also involved in the Artificial Bee Colony thread, the predecessor to Greg's ML thread:



I tried to help where I could using my Wave59 license, but in the end those guys were just too far ahead of me. That's what I get for being a high school drop out, it sucks to be a trader that is poor at math (if only I had known...)

I bring it up because I was thinking of maybe getting together a consortium of sorts, like maybe John Ehlers, Suri Duddella, Manesh Patel, and @NJAMC, @Luger, @Fat Tails and some other fine folks who I can't think of right now, and see if maybe some sort of collaborative project could be started to further the work that Greg (NJAMC) is doing on the ML front. I figured maybe a few 1-hour conference calls or skype sessions might be enough to get some really great brainstorming done to make some real headway in this area.

Perhaps we should take this off-line... I've cluttered your thread enough.

Mike


That's OK, this is good stuff. I have the webinar on my to do list.

Follow me on Twitter Started this thread Reply With Quote
  #153 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,177
Thanks Received: 101,541



kevinkdog View Post
Without originally intending to, I think I did just tear your system apart - not by analyzing your results, but by asking you how you got those results. Based on your answers (no walkforward, no out of sample, optimized over all data set), I would have no need to look further at your results. They might be spectacular results, but optimized results almost always are. Doesn;t mean jack going forward.

Where it would get tricky is if you lied and said it wasn't optimized, or that it had x% out of sample results at end. How would I be able to tell you were lying? One way I know is the "if it looks too good to be true, it probably is" test. Another way would be watching it in real time (incubation). Maybe someone else has a better approach they would like to share.


I think this is why it is so hard for people who buy strategies or systems. How do they know if the system results are "real," or if they are just some optimized BS? Most times, a vendor who sells optimized systems would probably lie to you about how he got the results anyway. I know, because sometimes for fun I converse with vendors of these super duper systems, and 9 times out of 10 it is clear to me that
a) the results are garbage
b) the vendor doesn't know what he is talking about
c) the vendor's personality is some kind of combination of unethical, immoral, irresponsible and feeble minded.


I propose you keep tracking the system, and let's see how it does a few months from now. That should be very revealing...

You make good points and I agree completely.

I see the same stuff vendors are selling, and the worse thing is -- people are buying it. The other real problem with NinjaTrader's summary page is there is no position sizing anywhere. So you can make a system look great by just adding a few zero's to the position size, and it shows more trades, more profit. People then think it is not curve fitted because it has so many more trades.

The other big one is when people use limit orders. That's an easy way to make NT reports inaccurate.

I generally prefer systems that trade 1,000 times a year or more on 1 contract. If I start dipping below 500 trades a year, I get worried about over fitting. Obviously you take a very different approach, and this is just one of many ways we make a market together

I prefer systems that have very few input parameters, like 1 or 2 ideally, where everything else is dynamic and based on volatility and such in the market. That way I can optimize for my best score, and that golden looking equity curve we are all after, by combining that system along with many more into a portfolio and looking at how it benefits or balances the portfolio as a whole.

As for keeping track of the system, I wish I had fixed the code I was using to automatically post charts and trade results into a futures.io (formerly BMT) thread via the futures.io (formerly BMT) API. I just never had time to fix it, and certainly don't want to start now (I am preparing for vacation mode...). So I can't do any kind of live test. Closest I could do would be to shelf it and come back in two months and just run a new report to see how that out of sample data looks, but it would still be historical sim not live.

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #154 (permalink)
 
Big Mike's Avatar
 Big Mike 
Manta, Ecuador
Site Administrator
Developer
Swing Trader
 
Experience: Advanced
Platform: Custom solution
Broker: IBKR
Trading: Stocks & Futures
Frequency: Every few days
Duration: Weeks
Posts: 50,399 since Jun 2009
Thanks Given: 33,177
Thanks Received: 101,541


kevinkdog View Post
Tradestation I know is trying to work on better fitness functions.

What ever came of their acquisition of Grail Computer? I thought they would be killing it by now with all the work Grail was doing. Have they integrated it into a production product yet?

Mike

We're here to help: just ask the community or contact our Help Desk

Quick Links: Change your Username or Register as a Vendor
Searching for trading reviews? Review this list
Lifetime Elite Membership: Sign-up for only $149 USD
Exclusive money saving offers from our Site Sponsors: Browse Offers
Report problems with the site: Using the NexusFi changelog thread
Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
  #155 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,647 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


Big Mike View Post
What ever came of their acquisition of Grail Computer? I thought they would be killing it by now with all the work Grail was doing. Have they integrated it into a production product yet?

Mike

They incorporated at least some of Grail's capabilities into version 9.0. I think they are continuously more features.

Follow me on Twitter Started this thread Reply With Quote
  #156 (permalink)
 
NJAMC's Avatar
 NJAMC 
Atkinson, NH USA
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader 8/TensorFlow
Broker: NinjaTrader Brokerage
Trading: Futures, CL, ES, ZB
Posts: 1,970 since Dec 2010
Thanks Given: 3,037
Thanks Received: 2,394


Big Mike View Post
You make good points and I agree completely.

I see the same stuff vendors are selling, and the worse thing is -- people are buying it. The other real problem with NinjaTrader's summary page is there is no position sizing anywhere. So you can make a system look great by just adding a few zero's to the position size, and it shows more trades, more profit. People then think it is not curve fitted because it has so many more trades.

The other big one is when people use limit orders. That's an easy way to make NT reports inaccurate.

I generally prefer systems that trade 1,000 times a year or more on 1 contract. If I start dipping below 500 trades a year, I get worried about over fitting. Obviously you take a very different approach, and this is just one of many ways we make a market together

Mike

@Big Mike,

Unfortunately, it is easy to over-fit even a large data-set. It is a matter of having enough degrees of freedom in your "solution" or what I am less familiar with, but have seen none the less is a simple MA crossing that gets lucky. (Likely large drawdown ,MAE, or ETDs)

From the "black magic" side, all you really need to do is give an algorithm enough time and degrees of freedom (input values, products/sums of terms, square's, square roots, etc.) and you can match to the max of degrees (ultimately more good trades) of freedom. Sometime it is difficult to see all the degrees of freedom, but there is likely more than you can see.

This is why Forward or Out of Sample testing is critical. The more "over-fit" a function is, the fast it will fall apart as you test values that were not used to train the system. What you are looking for is a "fit" function. One that does okay, but generalizes the solution into the future. This is the goal of my Genetic Programming investigation.

Nil per os
-NJAMC [Generic Programmer]

LOM WIKI: NT-Local-Order-Manager-LOM-Guide
Artificial Bee Colony Optimization
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #157 (permalink)
 
NJAMC's Avatar
 NJAMC 
Atkinson, NH USA
Market Wizard
 
Experience: Intermediate
Platform: NinjaTrader 8/TensorFlow
Broker: NinjaTrader Brokerage
Trading: Futures, CL, ES, ZB
Posts: 1,970 since Dec 2010
Thanks Given: 3,037
Thanks Received: 2,394


Big Mike View Post
I looked at his Adap Trade stuff. What turned me off is it's for TradeStation (EasyLanguage) only, and given my falling out with MultiCharts I don't want to go back down that path.

I also could not tell what type of machine learning he was really using. Maybe @NJAMC could view his youtube videos and make a guess. My best guess was it is not the type of ML that @NJAMC is working on, or that Wave59 has, but instead just a Genetic Optimization type formula.

I was also a bit turned off by the "brute force" method of the app, basically trying all kinds of crazy indicators to find a good fit. This clearly is a concern for overfitting....

Anyway, his MSA software is another matter, and I think is quite good for analysis.

Mike

@Big Mike,

I do believe the Adaptrade Software is using Genetic Programming based upon configuration and function. I certainly cannot attest to the ability it has to solve the problem at hand.

As discussed above, it is very easy to "over-fit" a solution using any Machine Learning approach or (manually this is trickier but possible). I can tell you that I have many systems that will over-fit a solution to the training set. This is quite easy. The difficult part is to figure out how to get a ML system to create a Generic solution without access to the Out of Sample data.

This is a bit of a paradox for me right now. I want to create a solution that fits my training set (in sample data), but the second requirement is that it needs to also fit data I have not seen yet (Out of sample data). I can test the OOS data, but then it is touched and now should be considered "In Sample" data. It is unclear in systems like the Adaptrade system when it shows the OOS data on its charts, has it reviewed this data. It should be okay to calculate the answer (Profit or EC), but that data cannot be brought back into the Engine to affect the genetics of the system without it now being considered In Sample data again.

My current approach is to simply store anything that looks profitable. I then use a 2nd analysis which is manual for me to accept the strategy as generic. What I do is run all the "looks profitable" through NT (I use the optimizer, but only to load the different strategies for analysis) and check for nice EC curves and profitability OOS.

In the future, I may add more logic to truly analyse the "looks profitable" before storing. What may not be overly obvious is the longer you run a ML algorithm on a training set, the more likely it will specialize (over-fit) the training dataset. So the "looks profitable" strategies need to be pealed off as soon as they start to look good before over-fitting starts.

Nil per os
-NJAMC [Generic Programmer]

LOM WIKI: NT-Local-Order-Manager-LOM-Guide
Artificial Bee Colony Optimization
Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #158 (permalink)
 
treydog999's Avatar
 treydog999 
seoul, Korea
 
Experience: Intermediate
Platform: Multicharts
Broker: CQG, DTN IQfeed
Trading: YM 6E
Posts: 897 since Jul 2012
Thanks Given: 291
Thanks Received: 1,039


NJAMC View Post
@Big Mike,

I do believe the Adaptrade Software is using Genetic Programming based upon configuration and function. I certainly cannot attest to the ability it has to solve the problem at hand.

As discussed above, it is very easy to "over-fit" a solution using any Machine Learning approach or (manually this is trickier but possible). I can tell you that I have many systems that will over-fit a solution to the training set. This is quite easy. The difficult part is to figure out how to get a ML system to create a Generic solution without access to the Out of Sample data.

This is a bit of a paradox for me right now. I want to create a solution that fits my training set (in sample data), but the second requirement is that it needs to also fit data I have not seen yet (Out of sample data). I can test the OOS data, but then it is touched and now should be considered "In Sample" data. It is unclear in systems like the Adaptrade system when it shows the OOS data on its charts, has it reviewed this data. It should be okay to calculate the answer (Profit or EC), but that data cannot be brought back into the Engine to affect the genetics of the system without it now being considered In Sample data again.

My current approach is to simply store anything that looks profitable. I then use a 2nd analysis which is manual for me to accept the strategy as generic. What I do is run all the "looks profitable" through NT (I use the optimizer, but only to load the different strategies for analysis) and check for nice EC curves and profitability OOS.

In the future, I may add more logic to truly analyse the "looks profitable" before storing. What may not be overly obvious is the longer you run a ML algorithm on a training set, the more likely it will specialize (over-fit) the training dataset. So the "looks profitable" strategies need to be pealed off as soon as they start to look good before over-fitting starts.

I have played with the Adaptrade software for a while. It was interesting, but not my cup of tea. In regards to the contamination of data set of OOS since it plots that in the chart. There is a thread in the googlegroup for them. As far as I understand it, it remains untouched as the results are not recycled. Basically all results that are used for the GP/GA optimization and next generation are based on the in sample. Then that goes to seed the next generation. The OOS results are shown but are not used to generate the next generation. But as a user you can see whats going on. You can go check the google group to be sure or ask questions that are answered by the programmer/founder of adaptrade himself.

Reply With Quote
Thanked by:
  #159 (permalink)
 
liquidcci's Avatar
 liquidcci 
Austin, TX
 
Experience: Master
Platform: ninjatrader, r-trader
Trading: NQ, CL
Posts: 866 since Jun 2011
Thanks Given: 610
Thanks Received: 1,091


kevinkdog View Post
I have never used the product above, but Mike Bryant has some other good tools, for free (through his newsletters) and for pay. He has some very good ideas on trading system development.

@kevinkdog I find Adaptrades MSA is an excellent piece of software. If you have a solid system on 1 contract you can plug your test results in and really see how position size using different methods affects your outcome. Very powerful tool. Also has a nice portfolio feature and some other features that are good for analyzing a system. There a few things I wish it would do better but overall it is some of the best money I have ever spent. It is really inexpensive for what it does.

"The day I became a winning trader was the day it became boring. Daily losses no longer bother me and daily wins no longer excited me. Took years of pain and busting a few accounts before finally got my mind right. I survived the darkness within and now just chillax and let my black box do the work."
Reply With Quote
Thanked by:
  #160 (permalink)
 kevinkdog   is a Vendor
 
Posts: 3,647 since Jul 2012
Thanks Given: 1,890
Thanks Received: 7,338


In an earlier post, I shared with you one way I track a trading strategy. Here is another tracking tool I use...

When I used to work in aerospace (or the "real world" as I sometimes refer to it), our small company ($250 million annual sales) would have a weekly meeting called "How We Doin." Incorrect grammar aside, it was an excellent way for the managers of the company to quickly see how sales were for the month and quarter, what quality and production problems were occurring, and just a general sense of where the company currently stood.

Now, fast forward a few years. I am trading full time, working alone. But, I still want to see at a glance "how I'm doin" with my strategies and trading. Obviously, my account statements and equity curve tell the overall story, but that is not enough detail for me. What strategies are doing good? Which are underperforming? Of strategies I am incubating, how do they look? Should I make some changes in what I am trading? This "How I'm Doin" report can help me answer all of these questions.

I developed a little spreadsheet (sorry, I am not sharing it, but it is easy enough to do yourself) to help me with this task. It tells me at a glance how my strategies are doing, and I can easily drill down and see detail if I need to.

First, there is a summary page. I include every strategy I am trading live. I also include in another section the strategies I currently am incubating. This summary sheet collects all the data I am interested in (of course, if you did this yourself, you'd likely pick different metrics than I did). This summary sheet gets the data from the individual sheets, which I will describe a bit later.

To keep things simple, I base everything on one contract being traded, even though that is usually not what I am actually trading. Why? My goal with this spreadsheet is to see how my strategies are doing compared to how I thought (calculated) they'd be doing. If I included position sizing, it would muddy up the view for me.

Of all the numbers on this sheet, I am primarily interested in two columns:

1. Return Efficiency - How am I doing, compared to my expectations? That is how I define return efficiency, and it is simply my actual return divided by my expected return. If my strategy is performing exactly as I had calculated, it will be 100%. Obviously, I want this to be close to or above 100%. Typically, when I take all the strategies together, I find my efficiency is somewhere between 70-100%. So, this says that if my historical testing says I should make $10 a year, I am actually making somewhere between $7-10.

2. Drawdown Efficiency - This is how I am doing with regards to drawdown. Just like with return efficiency, I calculate this as my actual drawdown divided by my expected drawdown. I then subtract the result from 1, to make the number 100% the ideal value. It is a bit backwards to do this, but I do it that way so that both efficiency numbers have 100% as their ideal value. Then, the closer to efficiencies get to zero, the worse off things are.


Once a month, I go through and update each of the individual system sheets with performance data, and that automatically updates the main sheet.

In the next post I'll show an individual sheet.


Follow me on Twitter Started this thread Reply With Quote




Last Updated on January 6, 2016


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts