NexusFi: Find Your Edge


Home Menu

 





iSystems Journal


Discussion in Trading Journals

Updated
      Top Posters
    1. looks_one SMCJB with 94 posts (273 thanks)
    2. looks_two Mabi with 15 posts (15 thanks)
    3. looks_3 mattz with 7 posts (6 thanks)
    4. looks_4 tpredictor with 7 posts (11 thanks)
      Best Posters
    1. looks_one Big Mike with 14 thanks per post
    2. looks_two kevinkdog with 4.8 thanks per post
    3. looks_3 SMCJB with 2.9 thanks per post
    4. looks_4 Mabi with 1 thanks per post
    1. trending_up 40,708 views
    2. thumb_up 394 thanks given
    3. group 53 followers
    1. forum 179 posts
    2. attach_file 91 attachments




 
Search this Thread

iSystems Journal

  #61 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

Had some RL issues that have kept me from working on a lot of things including this. Probably won't be able to dedicate the time intend to for at least another week, but I will be back soon.

Still have a couple of systems going, glad to report that they are doing okay, and that not only am I still profitable, but I've almost eliminated my drawdown. Also interesting to note that several of the systems I did switch of continue to do poorly, while several others are flat. Only one of the switched off systems has had positive results in the last 6 weeks.

Started this thread Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Are there any eval firms that allow you to sink to your …
Traders Hideout
New Micros: Ultra 10-Year & Ultra T-Bond -- Live Now
Treasury Notes and Bonds
Deepmoney LLM
Elite Quantitative GenAI/LLM
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
NexusFi Journal Challenge - April 2024
Feedback and Announcements
 
  #62 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

Everything back to normal so hoping to make some progress before getting into 'Holiday Mode'.

Was having a very nice month trading the few systems I have left but then on the 29th had a large down day (70% larger than my worst previous day) which pretty much wiped out the month, leaving me with a small monthly loss.

With regards to my attempt to use ML to pick systems, I'm hoping to tune my Random Forest this week with an emphasis on 'True Positives' rather than overall 'Accuracy'.

A collaborator friend of mine recently asked whether I would create him an isystems dataset so that he could test some portfolio optimization rules he was looking at. Since this is his project and not mine I'm not in a position to share what the data was or what the rules he was testing where. I will say though that (to me at least) it illustrated how non-robust this set of systems are. Also while there was some early success (2008/9) since 2011 portfolio performance was generally very poor. This could be random, or it could be showing how much more difficult and competitive this space has become as computers have become more and more advanced.

Started this thread Reply With Quote
Thanked by:
  #63 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192


Last weekend my laptop with all my R code (and hence my isystems analysis) got knocked off the coffee table. Unfortunately it was not save-able. I have a new laptop on the way but no more analysis for the next 10 days at least. This is a shame as my random forest selection system was showing promise and will have to wait.

Regarding isystems in general I have recently discovered that some systems do get deleted, and don't show in the performance reports anymore. Unfortunately this does bring survivorship-bias into the equation!

Regarding the systems I am still trading - had another nice $8 run up this month, and was within $1k of break even again, but then got hit with a $3.5k drawdown yesterday. My trading is up, but my nearly $5k in fees and commissions makes that negative.

One of the few systems I'm still trading continues to do poorly. Plan to do a deeper review in the next few days and maybe switch it of before month end. We will see.

Started this thread Reply With Quote
  #64 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

New laptop arrived and all the important stuff reloaded over the weekend. First thing I plan to do, probably this evening watching the football, is code up a function that will automatically create charts to display and visually check system performance. This is taken from @kevinkdog's teachings and that I illustrated in this post here. I reviewed the systems I am trading around the Winter Solstice and while none of the systems I am trading are doing spectacularly well - none are doing that poorly either - so decided to keep them in place for now.

Then the good news. Had a nice little runup the last week of December and at year end, after 5 full months of trading I was within $923 of breakeven. That's over $8k above my low water mark up, but still almost $6k below my high water mark which was set - set on my 19th trading day. That equates to $5500 in trading profits and $6400 in commissions and system fees.

Then the better news. My end of year good fortune continued into the new year. As of Friday's close I'm only $2k below (September) account highs! Unfortunately I've been here twice before (both in November) and both times saw $5+k drawdowns immediately!

Started this thread Reply With Quote
  #65 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

I guess when you have a team with a top rushing offence and a top D playing another team with a top rushing offence and a top D it's never going to be that exciting, but it was a good game.

Ran into a slight problem, that we're now in 2018, so the isystems web scrapes return the results slightly differently than previously. Not anything I couldn't fix once I realized that was the problem. Not finished yet but almost there.


Started this thread Reply With Quote
Thanked by:
  #66 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

General Update

Another good week in the markets and I'm happy to announce that I believe I've put in a new 'iSystems' account high and exceeded Septembers previous high water mark. Remember this is just a few systems that I've left active from my initial experiment, and not part of any official ongoing experiment.

Finally finished my 'plotting function'. Took a little longer than I expected as I decided to switch from R's base graphics functionality to ggplot2 which is a lot prettier. ggplot2 is something I have wanted to learn for a while, so this became my 'learning on the job' project, and I'm glad I did it. I have a set of 4 different charts, but the interesting one, and the one I've discussed before is below. System 10417 is one of the longest tracked systems with a decent positive PnL - not one I trade, but a good illustration. Interesting system. For it's entire 70 months it's made $75k, but its best run, month 15 to 48 it was up approximately $110k. If you were trading this system would you have pulled the plug, and if so when?



Using Random Forests to Predict "Good" systems

When I went back to revisit my random forest project I was disappointed to find that the last file I had saved (before my laptop disaster) was almost a week old. So I obviously hadn't saved the work from my last few days. That's very rare for me but definitely something I'm more guilty of with R than anything. I tend to have to many different scripts open at one time, testing individual snippets of code. This meant I had to recreate my previous analysis.

At this point in time I have a data set with over 15,000 records, where each record has 13 consecutive months of a single systems results. Each record is given a Good/Bad flag depending upon whether month 13 was positive or negative. The goal then is to use the first 12 months of the data, and some additional analysis of those 12 months, to predict the Good/Bad flag. I am doing this with a Random Forest ("RF"), which is a popular Machine Learning ("ML") Algorithm, that creates hundreds of different decision trees, and then classifies based upon the popular vote of those decision trees. There are several ways that you can 'tune' an RF most of which effect how the trees are selected and built/expanded. Number of Trees, number of variables/features considered at each node, minimum size of each node and maximum tree depth are the four most popular. R is unfortunately not multithreaded and running an RF with this dataset takes almost 50 seconds. Hence to analyze 4 different options of each of the four tuning options, means running 264 different RFs which takes over 3 hours. This doesn't take into consideration that since the features are selected randomly, different runs on the same data will yield slightly different results. Hence proper model testing also involves cross validation, most often k-fold cross-validation. Basically this means running the same analysis multiple times and combining the (error) results. Now we are talking about hours multiple times. In other ML projects I have found that tuning RFs can improve results significantly. With this dataset my initial results show little improvement by tuning.

The problem I run into is that RFs, like most ML algo's optimize to have the lowest error rate. Since 57% of the records are classified as bad, classifying every record as bad immediately means the algo is 57% correct. Since there are more bad than good, improving the 'bad' classification yields a better result than improving the 'good' classification. But I don't care about the bad classifications, I only care about good classification. Even if I can correctly predict bad systems, I can't bet against them. So what I care about are good predictions, and how many of the good predictions are correct. Technically speaking I'm less interested in the error rate/accuracy and more interested in what is called the precision, or the True Positive vs the False Positive rate. One way to do this, is to adjust how the RF counts votes/classifies. Normally voting is decided by a simple majority, but that can be changed. While a RF gives you it's classification of each data record, it also gives you the voting results. Hence it's very easy to only consider positives where say 75% of the trees predicted positive rather than the normal 50%. While this decreases the number of predicted 'good' classifications, it increases the accuracy of those classifications, aka the precision. The higher I set the required cutoff the greater the precision becomes. If I set it to 90%, every system predicted to be 'good' actually turns out to be good. The problem is there's only 8 of them (out 15+k). Initially I was disappointed at this point, and started trying to improve the precision. Eventually I realized I was approaching it wrong, and needed to balance the precision vs the number of systems, in order to maximize PnL. This is where the results begin to get interesting, but I need to double check some things first.



In the charts above
"Pct Cutoff" is the "Percentage of Trees that are needed to vote good" inorder for a record to be classified as "good"
"Number of Systems" is the number of records that were classified as "good"
"Precision" is True Positives / (True Positives + False Positives)

Started this thread Reply With Quote
  #67 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

Fixed the issue that one of the two charts posted yesterday wasn't showing up.

This chart is rather crude, but blue represents trading the highest rated system every month, while red is trading the lowest rated. Rated meaning percentage of the trees within the RF (500 trees total) voting "Good". The downward sloping red line for the last two years is because the RF is picking systems that aren't trading, hence the monthly loss is the cost of the system. This makes me realize something. I am still trying to predict good vs bad where good is positive and bad is negative. I'm not yet trying to identify the systems that make the most money. For example the RF could be giving a system that $1 every month a higher rating that one that makes more money but has the occasional down month.

edit: I should point at that the training data only includes trading results and other statistics derived from the trading results. There is no descriptive information (System Number, Name, Developer, Contract etc) given to the RF. So when it is identifying systems that have stopped trading and are losing money consistently, that is NOT because it has the system number and can tell that from previous months.

The eventual plan will be to have several models and create an ensemble model.


Started this thread Reply With Quote
  #68 (permalink)
dadarara
TelAviv/Israel
 
Posts: 9 since Jan 2016
Thanks Given: 1
Thanks Received: 1

algo by the name : CIRUS RVST SILVER ARTICROAK

results for live activity:
Jul17 - Feb18(curr)
+2700,-3913,+3910,+5460,+4881,+9575,-2727,+3660

looks amazing.

what am I missing? the Live activity is looking very good and in line with the historical run.

so why do I need anything else?
why this should not be the first choice ? and maybe the only one?

Reply With Quote
  #69 (permalink)
 
SMCJB's Avatar
 SMCJB 
Houston TX
Legendary Market Wizard
 
Experience: Advanced
Platform: TT and Stellar
Broker: Advantage Futures
Trading: Primarily Energy but also a little Equities, Fixed Income, Metals and Crypto.
Frequency: Many times daily
Duration: Never
Posts: 5,041 since Dec 2013
Thanks Given: 4,375
Thanks Received: 10,192

Greetings @dadarara and thank you for your interest in this thread.

First let me say, when I did my initial work this system was only weeks old and as such was eliminated from the analysis. Since I decided the hypothesis that recent performance similar to that in the back test isn't necessarily a good indicator of future success, I have not analyzed any new systems. I know from reference tables though that isystems have added a lot (and deleted a few) systems recently.

Your question gives me an oppurtunity to show off my latest system charts...



Two things jump out at me looking at that chart.
  • First, the fact that the equity line is below the lighter of the two gray funnels means that the system is performing 2 standard deviations worse than the backtest results.
  • Second, in the eight months that it has been live, it has had two drawdowns that were 4 times larger than the largest drawdown seen in the backtest
Performing some quick calculations the in backtest performance was $7540/month with a standard deviation of $4716, while since live it is $2943/month with a standard deviation of $4387. So your getting less than 40% of the performance with almost the same risk. (Ave/SD's 1.60 vs 0.67). So from a "compared to it's backtest perspective" I don't think it's in line with it's historical results at all. Of course that doesn't mean it isn't a good system.

Started this thread Reply With Quote
Thanked by:
  #70 (permalink)
dadarara
TelAviv/Israel
 
Posts: 9 since Jan 2016
Thanks Given: 1
Thanks Received: 1


whooo
an excellent analysis. I am really amazed at the depth of your knowledge and how passionate you are with this subject.
great great work.
thank you for somewhat opening my eyes on this.
What I learn from this is that there is probably no system (openly available to public) that will behave exactly like its historic run. some will do better than others and only incubation/live period can tell that.

I myself have written my first algo on Ninja platform, on DAX, which I run in incubation for about a month now. (live data/demo executions).
it more or less follows the historical projections. I mean after each session I run it again but this time as a historical run.
there are differences in the results. not huge ones but they are there. I have only one month of data so its not really enough.
I would like your opinion if I may. I am not very scientific about this all, and probably misunderstand a few things.
I am puzzled. From one hand the P&L/yr on historical 5 yr data is above 200%, taking the highest drawdown of 17k+40%.The session to session success rate is 65%.
but the Sharp/Sortino ratios are quite low (0.7/1.8). and the profit factor calculated is 1.3.
my algo makes between 4 to 12 trades per day. the win/lose ratio is 2.


So on surface, it looks promising, but the ratios are not very good. Compared to some of the algos in the iSystem, they have less P&L numbers but high ratios, together with even bigger drawdowns. if I compare to same level of drawdown , their P&L drops considerably.
So my thoughts are, that even if my ratios are bad, the reward is very high (assuming it will stay that way).
So I am confused having the low ratios and high P&L.

Also, I am of course trying to play with the parameters so that the curve will be nice looking. (curve fitting?)
and the total P&L and drawdown will be good. But what does it mean that over the 5 year market conditions the totals are positive, so why is it bad?
isn't it logical to assume that the better the historical results will be, so the realtime future results will be as well? I mean obviously the "past results do not...." but I rather start live testing with something which has the best past results rather than with algo that on paper doesnt make money consistently.

I tried to run walk forward optimization on some parameters, but it is negative compare to the standard run over the 5yr period. So whats the problem with staying with the same parameters that work for 5 years? (assuming of course that the incubation/live prove it to be close to expectation) Moreover, running walk forward proves that past result of a period doesn't promise good results for the next test period. Seems like an average parameter set that is good for whole of the 5 years may as well be the best option, suitable for most of the market conditions.

again, I am missing lots of things. and trying to learn on the run. apologies if some of my thoughts may be silly.

Reply With Quote




Last Updated on October 18, 2022


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts