Alternatives to Monte Carlo testing - Matlab, R project and Python | futures io social day trading
futures io futures trading


Alternatives to Monte Carlo testing
Updated: Views / Replies:900 / 6
Created: by Big Mike Attachments:0

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors – all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you don’t need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 
Thread Tools Search this Thread
 

Alternatives to Monte Carlo testing

  #1 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received

Alternatives to Monte Carlo testing

Source:

Alternative to Monte Carlo Testing | rbresearch


Quoting 
When we backtest a strategy on a portfolio, it is a simple analysis of a single period in time. There are ways to “stress test” a strategy such as monte carlo, random portfolios, or shuffling the returns in a random order. I could never really wrap my head around monte carlo and shuffling the returns seemed to be a better approach because the actual returns of the backtest are used, but it misses one important thing… the impact of consecutive periods of returns. If we are backtesting a strategy and we want to minimize max drawdown, consecutive down periods have a significant impact on max drawdown. If, for example, the max drawdown occured due to 4 consecutive months during 2008, we wan’t to keep those 4 months together when shuffling returns.

In my opinion, a better way to shuffle returns is to shuffle “blocks” of returns. This is nothing new, the TradingBlox software does monte carlo analysis this way. I had a look at the boot package and tseries package for their boot functions, but it was not giving me what I wanted. I wanted to visual a number of equity curves with blocks of returns randomly shuffled.

To accomplish this in R, I wrote two functions. The shuffle_returns function takes an xts object of returns, the number of samples to run (i.e. how many equity curves to generate), and a number for how many periods of returns makes up a ‘block’ as arguments.The ran_gen function function is a function within the shuffle_returns function that is used to generate random blocks of returns.

shuffle_returns returns an xts object with the random blocks of returns so we can do further analysis such as max drawdown, plotting, or pretty much anything in the PerformanceAnalytics package that takes an xts object as an argument.

This is not a perfect implementation of this idea, so if anybody knows of a better way I’d be glad to hear from you.

The example below uses sample data from edhec and generates 100 equity curves with blocks of 5 consecutive period of returns.

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).


R

 
Code
require(PerformanceAnalytics)

#Function that grabs a random number and then repeats that number r times
ran_gen <- function(x, r){
  #x is an xts object of asset returns
  #r is for how many consecutive returns make up a 'block'
  vec <- c()
  total_length <- length(x)
  n <- total_length/r
  for(i in 1:n){
    vec <- append(vec,c(rep(sample(1:(n*100),1), r)))
  }
  diff <- as.integer(total_length - length(vec))
  vec <- append(vec, c(rep(sample(1:(n*100),1), len = diff)))
  return(vec)
}

shuffle_returns <- function(x, n, r){
  #x is an xts object of asset returns
  #n is the number of samples to run
  #r is for how many consecutive returns make up a 'block' and is passed to ran_gen

  mat <- matrix(data = x, nrow = length(x))
  for(i in 1:n){
    temp_random <- ran_gen(x = x, r = r)
    temp_mat <- as.matrix(cbind(x, temp_random))
    temp_mat <- temp_mat[order(temp_mat[,2]),]
    temp_ret_mat <- matrix(data = temp_mat[,1])
    mat <- cbind(mat, temp_ret_mat)
  }
  final_xts <- xts(mat, index(x))
  return(final_xts)
}

#get sample data
data(edhec)
a <- edhec[,1]
a <- head(a, -1)

start <- Sys.time()
yy <- shuffle_returns(a, 100, 5)
chart.CumReturns(yy[,1:NCOL(yy)], wealth.index = TRUE, ylab = "Equity", main ="Generated Equity Curve of Shuffled Returns")
end <- Sys.time()
print(end-start)
Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following 2 users say Thank You to Big Mike for this post:
 
  #2 (permalink)
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
  #3 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received


I posted this because I wanted to ask input about his conclusions. Specifically:


Quoting 
In my opinion, a better way to shuffle returns is to shuffle “blocks” of returns.

So let's say during the meltdown of 2008 your portfolio took a nose dive. But you have a 10-year series of data to feed your monte carlo sim, is it really fair to allow all 10 years to be mixed and mashed up?

No, I would agree with him -- it is better to keep those blocks (like 2008) together since they represented real world decline that did not exist in other areas of the simulated series.

Bootstrap sampling... Thoughts?

@treydog999, @artemiso

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #4 (permalink)
Elite Member
seoul, Korea
 
Futures Experience: Intermediate
Platform: Multicharts
Broker/Data: CQG, DTN IQfeed
Favorite Futures: YM 6E
 
treydog999's Avatar
 
Posts: 894 since Jul 2012
Thanks: 291 given, 1,006 received

I have read this article months ago and did find it interesting. the difference between bootstrapping (in general terms) and Monte carlo (as implemented by MSA) is sampling with and without replacement. Without replacement will be able to generate more extremes (larger streaks, larger max DD, different overall net profits etc) because it is not constrained. However this will distort the original sample distribution. MSA's implementation of bootstrap/monte carlo is sampling without replacement, so your distribution remains the same. Only the order in which the trades occur becomes shuffled. It depends on really how you want to attack the problem, both have pro/cons to them. But they are both legitimate methods of doing frequentist statistical inference analysis.


This methodology is chunking data then doing the bootstrap/ randomization. I actually do not like this method because who is to say that the largest max dd that occurred to you in backtest can not be surpassed by a randomization with sampling without replacement. I am fairly certain it can be surpassed with sampling with replacement as you could replicate your worst trades several times in the curve which would not occur with chunking. This especially becomes apparent as you go to extremely large numbers of trials as you will eventually reach the optimal setting (worse possible draw down) through random chance.

If you really really wanted to find out how bad things could get, just string together every loss the system had and add them up. That is really the absolute worse possible draw down given your back tested equity curve. Its also just as likely to happen as any of the other randomized equity curves generated by any method, sampling with or without replacement, chunking etc.

Reply With Quote
The following user says Thank You to treydog999 for this post:
 
  #5 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received

I actually don't fully understand what @treydog999 said, so I won't comment on that.

Just eyeballing it, both the ran_gen and shuffle_returns functions seem to be very slow. There should be an R profiler that you can use to verify this. If you intend to employ this frequently, it should help to rewrite it. You can generalize this approach and improve on most Monte Carlo implementations written out there (R developers tend to be poor programmers). This is a fun, relatively simple, but rewarding project, so I recommend everyone to try - I did something similar before so I'll send you some hints.


Last edited by artemiso; November 5th, 2013 at 08:53 PM.
Reply With Quote
The following user says Thank You to artemiso for this post:
 
  #6 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,218 received


artemiso View Post
I actually don't fully understand what @treydog999 said, so I won't comment on that.

Just eyeballing it, both the ran_gen and shuffle_returns function seem to be very slow. There should be an R profiler that you can use to verify this. If you intend to employ this frequently, it should help to rewrite it. You can generalize this approach and improve on most Monte Carlo implementations written out there (R developers tend to be poor programmers). This is a fun, relatively simple, but rewarding project, so I recommend everyone to try - I did something similar before so I'll send you some hints.

I agree, it is incredibly slow. But so is R and Matlab in general it seems, due to lack of multithreading. I am still trying to figure out if I can multithread individual functions.

Anyway, please contribute code improvements

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #7 (permalink)
Elite Member
Manchester, NH
 
Futures Experience: Beginner
Platform: thinkorswim
Broker/Data: TD Ameritrade
Favorite Futures: Stocks
 
Posts: 902 since Jul 2012
Thanks: 603 given, 1,785 received


Big Mike View Post
I agree, it is incredibly slow. But so is R and Matlab in general it seems, due to lack of multithreading. I am still trying to figure out if I can multithread individual functions.

Anyway, please contribute code improvements

Mike

Not sure if it's useful, but I sent you some hints anyway.

Reply With Quote

Reply



futures io > > > > Alternatives to Monte Carlo testing

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Possible to Access Trades Collection from Inside Optimization Method? (Monte Carlo) kbeary33 NinjaTrader Programming 3 August 13th, 2013 07:45 PM
monte carlo simulation interpretation help needed wallsteetking NinjaTrader Programming 2 October 21st, 2011 07:36 AM
Ninja Trader Monte Carlo Xeno NinjaTrader 8 April 12th, 2011 12:10 PM
Monte Carlo Simulation Big Mike Psychology and Money Management 3 February 7th, 2010 12:06 PM
NinjaTrader 7 to include Monte Carlo Simulator Big Mike Traders Hideout 1 June 14th, 2009 02:41 AM


All times are GMT -4. The time now is 12:47 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-11 in 0.13 seconds with 19 queries on phoenix via your IP 107.20.115.174