Walk Forward Optimization - memory consumption - futures io
futures io futures trading



Walk Forward Optimization - memory consumption


Discussion in NinjaTrader

Updated
    1. trending_up 1,918 views
    2. thumb_up 0 thanks given
    3. group 2 followers
    1. forum 2 posts
    2. attach_file 0 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Walk Forward Optimization - memory consumption

(login for full post details)
  #1 (permalink)
Germany
 
 
Posts: 24 since Feb 2011
Thanks: 1 given, 0 received

Hallo,

I'm trying to run a walk forward optimization on daily data from 1983 till now.
I try to simultaniously optimize 2 parameters (1. param: from 0,1 to 1; increment 0,1; 2nd param: 1,1 to 10, increment 0,1). Optimization period is 250 days, test period is 80 days.

During every attempt to do the optimization RAM memory is filled up until NT throws the OutOfMemoryException.

The reason for this obviously are indicators used within the strategy that read values from a txt file.
During OnStartUp() of the indi the txt file is read into a string array. This array is used afterwards in OnBarUpdate() for
indicator calculations. The array also contains daily data and each day is represented by one row of data. The array itself consists of 5 columns and 7000 lines. Does anybody know how to calculate the amount of memory that is consumed by this array? I'm actually surprised that it's big enough to freeze the system (32bit, Win XP, 3GB RAM).

Single backtesting this strategy works fine, using the GA during walk forward optim freezes the memory.

Anybody with experience and ideas in that field who knows how to reduce memory consumption?

Here comes the script that constructs the string array from the text-file.

 
Code
public string[,] tabelle(string pfad, string trennzeichen)
    {
      //initialisieren
      string[,] table={{"",""},{"",""}};
      
      try
      {
                  
        string[] fileContent = File.ReadAllLines(pfad);
        int linecount=fileContent.GetLength(0);
        //Spaltenzahl bestimmen
        string[] datensatz=Regex.Split(fileContent[0], trennzeichen);
        int rowcount=0;
        foreach (string wort in datensatz)
        {rowcount++;}
                    
        //table dimensionieren
        table = new string[linecount,rowcount];
          
        string zeile;
          
        for (int i = 0; i < linecount; i++) 
        {
          zeile=fileContent[i];
          datensatz=Regex.Split(zeile, trennzeichen);
            
          int j =0;
          foreach (string wort in datensatz)
          {
          table[i,j]=datensatz[j];//Print(table[i,j]); //Datenmatrix erstellen
          j=j+1;
          }
        }
      }
      catch (System.Exception excep)
        {
          MessageBox.Show(excep.Message);  
        }
        
      return table;
    }
Within OnBarUpdate() relevant the relevant row of the array is avaluated and strings are transformed into double values.

Many thanks in advance for suggestions and help!

Reply With Quote

Journal Challenge February 2021 results:
Competing for $1500 in prizes from Topstep
looks_oneSBtrader82 's Trading Journalby SBtrader82
(173 thanks from 31 posts)
looks_twoJust BEING a Trader: Letting Go!!by iqgod
(120 thanks from 33 posts)
looks_3Wisdom is Emptinessby Mtype
(68 thanks from 25 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(35 thanks from 17 posts)
looks_5Journal for peanuts1956by peanuts1956
(23 thanks from 13 posts)
 
 
(login for full post details)
  #3 (permalink)
Denver, Colorado, USA
 
 
Posts: 36 since Aug 2011
Thanks: 6 given, 17 received


The main issue is probably that you are using 32bit NT and Windows. Its maxed out at around 3.25 GB maximum space in Windows 32bit, and a limit of .NET on your configuration is that it limits you to using 2 GB. NT is based on the .NET framework. You may want to consider upgrading to a fully 64bit trading system, since the RAM limit is quite high.

Memory Limits for Windows Releases (Windows)

Reply With Quote


futures io Trading Community Platforms and Indicators NinjaTrader > Walk Forward Optimization - memory consumption


Last Updated on September 7, 2011


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Battlestations! Show us your trading desk - $1,500 in prizes!

March
 

Call Option Buying: The New Pain Trade? w/Carley Garner

Elite only
 

Importance of Finding Your Own Way w/Adam Grimes

Elite only
 

New Challenge (TBD)

April
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts