I went through the screens you showed on post #295 (I was unable to find screen #3 or Screen #4) so here is my question. Each time I bring up journal.xlsm I get the screen below with the security warning. Should I get this screen each time and each time I need to "enable content". If I look further for screen #3 and #4 will this solve the problem of the security warning coming up each time I start up Journal.xlsm?
Well, you can also choose to enable macros by default, but I would recommend to NOT do that as macros can be harmful (just like a virus). So I leave it on this setting that excel asks me each time.
I'm at the moment redesigning the Monte Carlo simulation more or less from scratch to make it much more powerful and perhaps even faster at the same time.
The reasons for this are:
1. limited functionality and poor performance of the old one
2. this thread: Risk of Ruin as I had the feeling that the available risk models are greatly inferior to Monte Carlo simulations (at least in this field where there is no strong correlation between individual datapoints) and we hit a brick wall with these models....
What I would be really interested in is:
1. how many trades per iteration (in a row) would you like to simulate (maximum)
2 How many iterations would you like to run (always fixed)
tellmetellmetellmetellme
vvhg
P.S. I had to use some smileys as the topic is rather dull (yes, even I think it's dull....)
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
Now that I converted the mayor parts of the MC to VBA this whole thing is waaaaaaaaay too slow. At some point in the code I seem to have f****d up quite badly.
But I have not the faintest idea of where that might be! Note to myself: Remember that I am not a programmer, not even a bad one!
If any VBA gurus are out there, now is the time I could really use your help.....
the hard part is debugging, add print statements to display time to see how long it take to reach a particular print statement keep narrow it down until you find which line of code creates slowness.
The following user says Thank You to cory for this post:
I'll try that tomorrow, and I think I might have an idea where to look for it, but the main thing is probably my elegant coding style, haha.
If the debugging doesn't get me further, I'll post it here.
Thanks to @cory again!
I found the brakes, it was the way I wrote output to the cells:
instead of this method:
I now use this:
and now it's a f*****g race horse.
500 iterations with 1000 trades each take about 10 seconds on my machine.
You can chose now how many iterations of how many trades you want to run.
You can also chose to perform the simulation with fixed position sizing or with fixed fractional position sizing (money management).
Attached is the beta to play around with (@madLyfe hint, hint ) (Excel 2007 & up only).
As soon as we have some confidence and the glitches are out, I am happy to call it a final....
vvhg
Hic Rhodos, hic salta.
The following 5 users say Thank You to vvhg for this post:
No, this is simply due to half entered data, wanted to give you sth to play around with.
The links were very well hidden in charts, in fact, I had to delete the charts, these links can be like viruses.....
Have you tried the MC yet, it should be rather quick now?
ya everything seems really snappy.. i dont use MC at all, but going through the options it does seem to be working well.. i dunno if its working correctly tho(i wouldnt know what that looked like).. im not getting any errors either...
dont believe anything you hear and only half of what you see
Hi guys and girls. I have made a new version of my importer of the results from Ninja.
New version now supports:
1. The en-US locale (choosable) which might have been a issue for those of you having it as default in your system:-)
2. Definable delimiter of the cells in the csv file (defaults to ",").
3. Better handling of the dates and times
How to use:
1. Export the trades list from the backtest results from ninja in CSV format. The format of P/L must be in points, I will add the support for other two types later.
2. Load the file in the program
3. Click Import and choose the destination to save the journal.
done...
The built exe is standard .NET assembly. If you have .NET you should be able to run it. If you have NinjaTrader it will run:-)
I have attached also the sources, so feel free to modify.
Happy browsing through your data...
The following 5 users say Thank You to petrmac for this post:
It is amazing what is possible when someone knows what he is doing, (unlike me trying to fry my cpu with the worst vba code known to mankind).
Thank you a lot @petrmac! I will play with it tomorrow, also a good way to test my MC engine a bit more....I want to have a few more performance tests.
And perhaps I find a way to make it a bit faster (although it is probably fairly optimized now, at least for my skill level which pretty much reflects the fact that until two days ago I only ever copy pasted vb code).
The amazing thing is that although this time is obviously too short to learn vb, it is plenty to learn to dislike vb.....I rather have C#.
vvhg
Sent from my iPad using Tapatalk HD
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
actually i was gunna post in hopes in thats what hes talking about! i was reading about his tool awhile ago but thought it was a dead project.. if its able to import trades into our journal i have some cool ideas for it!
dont believe anything you hear and only half of what you see
Yes, it's awesome, makes my efforts look like a kindergarden project, lol!
The only thing I ask myself is whether it has to import into a clean journal or whether it finds the end of the existing records and pastes them below....
I also have a few ideas what more to do with it, but that's entirely out of my league.
vvhg
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
It creates a new journal each time (it is compiled in the program as embedded resource). But I guess the adding to existing could be implemented. It would just take some time.
I might also add in a more recent version of the journal, once it is stable.
The other idea spooking around in my head is probably quite complicated: grouping of individual trades to one trade with multiple cars and exits. the grouping will probably require some rather complicated logic, the reformatting bit is probably the easier part of it.
That should be rather easy as the journal tab hasn't changed since a few versions, don't know which one you are using though...
The import is compatible with the journal in terms that it takes the trades and if they have a numeric suffix in the EntryName lets say:
KumoBreakout
KumoBreakout2
KumoBreakout3
it will aggregate those three into one trade as three vehicles. The only thing you have to do is to mark each individual trade properly. I do that that I have three profit targets (two trailed to some extent and one runner) with different name for each and they merge nicely in the excel.
When there is new entry that has no suffix it starts counting again.
So this is already supported.
The version of the journal is something like 2.1 or something... not terribly old, but not new either.
The following user says Thank You to petrmac for this post:
OK, I just had a look, nicely done!
Though I have no idea how to mark the trades outside a strategy ( I use the ATM).
Could that be done with minutes as well?
But the journal tab has not changed since then, if I'm not mistaken.
vvhg
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
would you be able to make it import to our journals? and if thats possible i would suggest some feature requests from there depending on how the whole process works..
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following user says Thank You to madLyfe for this post:
1. grouping by time for those using ATM strategies
2. adding the newly parsed record to an existing journal (requires xml excel file and user needs to locate it on the disk). I don't want to perform the heuristic analysis, so it will be perhaps required to insert a line after which to append...
What do you think?
The following user says Thank You to petrmac for this post:
ya that would be awesome to work with this project/journal that way the users here at futures.io (formerly BMT) have something stable/reliable/relatively simple.. i think @vvhg has the journal fields pretty solidified so imputing to them shouldnt be an issue(future changes wise)..
one issue i can see off the top of my head is that the instrument list that is in the journal is not standardized.. i say that in we are allow to name the instruments anything with want in the instruments list.. we will most likely have to adopt the ninja naming scheme..
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following user says Thank You to madLyfe for this post:
Ok, one issue I am not able to overcome is the destruction of the header format in milt line cells.
Perhaps we could do something with them in the excel first..
It is not a big issue, but the lib I am using for writing excel cannot do anything about it.
We could either replace by shorter text so that it fits one line or use "real multiple lines".
As I don't really have any clue by what means you get the data into the sheet, the next sentence might or might not make the least bit of sense.
I don't really understand why the cell gets changed in the first place as you don't write to it, or does it just break the format even when the cell is not touched?
Also what do you mean by "real multiple lines"?
I thought if the problem persists, perhaps we could tackle that problem from the vba side if that's easier...
If you see the resulting file it damages text in some header cells in journal tab. I am skipping them in the program so I have no idea why they look so bad.
The cells are formatted to show two lines even though there is just one cell. That is probably the issue.
Those which fit one line are not damaged by the import.
vvhg, could you please add in more instruments in the excel? I will then update the program with newer version.
I vote for 6e, e7, nq, tf, m6e, es :-)
The following user says Thank You to petrmac for this post:
im gunna go ahead and say that we should just add all the eminis, currencies, bonds, softs, metals, etc, otherwise will forever have requests.. here is another issue, maybe in your program @petrmac you can add an option to omit sim trades from the import to the journal?
dont believe anything you hear and only half of what you see
If you aren't backtesting, there is exactly the same grid in the Ninja control panel -> Account performance tab - > Trades tab.
Then use the same way to export the results as described earlier.
The following user says Thank You to petrmac for this post:
I'll try to change the format of those cells to sth more compatible, and if that is not possible, I will write a fix for that into the sheet VBA.
I could easily add loads of instruments, but there is one disadvantage: the manual selection in the corresponding instrument fields will take considerably more time if you use the drop down list (you have to scroll there).
If you type into these fields everything stays the same (I use Auto Fill, which makes life quite easy with the journal).
But I think this price we sure can pay, and users can always delete the symbols they don't need.
Shall I get the commissions from a broker or do you want me to leave them blank?
Could you both please revise your posts where you mention the symbols and see if the list is "complete"?
@madLyfe I'm not searching for all the symbols myself, forget it.
If you want all metals, bonds softs, minis and what not included, tell me at least the symbols (don't even think about posting a screenshot of all NT available instruments!), also could you post your current list so I can copy/paste that?
Well, it's actually not that hard to add an instrument, any below average excel user even without expert programming skills should manage....
You can omit them when you export the list: Account Performance > Advanced > deselect the sim accounts.
thanks that worked! things im noticing:
- for some reason the inputted date isnt taking on the journals formatting
- it is writing to the comment cells which should be user selectable like the method cells are..
- i dont think the 'minutes in the trade' in the trade are working properly.. if in a trade for under 1 minute should = 1 as well im thinking, just an idea..? my trades im testing were longer than 1min but showing 0 min.
- we are going to need some type of notification that alerts the user if they have already inputted those trades for us forgetful/clumsy traders..
im sure youve have already thought of this but cant you just have the inputter skip the top cells and start with row 3? im not a programmer so if that sounds to simple and dumb its prolly because it is..
and other than the top cells issue thats what im noticing right now..
after looking over it, NT uses the standard naming scheme for instruments as youd expect, 6E = 6E, CL = CL, etc.. we shouldnt actually need to add an overwhelming amount of instruments for people.. maybe just a few more of the common ones for people and everything should work as expected.. plus that can be towards the end of this once we get everything running smoothly..
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following 2 users say Thank You to madLyfe for this post:
If you see the resulting file it damages text in some header cells in journal tab. I am skipping them in the program so I have no idea why they look so bad.
The cells are formatted to show two lines even though there is just one cell. That is probably …
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following 2 users say Thank You to madLyfe for this post:
- formatting is according to the selected locale. Should I add other ones? It has to correspond with the csv locale though.
so if the system locale is en-US the dates would be formatted in that way... but let's say this is minor issue
- yes it is writing everything that could be read from the csv - that includes instrument and method. In case you have multiple entry methods in you strategy, this makes sense.
- minutes were written incorrectly - now will be fixed in new version
- Notifications might be nice but let's try to fix more important issues first like trade grouping. Keep in mind, that this could prolong the import, especially when there are many down in the journal already.
The following 2 users say Thank You to petrmac for this post:
The other thing I was thinking about is if it would be possible (within practical limits of course) to make the import tool an excel add-in?? It would be quite cool to import data into your journal where you only had to point to the csv and the settings are all saved in there so that's really all you have to do....
vvhg
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
On currency products, due to the error in Ninja export, there is no MAE / MFE / ETD value when exported in points (the values are zero in the csv).
How do you suggest to fix that? Maybe calculate the amount of ticks from the money value???
I don't understand what you mean. It currently writes the method, isn't it?
You mean to skip writing of the method to the journal so that you could fill it yourself??
The following user says Thank You to petrmac for this post:
OK, here is my RC: the bugs I could find and the ones pointed out to me are killed.
@petrmac, try this one to see if the problem with the heading is gone...
vvhg
Hic Rhodos, hic salta.
The following 2 users say Thank You to vvhg for this post:
I don't see any other option, just remember to factor in commissions....
Exactly, the best would be to make it user selectable to write or skip it. As you pointed out, for strategies it makes sense to write it.
But for discretionary trading it would be better to skip it, so while trading you indert method, grades, comment and thoughts and after trading hours you just import all the numbers....
vvhg
Hic Rhodos, hic salta.
The following user says Thank You to vvhg for this post:
Fixed issue:
minutes in trade are now calculated correctly
Now featuring new embedded version of the journal. Thanks @vvhg. It is MUCH faster and also a bit smaller. Also the headers are now displayed properly.
Just one problem... the dashboard does not recalculate. Am I missing something?
The following 2 users say Thank You to petrmac for this post:
Somehow the current balance stays the same after import (used to recalculate after defining the starting balance) and the equity curve stays the same...
Same for some other fields... I'll do some more tests...
The following user says Thank You to petrmac for this post:
Thanks for the new version, vvhg. I'll give it a try....
In the meantime I am working on the correct ticks count for currencies futures... mostly there. Now just have to do the math during the import (deduct the commissions etc).
I'll post a new version today.
Petr
The following 3 users say Thank You to petrmac for this post:
vvhg, I actually have a tip for the feature in the journal.
Wouldn't it be nice to have some graph showing the gap between the real vs unrealized profit? (i.e. the amount of money left on the table due to the bad exit).
Perhaps even link that to the time series - so that we can see whether we are getting better or not.
I am struggling with the exits in my discretionary trading and this could help me a lot.
The following 3 users say Thank You to petrmac for this post:
- calculates the MAE/MFE from the currency amount of the trade (the export with units in currency is now supported). To do so it loads a csv with the instruments settings (instrument-list.txt) attached to the program.
- skips the Method and Comments fields in excel if data handling set to discretional (it puts there name of entry and followed rules for automatic)
- now features embedded journal excel beta 9
Here I would like to ask someone to fill in the settings for instruments into the instrument-list.txt file in order to support the currency format of data export:-)
Next feature to implement is to group the trades based on entry time (minute).
The following 3 users say Thank You to petrmac for this post:
Well, I had an idea to show the line/or bar graph with the horizontal axis = time and vertical = ticks and display to series (realized profit, max unrealized profit per trade).
However the time is not hour in day, but rather each day (and in more detailed view each trade).
Maybe something like in the picture here but the lines in same graph or perhaps stacked bars (two for each trade).
What do you think?
The following user says Thank You to petrmac for this post:
This is essentially what I am doing in the importer except I don't want to have anything to do with Microsoft Interop libraries. The are difficult to handle and the API is not well made. I prefer the library I am using right now.
Anyway, this does not solve the issue how to import to excel from within the excel:-)
I can do some research when I have more time...
The following user says Thank You to petrmac for this post:
I don't like the first graph either. It is to jagged so that one is easily lost in it.
But I think that losers (SL hit) are not that interesting in this case, since we are now concentrating on the money on the table... What about omitting the losers?
Just keep it simple... Two lines, one realized, one potential...
IDEA! ok i know were working on getting everything working for the normal features that we have outlined, but if i dont get this out it might get lost in the space that is my head!
i think an updater needs to be built into this importer.. so it will become and importer/updater app.. when new journal versions come out, all one will have to do is load up the old journal with all of the users trades, and then load up the new version of the journal, and it will import all the data safely so the end user doesnt have to worry about any mistakes complications! this would also help in diagnosing issues here because we could eliminate the end users error of 'mismigration'..
what you guys think?
this sounds pretty solid to me, but i also dont use all of the graphs as much either..
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following user says Thank You to madLyfe for this post:
Hmm, seems to me like an error in date.
Either there is a mismatch in the locale used to export the csv and the one selected in the importer or there is some malformed input.
Could you please send me a sample export file so that I am able to see how the dates are formatted?
Thanks
The following user says Thank You to petrmac for this post:
hey @petrmac, im working on the text file for ya.. are you taking the description/name from NT? that way we can be consistent? also is this just for currencies?
dont believe anything you hear and only half of what you see
¯\_(ツ)_/¯
(╯°□°)╯︵ ┻━┻
The following 2 users say Thank You to madLyfe for this post:
Cool, that helps.
I have there just two entries for testing... so the description doesn't matter. You can easily use the ones from NT.
As for the list... no not just currencies, it is needed for all instruments if you for some reason want to import file that is in currency format (i.e. not points).
So put there anything you like...
The following user says Thank You to petrmac for this post:
ok ill use the NT description as the standard naming scheme them.. i am going to include the currencies that i have currently added(the most popular and a few extras) and then ill add the rest of what we have built into the journal as default. does that sound good? and like the journal, users and add their own instruments to suit their needs as they like beyond the defaults?
dont believe anything you hear and only half of what you see
The issue pointed to is fixed.
I improved the error handling and thanks to @madLyfe there are already some instruments in the list. The importer can now read trades with them.
USAGE comment:
be careful when deciding the delimiter and the locale. Ninja will export the data based on the Windows regional settings and that can make a huge difference in the format of almost everything (decimal numbers - "." or ",", dates).
If there is a locale that is gravely needed, just drop me a line and I will add it. This is not a big change.
The following 5 users say Thank You to petrmac for this post:
The following is a PM exchange I had with vvhg - posting here at his suggestion:
Hi VVHG
Just looking at your spreadsheet and thx for your efforts! I am not clear as to where the raw data comes from? Does it somehow feed in from NT? Or can I migrate data into it from existing spreadsheets? I do have spreadsheets of a different format and without some field information - ie I have not tracked mae/mfe or time in trade and I dont use VWAP. If have some fields blank will this cause serious functional issues? Also I am an all in all out trader so do I simply leave exit2 and 3 blank?
Sorry if some of these questions are answered in the thread.
many thx
Lindsay
Unfortunately at the moment you have to enter the data manually ( or use copy paste column by column from other excel files, but make sure to not paste the format, only paste values. How to do that can be found in a liknk in post #2 "migrating from older versions"), but we are working on a tool that will be able to import from NT, you can download it in the thread to play around with it.
If you don't enter mae and mfe that's ok but obviously some tools of the journal related to mae and mfe will not work. If you plan on using mae and mfe in the future, I would suggest to use two journals, one for the old data and one for the new data including mae and mfe ( otherwise when mae and mfe are only entered for half the trades it may lead to skewed results). Vwap can be omitted, I don't track it myself...the only thing not working then of course is the vwap statistics on the overview tab.
Regarding exits, it is up to three but if you only need one, that's fine ( obviously exit three statistics would not show as they don't apply).
Would it be possible for the spreadsheet to have the capacity to accept previously calculated trade system statistics so that it could generate synthetic trades/curves and other stats? For example - I know my
ave win
ave loss
win %
% of account I risk per trade
this defines my methodolgy
It would be great to be able to input those and be able to access the other functions on the SS.
dont believe anything you hear and only half of what you see 
)
im not getting any errors either...
