Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
In reading a number of other trading journals on this site, I was impressed by the diversity of the ideas and backgrounds of the various authors. This makes sense; there are probably as many ways to trade as there are traders.
As I start to write down my ideas, I envision that in addition to the usual trade logging, I plan to document the process behind the development of technical ideas I have about trading, as much as my actual trading history.
In fact, this will be the focus in Part 1 of my journal. I’m not really a fan of the term “technology evangelist,” but as you will see in the next several posts, I have some ideas which I believe will help not only me, but other traders too.
I’ve been thinking of doing this for some time, but the impetus for my taking the plunge now is that I am in the midst of reading “ Designing Your Life: How to Build a Well-Lived, Joyful Life” by Bill Burnett and Dave Evans. Obviously, I used the book’s title in selecting a name for my journal.
I frankly can’t recall how I first heard about the book, and after purchasing it the book sat on my shelf for a long time, until I finally picked it up. Briefly, the book was written by two professors in design schools who lamented how many students came to them with little or no plan for their futures. They realized the principles they taught in their design classes could be applied more broadly to help these students better plan their lives. First they developed a course based on this idea, which became a book that went on to become a NY Times #1 bestseller.
Maybe it was the NYT website that started me down the path, but regardless, why choose this book as the starting point for my journal? Several reasons. First, I really like the word “Design” in thinking how I want to approach trading.
Design is a more holistic concept than development. I’m aware of the important “left brain” tasks needed to become a successful trader, but these are only pieces of a larger puzzle. To become successful, there needs to be a cohesive strategy. For the strategy to work, parts may be updated or swapped out, but in the final analysis, the goal needs to be to have everything work together seamlessly. When one looks at a classic design such as a sports car or well-designed smartphone, there are no jagged edges. It is taken for granted that each part has been well thought out, both as a component and how it fits into the larger whole.
Secondly, right in the introduction the authors talk about 5 fundamental mind-sets that are used throughout the book. They are: Be Curious, Try Stuff, Reframe Problems, Know it’s a Process, and Ask for Help.
It was my reading about the last one, “Ask for Help” that was the tipping point to get me to share my ideas in a journal. Here’s what the authors had to say about that:
Swap out the word “life” with “trading career”, and you, dear reader, can figure out what I’m asking of you.
I think I have a lot of great ideas related to trading that I would like to share with you. But they need fleshing out, and at times respectful criticism. I’ll be the first to admit that some of what I think is a great idea now may turn out to fail miserably. But I am one person and you are many. If you can help me, I would appreciate it.
And if I help you, all the better as well. As they say in the Home Depot (the last time I visited the US) “Let’s do this.”
As you can see, I joined nexusfi.com in October 2011, back when it was Big Mikes Trading Forum.
To give you a personal history in under 100 words: I’m an American IT guy who is happily married to a woman who insists on getting jobs all over the world. From a starting point of Washington, DC in 2001, we’ve lived in Sarajevo, Bosnia, back to DC, then Skopje, Macedonia, back to Philadelphia, PA, and now Istanbul, Turkey. As I write this, I am surrounded by half packed bags; our next destination is Amsterdam.
Trading is pretty well suited to my lifestyle. I do regular IT work remotely too, but as long as I have a good internet connection, I’m able to trade. My first trading platform was ThinkorSwim, which I still use for some things and will show you later. Once I got involved with this site though it wasn’t long before I started using NinjaTrader.
If you look at my activity on FIO, you’ll see that so far I’m best known for creating the BaBAR (Backfilling Bid Ask Recorder) tool and using it to share data on the monthly data sharing thread.
These days BaBAR is not needed as much as it once was. Bid/Ask data is available in NT8 intrinsically, so there is no need to create a second database (GOMFOLDER) to hold this information. Also, NinjaTrader now has a download service, so there is less of a need for flat file data.
But like the bear who went over a mountain and saw another mountain, I anticipate that there will be a new version of BaBAR at some point. IQFeed is talking about adding additional fields to their data feed which may be useful to traders. This hasn’t even hit beta yet, but when it does, I’ll be on the case.
The other reason I wanted to talk about BaBAR is that the software development effort shaped my view of the market. Kind of like the Karate Kid, who doesn’t realize while he is cleaning cars that “Wax on, Wax off” was also teaching him martial arts, my focus on developing software to track the minutia of trades going off at the bid, ask or other prices was showing me how markets work.
It drilled into my head that the market is composed of players who alternately are aggressive and passive in the market, which is really just a never ending two-way auction. Anything can happen at any time, but understanding how the proverbial price action works can give you an edge in the market.
I came to view Gomi’s toolset as essential to my understanding of how a market works. In fact, I extended Gomi’s tools for my own charts. I shared these changes with Gomi, but he declined to incorporate them into his releases on FIO. As I migrate to NT8 I plan to create indicators that include these extensions, which I will also share. When I do so, it will not be a simple “download this” share. I think it is equally, if not more important to explain my thoughts behind creating an indicator so others understand why it was designed the way it was.
The other take away from the BaBAR experience was that sometimes developers of trading platforms leave important things out; in this case which side of the market a trade went off was left out of NT7. I don’t think this was done with any bad intent, it just reflected the limits of the software designers’ mental map of how markets work, and what they thought traders would need to interact with the market.
Once Gomi came along and showed there was value in Bid/Ask data, it became widely accepted and it’s now hard to avoid. I don’t know if Gomi came up with this idea himself or saw it on another platform and migrated it to NT, but I’m trying to make a broader point.
All software designs are imperfect, or more neutrally and usefully, all designs have limitations which can be extended. Going forward I’d like to share some of my ideas, and get your feedback.
I’m going to say something provocative: I think trading platforms for retail traders have a huge hole in them. A hole that is far, far larger than NinjaTrader not including historical Bid/Ask data in version 7 of their database. This hole may not exist in every trading platform, but it has been true of every trading platform I’ve ever used or even tested. And as I will explain, filling in that hole has proven to be quite a challenge.
Put simply, if one views day trading as an activity similar to competitive sports or performance art, [as I do] where the performance of the person is developed and honed over time, then the next logical thought would be to compare how technology is being used in these domains to assist the person’s development, and then look to see if trading platforms do something similar.
Oh look, a huge hole.
If you were to go to an Olympic training facility, regardless of country or sport, you would see no end of high tech gear designed specifically to measure every aspect of an athlete’s performance. You would see things like cameras that monitored their form, devices that measured their physiology. You’d see questionnaires that monitor the athlete’s mental state. Their caloric intake is monitored. You would probably find that all of this data in a centralized database which would be mined to see ways to improve performance over both short and long timeframes. Even if there was illegal activity such as doping, IT TOO would be carefully tracked. In short, the point of all of the technology would be coherently focused on improving performance.
Is such a thing found in retail day trading platforms? Nope, not even close.
I recall when I first started investigating the idea of retail trading I would often wonder why nobody had focused on this problem. Yes, there are trading journals, of which this is one, but really stretching the imagination to figure out how trader performance could be quantified and characterized was not something that I saw presented anywhere.
My thought back then was that others must have figured out why this wasn’t important. As a complete noob, who was I to question how things were being done? Yet as I learned more, through trial and error, the thought that something was being left out never left me. In fact, I now see leveraging technology to monitor and improve trader performance as one of the pillars needed for me to attain success in the markets. Maybe your success too.
Another book which I’ve enjoyed for many years now is: “ The Visual Display of Quantitative Information" by Edward Tufte.” [Amazon link] I could go on at great length about this book, and I highly recommend you read it at least once, but suffice it to say the book encourages creative ways to think about the presentation of data. In the current context, it made me wonder how a day trader’s performance could be represented graphically.
Sure there are Excel spreadsheets listing trades and how they worked out, but what about something that had a visual immediacy to it. Here is a famous “figurative map” made by Charles Joseph Minard which Edward Tufte references in his book. Perhaps you have seen it.
It is a graphic of Napoleon’s forces as they attacked Moscow and returned in 1812 and 1813. The width of the lines show the size of the army as it progressed through the battle. When you realize they started out with over 400,000 soldiers, the graphic becomes epic, at least in my mind, considering the human tragedy represented by each thinning of the tan and then black lines, culminating in the difference in width of the start of the brown line and the end of the black.
I don’t think a trading chart will ever be anywhere near THAT epic, but thinking about this graphic and others in Tufte’s book made me wonder how each trade can be represented graphically.
Of course there are zillion charting packages out there which will chart the market, and slop all manner of indicators on top of a chart in an attempt at divination, but why has no one had the foresight to put what the trader is doing on the chart?
Such things as where they entered, where they took profits, and was it a good place to jump off or not? Yes, most packages put little diamonds and whatnot on a chart where a buy and sell occurred, but indictors that show a trader’s intent and how well a trade worked for them, which they could later share with others? No way.
A number of years ago, back when ThinkorSwim was my only trading platform, I would often ask myself “Why can’t I put myself on the chart?” Meaning if I can make a chart of say APPL and the NQ to see their relationship, why can’t I do things like put my P&L on a chart to see where I was making or losing money? When I suggested this to ThinkorSwim, their response could be described as “when they stopped laughing, they said no.” I’m exaggerating of course; they weren’t that dismissive, but they had no interest in providing the feature either.
Later, when I started using NinjaTrader 7, one of my thoughts was to try to develop this feature myself. I thought I could do this because NT allows more “under the hood” programming. Still, the idea of putting me in the chart was something that was not supported. As with Gomi’s Bid/Ask information, the programmers just didn’t think that anyone would want to do this.
I think this [major] oversight occurred because traders and trader platform developers tend to be different people. A trader can tell you for example that a 2 point move against their long position feels VERY different if they have a full position on that is below their entry, especially on a day when they’re already down, as opposed to the feel of the exact same market move on a winning position, perhaps one that has already been partially scaled out. Yet to those who write code, a two point move is pretty much a two point move. The market is the market, just the facts please.
This really struck me as odd. We’ve all heard about how the “Holy Grail of Trading” doesn’t exist. Then you hear the corny line “Well, actually it does exist; the Holy Grail is within you.” Which brings me back to why I was surprised that nobody, at least on any platform I had seen, wanted to put a trader’s activities on a chart, beyond perhaps little diamonds where you bought and sold.
Naughty little coder that I am, I looked at the internals of NT7. It turns out that the database with tick, minute, hourly etc. market data is held in the “NinjaTrader 7\db” directory, but the trader’s activity is held elsewhere, in a relational database. (Called NinjaTrader.sdf in the NinjaTrader7 folder)
As fate would have it, my IT experience has been in large measure database- centric, so I downloaded some tools and had a look at the NinjaTrader.sdf database. Fortunately, it was not encrypted or password protected, so it wasn’t long before I figured out that the “nt_order” table held the trader’s orders, and “nt_execution” held the executions (buys and sells) of the trader.
None of this information is available when writing indicators using NinjaScript in NT 7. BUT… NinjaScript is pretty much the C# language, so I wondered… Could I make a database connection within an NT indicator using the usual C# methods for accessing Microsoft SQL Server databases, analogous to how the Gomi tools access external data from the GOMFOLDER? Short answer: Yes.
Longer answer: Yes, but it’s something wholly unsupported by NinjaTrader, has problems with performance and causes NT 7 to crash sometimes, mostly in fast markets.
I first started looking at the NinjaTrader.sdf database as described above in early 2016. In the past year and some, I looked for ways to get my approach to work without problems. The exercise was very frustrating to say the least.
BUT THEN… NT 8 came out. Recently I started testing to see if NT 8 would work better than NT 7 to do what I wanted. Short answer, YES, it works GREAT! No crashes and it works in the blink of an eye.
Longer answer: NT 8 is built on a newer version of Microsoft’s .NET framework. This newer framework allows increased flexibility in dealing with datasets. (Google LINQ if you like) Also, NinjaTrader has been promoting what they call the “NinjaTrader Ecosystem” which encourages third party developers to create a diverse set of add on applications. I am not sure if the enhanced flexibility provided by NT 8 is due to the newer version of .NET, or conscious decisions on the part of NinjaTrader’s software architects to maximize flexibility, or what ratio these two reasons might be responsible, but in the end, WHO CARES?!
Developers can now create indicators in NT 8 that directly access a trader’s orders, positions, equity, etc.
One might say that NT 8 has quite possibly uncovered the Holy Grail of Trading. By accident!
Of course that’s a provocative statement too. Obviously there’s no magic indicator that will tell you how to find only winning trades, but with NT 8, there is the potential to put “the holy grail within you” on a chart. We now have the potential to show in much greater detail how a trader acts and reacts to the market at each point of a trade.
We can also show "the holy grail within each other" if we start to share these graphs around. I'm very excited. Can't wait to get my move behind me!
Before I discuss where I am at in developing indicators for NT 8, I want to share excerpts from an informal white paper of sorts that I wrote in early 2016 to help contemplate the kinds of indicators which could be built that more creatively represent trader activities on a chart. I called these indicators TPIs, short for Trader Performance Indicators.
I want to list some of my thoughts here, to see if others find them interesting, and to see if they have further thoughts for improving my ideas.
Daily Equity Curve – An indicator, appearing on a pane below the main chart which shows how a trader’s equity curve changes over the course of a trading session. Some features:
Lines for both realized and unrealized profit. This way a trader could see if they were in a trade too long and gave too much back.
The option to display in cash, points or ticks, for those who don’t want to see dollar signs flashing while they are in a trade.
Rate of Equity Change – An indicator that is essentially the derivative of the equity, showing were there were large gains or losses.
Size of Position – an indicator on a lower pane showing the size of the trader’s long and short positions over time. I also call this “personal volume.”
Percent of Margin Used – Perhaps included in the Size of Position indicator, an indicator that shows where a trader used more or less margin.
Beyond looking at the trader’s orders and executions, I also wondered about physiological monitors. Taking the Olympic analogy quite literally, why not have heartbeat, skin conductance, etc. available on a chart? Then the TPI chart would start to look more like the output of a lie detector! What a great way to avoid fooling yourself. Seriously, I’m not sure how useful such an approach would be, but as Gomi’s files have demonstrated quite clearly, once the physiological data over time is available in a file, it can quite easily be put on the chart using an NT indicator.
So far, all of these indicators are envisioned to appear in a pane below the main price graph. I also thought about what might be useful in the same pane as the charts that I use, and came up with an indicator I decided to call OCTANE.
OCTANE stands for On Chart Theoretical Average and Next Entry/Exit.
As many of you know, by scaling in and out of a position, the theoretical average, or breakeven point of a trade can change. Once the theoretical average of a trade moves below a trader’s stop price (on a long trade, or above on a short trade) then the trade is guaranteed to make money.
Recall above in my “much bigger hole…” post I talked about how the same move in a market would be felt differently by a trader depending on their position. Well, if you can see graphically that the theoretical average of your trade is such that you are going to make money, then you are less likely to trade in fear and make bad decisions.
This is the idea behind integrating the theoretical average of a trade with a chart. Working with NT 7 prototypes (that weren’t ready for prime time) I could see this was a really great idea. In addition to taking the fear out of many trades, it is also useful in making decisions about when to add to a winning trade.
Like many traders, I’m a big fan of Volume Profile indicators. If you can’t have a guaranteed moneymaker of a trade, one thing you can strive for is to have a theoretical average that is at least “protected” in a market profile sense. If you were in a winning long position and there was a big Low Volume Node (LVN) between you and the market, you have a better chance of keeping the trade profitable, as the market has a tendency to stop and reverse at the LVN before erasing all of your profit on the trade.
Also, besides market profile, if your theoretical average was below a support line, you would be similarly protected.
Being able to strategically place your target limit orders and locations where you might add to a trade such that the resulting theoretical average was in a “good place” would be a useful tool in my estimation. This is the point of the “-ANE” part of the indicator. The idea is that the resulting theoretical average of the Next Entry and/or Exit is calculated and placed on the main chart pane as a horizontal line. The trader would know ahead of time where the resulting theoretical average will be before the orders are filled, and how the resulting theoretical average fits in to the market profile, support and resistance levels, etc.
By seeing how a scale out (or adding to a trade) will affect your theoretical average, you can see if the resulting market position would put the TA in a subjectively better or worse place. The limit order could then be adjusted to do one of two things, or hopefully both. Optimize protection of the resulting TA, and move the limit order to where it is more likely to be hit in a market profile sense, e.g. In the middle of a High Volume Node. (HVN)
Like I said, I’m surprised that trading platforms don’t have the capability to plot these things already.
Having said that, with the new capabilities available in Ninja 8, at least now we can begin to address the shortfall!
Hi bob7123. I have the jigsaw trading DOM as my style is geared towards scalping but longer term I am wanting to incorporate InvestorRT charting software from Linnsoft. FT71 has been collaborating with Bookmap to include cumulative delta in their version called S5 Bookmap. Morad also does a number of webinars about campaigning (scaling in and out of positions), which moves the theoretical average.
I see you use S5 as a broker so you can ask them about it....
Actually, I shared most of the ideas above, including TPIs and OCTANE with FT71 and Stage 5 in early 2016.
Stage 5’s response was that we could work together, but with the caveat that I would have to surrender all rights to my ideas and sign an NDA. In return I would receive only a small percentage of any revenue generated by my ideas.
I did not want to surrender my right to innovate regardless of remuneration, so we failed to come to agreement. Subsequently Stage 5 indicated they do not wish to work with me at all on software development. This saddens me, as my intent, both as part of the Stage 5 client community and FIO community is to participate in a manner that is helpful to all. As Big Mike says, pay it forward. I think I’ve demonstrated that with BaBAR, and plan to continue to operate this way.
I certainly credit FT71 with pointing out the value of understanding and monitoring one’s theoretical average while in a trade. In fact this is hard to avoid if one looks at his webinars archived on this site and elsewhere. I’d go as far as saying it’s a pillar of his trading technique. But it’s hardly a concept he invented. If one looks around, perhaps Googling “theoretical average in trading,” there are many entries. Also, if you look above, I openly share a lot more ideas than just the current theoretical average. I discuss graphing how the execution of limit orders might change the TA in the future, as well as a bunch of things not directly related to TA at all.
Recently, Stage 5 added a graphical current theoretical average indicator to their version of Bookmap. I’m not going to speculate how my communications in 2016 might have affected their development process, but as this indicator is to date free of charge, I am reassured that in any case I made the right decision, as small X 0 still equals 0.
Let me be clear that despite my disappointment, I am still glad to call myself a Stage 5 client, and as far as I’m aware, Stage 5 is still interested in doing business with me. In fact, beyond my pledge to only trade CME contracts with Stage 5, in his most recent email to me FT71 suggested that if I go the commercial route with some of my ideas, Stage 5 would certainly consider working with me to resell what I might come up with.
But really, that is not my goal right now. I want to improve the technology available to me and other traders by a quantum leap, regardless of platform. Going back to the Olympic analogy, specifically skiing, it seems to me that current platforms will tell you in painstaking detail all about the mountain [market] but fail to say more than a whisper about the skier. [trader] I think we can do a lot better at showing how the skier interacted with the mountain.
Frankly, fixing this is not rocket science. Once the trader’s orders, positions, equity, etc. are available to indicator developers, they can come up with all manner of ways of graphically displaying this information, I expect including ways I haven’t considered yet.
NT8 seems to have made this possible, regardless if it was done by design or accident. I expect [hope actually] that other platform developers will see what I’m suggesting, make similar changes to their platforms and share these changes openly so TPIs can be developed in a straightforward manner. We will see.
Separately, as an aside you mentioned cumulative delta. That was the point of the toolset Gomi created for NT7 a number of years ago.
Hi Bob, are you planning to share some of your ideas with us? I find this one of the best threads i have read. Hope you can keep adding more entries
Regards, David
Fascinating posts. Totally agree about the lack of measurements of the trader. I'm with Stage 5 and it seems like they have a lot of data now on that information. But the interface is difficult, and it's hard to decide exactly what to do with that information. I purchased Edgewonk journal which had more depth it seems, but it's not real time. If there was a way to integrate Edgewonk into real time trades, I think it would emulate some of the things you're trying to do.
Looking forward to more information on this space.
