Trading EUR/USD just now I made a few bucks using the wisdom I gleaned from the set of the jaw of some trader I saw on TV sometime ago--"most of the time a range trade does not break out." In fact the range did not break out. In a previous life I would have bet price went lower.

This perspective assists one of my past endeavours--to get to the bottom of "since I'm losing all the time should I do the opposite of what I do, and how do I know what the opposite is?"

The answer is: practice. Try conceiving the opposite of what you think and trading that. This would be one instance where paper trading saves you money.

bnichols 

Interestingly after all that I got slammed today by exactly this--the rise in EUR/USD in the 45 minutes prior to the close today when price made a ~ 50 pip beeline for the floor pivot after trading downward most of the day. I was short at the time, not an issue usually since I know how to recover from such a situation--just have to be quick. This time I was groggy (party with family friend last night :-/) and simply did not react, made every rookie mistake in the book, immediately quit trading for the day.

Mind if I call you out again? Are you using floor pivots on the cash instrument? Are you sure that works, especially when the difference between the future and the cash is large?

Hope the party was fun.

bnichols 

In still other news, stumbled across Mahalanobis distance as a better alternative to the pseudo Euclidean radius function I've been using to create the probability density function.

Sometimes what you say goes over my head by about a mile, i.e. you write stuff and I think "how come he's worried about that when all I can manage to worry about is whether the last candle closes up or down?" Anyway, you mention probability density functions a lot. Can you describe in one concise sentence what a probability density function is?

bnichols 

As an aside--if the "last edited" footnote at the bottom of each post continues to give me 1 minute to update a post I'm going to quit this forum in a couple of posts. Drives me freaking nuts. Not sure why.

That is one of the compromises of public forums. It's so you can't go back and edit your posts when you're in mid-argument with someone, at least not without leaving evidence. I think it's designed to cut down on trolling.

Adamus 

Mind if I call you out again? Are you using floor pivots on the cash instrument? Are you sure that works, especially when the difference between the future and the cash is large? Hope the party was fun.

I trade spot exclusively. I tried keeping a futures window open for comparison but was surprised to discover futures hath mysteries unique unto itself--totally distracting. So the floor pivot refers to cash only. Otherwise like wearing 2 watches--never know what time it is.

Party was great! No arrests this time.

Adamus 

Sometimes what you say goes over my head by about a mile, i.e. you write stuff and I think "how come he's worried about that when all I can manage to worry about is whether the last candle closes up or down?" Anyway, you mention probability density functions a lot. Can you describe in one concise sentence what a probability density function is?

Think of the mice that have infested my house this winter. Collectively the population is large, but the number varies from room to room, highest density overall in the vicinity of the kitchen. The probability therefore that a mouse will wander into a trap under the kitchen table is greater than the probability a mouse will be caught under the couch in the living room and so far experiment verifies that. I suppose that's several sentences.

Regarding the learning math topics I discuss, may seem esoteric if one hasn't spent the better part of one's life puzzling over them I guess. On the other hand accounting principles that seem transparent to my wife are totally opaque to me--e.g., how to balance a cheque book--never studied them and shied away when given the opportunity. Somewhat like the intricacies of taking a bus--wouldn't hazard that.

Edited to add: Seriously though--I don't think I'd ever heard of Mahalanobis before last night--have to admit the name rolls off the tongue--and never realized I needed his metric until this moment, and that's a month after I was constructing the density functions, pondering what was the simplest way to record the distance of a test vector from the centroid of non-circular distribution. Reading the description of the metric it was obvious he had precisely this problem (of cluster analysis) in mind. BTW--I try to stick to jargon that gives a minimum 500,000 hits on Google

Adamus 

That is one of the compromises of public forums. It's so you can't go back and edit your posts when you're in mid-argument with someone, at least not without leaving evidence. I think it's designed to cut down on trolling.

You read all the way to the bottom of the post! I'm not used to arguing on this forum but would look forward to it--all I have is Forex charts for company. I've taken what you said to heart and deleted those sentences. Fit of hysteria that will pass after a nap, no doubt.

Since the last post I've been trading short time frame spot EUR/USD manually on a small ($6000) account with mixed results--more or less break even--and working non-stop on the bot.

The good news is I have little or no fear anymore of blowing out the account, but having overcome so many obstacles have yet to overcome what I assumed would be a short lived newbie problem; namely, after a relatively large number of successful trades losing everything and more in a few missteps. As usual, resolve returns once our backs are against the wall. This survival instinct may be our problem (we can't function unless we're in crisis) or it may be the key to our salvation.

The issue with manual trading continues to be that I drift off mentally and yet continue to trade. I suspect this is mainly because I'm still focused on gaining experience (to put into the bot, I tell myself) and less on making money (& keeping it), because it's becoming clear the losses occur on low probability trades (50% or less chance price will move in the "desired" direction, when in fact what we desire has nothing to do with it).

If price is not moving, momentum gone to zero, especially when we expect price will not move for some time to come, let's not anticipate a breakout on no news.

In other words while I know the difference between trading and gambling, when the market slows I have an inclination to gamble, whatever the underlying psychological cause. The market is not a casino and it is not influenced by any move I may make, and yet when the action dies I start to behave as if it were. Having written that down at long last I may begin to deal with it.

Regarding the bot, I've taken some time off from straight implementation of the 2D probability based strategy to study how it can learn, which turns out to be more involved than I first imagined. (Reminds me of the observation that if at the outset we had any real idea how difficult a given course of action might prove we would never take the first step )

The approach I took to bot learning ("adaptation") was borrowed from neural network back propagation, which eventually led to wondering if the inputs themselves should be weighted. This notion of weighting inputs probably sprang from a "eureka moment" in which I realized the buy/sell signals being generated by a prototype strategy based on the buy/sell clusters were more often than not exactly backwards. In other words, it finally dawned on me I was mixing (confusing) short/long signals with buy/sell signals respectively--the probability density based system was telling me buy/sell signals are rare events that emerge from short/long conditions, respectively, and that overall the likelihood represented by the e.g. buy cluster is that price is in a downtrend, not that it is a reason to buy.

In still other words, all the 2D PD system so far is capable of is reminding us when trading a given time frame one buys only after price has declined "enough" (objectively speaking), and sells only after price has advanced "enough". What any 1st grader knows. All the probability density clusters identify is short/long conditions from which buy/sell signals emerge.

Finally, since the clusters have (essentially) been low pass filtered (smudged) there may be no effective difference between a given point (initially representing an isolated vector representing a buy condition, say) in the "buy cluster" and neighbouring vectors representing "remain short conditions".

Believing we could perhaps "fix this" by adjusting the weights (or, the "meaning') of the 2D probability density functions input to the system I wrote a number of neural-net based models that finally involved individual parameter (or "feature") vectors. We might recall the previous 2D probability density functions are functions of 2 feature vectors as ordinate & abscissa, so latest work perhaps amounts to 1D buy/sell clustering of an individual parameter (think of a bimodal line graph of a distribution), although I haven't yet clustered the 1D data.

IMO this step uncomplicates the strategy, since it's a more direct representation of the Top Dog system our system attempts to model.

The main reason I haven't spent more time on theory (e.g. 1D clustering) is I ran into a technical issue that's taken a few days to work around. The issue involves deserialization in C# of objects stored as binary files on disk by DLLs referenced by NinjaTrader. To make a long story short--so far it appears NT absolutely refuses to deserialize objects in DLLs referred to from strategies.

Regarding deserializing objects in external DLLs loaded by NT, I wouldn't mind advice at this point, since I've tried everything I know, bearing in mind I know more than a little. "Unable to load assembly" error in this case has been traced to the attempt to deserialize the neural network previously trained in an application outside of NT. Serialized objects helpfully contain a reference to the assembly (in this case, AForge.Neuro) that serialized them and even though test apps deserialize the net without complaint, NT refuses to follow. The usual cause of an "Unable to load assembly" in this situation results from mismatched write/read DLL's, and the usual solution is to write an overriding binary formatter binder to force the object to load the currently defined object. Unfortunately NT appears to shoot itself in the head prior to exposing this option.

That said, I wrote a method to store the trained network as a text file and built the interface to NT just in time for the long weekend. The good news is the strategy loads without incident-especially none of the weirder output I've seen from NT on this project. The bad news it's the long weekend and price is absolutely flat. Not surprisingly all the bot returns is "hold" on all time frames.

Note to self--bot should be saying "Flat".

"NN.cs.zip" is the Visual 2010 code to implement the current neural net, assuming you've already loaded AForge.net.

Well done with the work-around - I expect you will or have already come to the same conclusion that I have - that NT7 is good for strategies and automated trading - but not that good.

Adamus 

Well done with the work-around - I expect you will or have already come to the same conclusion that I have - that NT7 is good for strategies and automated trading - but not that good.

Have to admit I haven't had this much trouble with NT since the early days, when it finally transpired my anti-virus was causing NT to abend with all kinds of ridiculous errors. Unfortunately it's been my experience there's no point in contacting NT support about issues of this sort, except in the hope another user lurking on the site has been through the same thing and can offer a solution.

This morning, having spent all weekend getting rid of the what appeared to b causing the trouble (replacing .Net binary deserialization of neural net weights in the DLL with boiler plate text file load) NT is once again whining that it can't find the right version of AForge.Neuro.dll. Needless to say, NT has been restarted, reloaded, reinstalled--reinitialized in every conceivable way--and is being provided the correct AForge DLL version ; test programs using the custom DLL and the same AForge DLL work flawlessly.

Not quite at the hair tearing stage--have beaten NT before and will do it again--but right now to paraphrase that old cartoon, "Wish NT would give me the same answer twice"

Update--as many times before defanged the strategy by removing all references to the custom DLL, removed it from strategy references, wiped NT of all related assemblies, restarted NT and connected to the broker, ran the strat with no errors, quit NT, reinstalled the AForge DLL (and every DLL in the Visual Studio Release directory) in the NT custom directory, restarted NT, edited the strat to use the custom DLL and added the custom & AForge.Neuro DLLs to the strat references, recompiled and restarted the strat with fingers crossed and bated breath.

Lo and behold, strat runs with no errors. Remains to be seen if it's profitable--the whole purpose of this latest battle with NT.

Not sure exactly how NT gets its DLL knickers in knot but there has to be a pattern here somewhere.

While this strat is intended simply to test a hypothesis about fuzzy learning it is still capable of generating buy/sell orders, presumably with some sort of edge. If that turns out to be the case (i.e, if it performs better than a coin toss) I'll post the code.

Just a quick update before I hit the sack (up for 24 hours or so--apologies if the following doesn't scan)--system development bogged down running data through neural nets since the last post with not much to show for it--certainly nothing that might be profitable, code based on AForge.Neuro still being modified.

In the meantime I harnessed my real money manual trading to a slightly modded version of the stock NT SimpleMA strategy running on my paper account in the background on 1800 tick EUR/USD bars. The 2 moving average periods and stop loss were optimized on year-to-date data (26 bars, 60 bars and 24 pips, respectively). Backtesting suggests it should be marginally profitable and so far forward testing indicates that it may be (see illustrations below for details). It is not robust in the sense that results vary with the the selected bar (time frame) and it has not been tested on other instruments.

SimpleMA IMO is basically a momentum system, not much different than trading MACD (also based on 2 moving averages), buying and selling what would amount to MACD zero crossings (MACD(26,60,30) printed at the bottom of the chart in the illustration below for reference). The rationale is to put my interpretation of price action in some sort of context, which I find improves my objectivity & reduces stress, resulting in better entries and exits. The context here is the strategy's opinion of what price is going to do, whether or not the direction is clear to me and whether or not I agree with its opinion. In any event, the strategy is not sophisticated and has a rather significant drawdown partly because it is configured to produce a profit on average, whereas I trade specific events. If I choose to enter a real money trade based on its signals I can hedge my bets by exiting before it does if, in my opinion, it looks like the trade is ending sooner than it is geared to respond in a given case.

Also, if I enter a trade based on its signals and the trade goes south I can blame the strategy

At the moment the strat and the real (short) trade are up about 60 pips. Will close the real trade before retiring but leave the strat running on the paper account.

Figure 1. Current EUR/USD 1800 tick chart showing strat behaviour, 2 moving averages, trades at crossovers, and corresponding MACD below


Figure 2. Backtest result for the raw strategy. Real money manual trading results so far tend to be better because losses are cut early.


Figure 3. Cumulative PnL for raw strat

Since the last post I've been trading the EUR/USD macro style (the way I started years ago, on political and economic developments) rather than based on short time frame technicals (the subject of this journal) simply for income, with a relatively wide stop and large multi-tiered targets. Vacation season after all is fast approaching.

I'm using NT at the moment because it's acquiring data I want to analyse, and except for a glitch last night (subject of a rant in the "NT Truth" thread ) so far so good. After 2 of 3 targets hit, at this point have to decide whether to stick to my guns, assuming the consolidation below (1.28) 50 during N American market hours will eventually break down in the Asian session, or take the money and run.

EDITED TO ADD: BTW, I will close the trade before end of day and reopen if warranted, because it's my ongoing experience GTC Forex orders don't transit end-of-day well--much wackiness ensues (namely, some hours later the entry point jumps a random amount in a random direction, throwing NT for a loop, even though IB accounting seems to come up with the correct P&L when the trade eventually is closed). No doubt someone more wise to these things can explain it but for now, even though I suspect it's IB's fault I'll restart both NT & TWS. As my dad used to say--"If I paddle you all I'll get the right one."




If price does hit the 3rd target then in the absence of good news from Europe expect EUR/USD to go much lower (200 pips?)

Regarding system development I finally resorted to R (open source version of S--the stats analysis platform) to research ways to improve neural net performance--in fact the performance of any data pre-processor. Writing modules for NT may be interesting but debugging is not as much fun as it used to be.

Mainly I've been using Empirical Mode Decomposition (EMD) to reduce various series, including indicator outputs, to their Intrinsic Mode Functions (IMFs), applying transformations (including prediction) to the IMFs and then summing to reconstruct the new time series. The hypothesis is IMFs as inputs, rather than the noisy series from which they're derived, may allow individualized model parameterizations and hence more robust predictions. The hypothesis is being tested initially by comparing forecasts produced by standard GARCH and Arima models applied to both raw data and the derived functions.

To illustrate the concept, here is a test decomposition run last Friday on the request of a colleague on the last 20 years of daily close prices for the TSX (Canadian stock market index).

Bear in mind such models + the price of a latte will buy a cup of coffee :-/

The first pic shows the original record as the reconstruction of the first IMF and 1st residual. Remaining pics show successive IMFs and their "optimum" Arima-based 60 day forecasts. The final pic in the series shows the TSX forecast constructed from intermediate results, 85% and 95% confidence intervals indicated by red and yellow shading, respectively.

NOTE: the period of the dominant frequency (days) shown at the bottom of IMFs 7 and up is wrong, possibly an aliasing artifact due to the quick & dirty & independent spectral method used to determine the period. This does not affect results.

Original data + 1st IMF and residual


1st IMF + prediction


2nd IMF + prediction


3rd IMF + prediction


4th IMF + prediction


5th IMF + prediction


6th IMF + prediction


7th IMF + prediction


8th IMF + prediction


Residue + prediction


60 day TSX Forecast constructed from IMF predictions by summation:



While wonderful to look at, as expected the Q-Q plot from a GARCH analysis on TSX returns (1st difference of log data) shows that TSX data density deviates from normal (familiar S shaped curve), which means among other things we should be prepared to predict variance as a function of time as well for raw price data:

Q-Q plot from GARCH(1,1) analysis of TSX returns:

Short spot EUR/USD since last night (1.5 contracts). After visiting PP earlier now appears to have cleared (what I pegged as) support at 1.2815. Targets ~ 1.2745 and 1.2690 (Gann line corresponding to Murrey Math 0/8, "Ultimate S/R" on H1 chart)

Hour chart:


Edited to Add: Half hour before the close of the N. American session 1st target hit (changed to 1/2 contract from 1 contract initially) at 85 pips. Things remaining equal will exit the rest of the trade in about 20 minutes (1 contract at ~ 100 pips or so)

600 Tick chart: