Cory, you PM's me for a snap of the COT system/indicator. Here it is attached. There are two reasons I didn't follow through with it.
1) for reasons I have yet to determine, the COT will not always plot even though it's lifted from a continuously updating file from CSI. It just stops at some point and I don't know why. Maybe because it is updated every 2 weeks and that confuses something somewhere.
2) The oscillator from the COT data required optimization of wiggly lines, something I tend to avoid. However, there is clearly some validity to the approach. It was combined with a trend-following algorithm although I did test with nothing except COT and occasionally got some excellent results, but again with,imo, too much optimization. If you find this of interest I am happy to paste in the coding, although possibly I already uploaded this into the Tradestation downloads section, not quite sure.
In fact, I just pasted it in down below. There is quite a bit of stuff in there for writing out to SS files, but hopefully you can follow the main coding which I have not looked at in 4 years.
It may not be easy to see, but my EOD provider (CSI) makes it easy for me to plot the COT as an oscillator and in many instruments you can see clear patterns of accumulation and distribution by the main commercials. I think part of the problem nowadays is that there are many other major players whose positions are not included so the information may be seriously incomplete. That was not the case ten years ago. I don't think investment bank positions in Crude, for example, show up in the commercials numbers (though perhaps I am wrong there). Also ETF funds in ags heding with futures.
The GLD fund is the largest repository of gold in the world, or something like that, but if they mess around with futures (which I think they do not very much), they would not be 'commercials' I don't think. Or the four main firms driving most of the transactions in the stock markets: is such participation reflected in the S&P COT data? I doubt it.
In any case, the whole thing has gotten quite muddled of late.
I just threw up a CL chart. Even though many funds were getting into it of late, it is still a relatively real futures market, real in the sense that it is used for hedging, not just speculation. And very healthy volume levels.
Previously you would often see clear periods where the Commercials' buying would spike up and this often happened into sell-offs and peaked at the bottom. The past year or so there has been no clear relationship but the past year has been very strange for Crude after that blow off run-up to $150 and then the plunge back down below $50.
The attached pic shows all the recent action and cuts off many examples from earlier on where they came in hard on the buy side as the oscillator plots it just as the market finished a selloff. There are only a couple of clear examples of that on this chart. But there are many markets still where such relationships come through fairly clearly. These are long-term relationships, though which play out usually over 3-6 months, not 3-6 days.
Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).
After posting this, I was over at ATOL reading Henry Liu who has been putting out top notch financial stuff for years, often way ahead of most others. Detailed too. In this latest piece on derivatives problems, he articulates on the page in the link why so many of the COT stats may no longer be as helpful as they used to be, principally because large OTC positions are not reported even though many of the commercial (and other) positions that are reported may well be placed in relation to these hidden - and often huge - positions. In theory they are all counterbalanced by the two parties involved in each transaction and therefore net neutral, but there are third, fourth and fifth parties involved so you can get situations where there are net huge short bets on a market, none of which are showing up in the current bid-offer price valuation mechanisms, nor in proportional equivalents by the commercials who are often dealing with exchange-based balances between hedgers, speculators, option makers and holders etc. In other words, there are giants playing in particular markets whose imprints we are not able to see even though they have the muscle to move them in major directions both long and short term.
In other words, the shadow markets have morphed from being small things on the fringe to major, even dominant, influences, and yet they still remain invisible. This means that we no longer have efficient markets since price does not reflect the entire sum of market participants commitment/knowledge. This is yet another factor indicating that the US's virtual hegemony in financial markets the past few decades is now redundant, if not irrelevant. The shadow players own the turf, and they are not beholden to nations, governments, shareholders or even creditors. Everything is 'off the books'.
This will not last. Because it cannot. Ten more years tops, probably only 1-2.
Great article. Pretty much lays it all out and names the players. Twisted game of money and power...Should be required reading in schools. Our financial/political illiteracy is what allows this stuff to go on. People have been fleeced since the dawn of time but the scale is getting more global.
According to Greenspan et al, the market can police itself even against fraud because it is run by honorable people who have strong incentives to protect the market from such fraud. But the issue at the hearing was more than bureaucratic turf war. It was an ideological battle with the full power of the Federal government siding with Wall Street to suppress the dutiful carrying out of the statutory mandate of a small agency to protect the general public. Born was effectively silenced by a concerted effort by top officials in the Clinton administration after she responded to a challenge from a committee member on what she was trying to protect by saying: "We're trying to protect the money of American public."
The "market" if it was a pure market probably could police itself but the government is the the one who creates bubbles by being in collusion with large business. Legislation gets created to pick winners and losers in the marketplace - distortions then occur - bubble - pop - rinse and repeat. Too big to fail.... hmmmm. I would argue my trading business is too big to fail. I need to be bailed out of all my losing trades. aaaaaaaaaah really...? [/end rant]
I have been getting into studying history quite a bit the past year or so, especially from around 1900 - 1950's, but also back to 5000 BC etc.
Seems to me that there are always highly ambitious elements in any society. These elements/motivations give society a lot of its 'oomph' factor - in both good and bad ways. But left to their own devices they will tend to over-dominate the situation since much of their impetus comes from leveraging intelligence, influence, location, cultural mores etc. into more advantage for themselves than the average. Sort of like having a good trading plan. But if everyone has a good trading plan, it won't work since one man's profit is another's loss. Not always, of course, since rising population or discovery/conquest of new territories/populations or game-changing technologies can provide temporary boosts to the communal bottom line, i.e. what is available for all, aka 'rising tide lifts all boats' type scenario. Still, there have to be checks and balances. In feudal language, this was the ongoing dance between a central Court (the monarch/Emperor principle) and the Barons (local lords). In modern terms this was for a while the State and Industry. But now that State and Industry are essentially joined, we have no natural tension between the Monarch principle (representing that which joins the interests of the go-getters (barons) and the People (ordinary folk without whom the barons would have no means to leverage themselves into a superiority). In theory the State is the equivalent of a Monarch without the bloodline/family neurosis quotient. But really what we have as a State system increasingly is rule by Soviet (literally committee), or in Chinese terms a vast sea of Mandarins (govt department employees), some of whom are appointed via elections but most of whom become a class unto themselves and nowadays most of whom travel back and forth through an endlessly revolving door shuffling between the so-called 'private' (aka corporatocracy/industry) and so-called 'public' (aka 'govmint') bureaucracies to the point where they are functionally indistinguishable. Much of Sibel Edmunds' testimony, which shines particular light on just a few particular slivers of this sort of thing, reveals the dynamic which is as old as Time, or perhaps Human Beings we should say.
The simple fact is that we do not have a functional system of checks and balances, and given the great rise of fiat-currency based mercantalism along with the industrial revolution that has swept the world the past two hundred years and now, thanks to distance-shrinking technology, is firmly entrenched throughout the world, aka 'globally', what we are entering into is the end of capitalist/mercantilist opportunism, and the beginning of entrenched baronial stagnation, as those 'on top' fight with each other for increasing relative share of the communal pie, either absorbing or eliminating rivals, further consolidating the union of state and corporate power (Mussolini's practical definition of fascism) until ultimately we will end up with a virtual totalitarianism quite similar to Orwell's 1984, in which the language used to describe the system is pretty much the opposite in tone to what it means in practice.
For traders, I think this means that in a fairly short time we might find markets start to fundamentally change behavior, either in terms of how they operate or in terms of who has access to them - most likely both.
If US dollar hegemony ends, as seems increasingly likely in the next decade or less, one has to wonder if the US futures markets will still act as international clearing exchanges, aka a valid way for wealth transfer both in terms of hedging risk and big-player manipulations that end up fleecing large sectors of an economy as they manage various different booms and busts to their advantage.
I have no idea myself, but the situation as is is clearly untenable, won't last, and it will be hard to see what's going to happen until it does. Hopefully day traders will always have some markets to participate in because thanks to the internet it is so easy so it is also an easy way to pool money from large numbers of people that well organised operations feel they will be able to fleece, and our job as daytraders is to be smarter than 90% of the other guys out there who are getting fleeced. But you never know. Markets easily available for public participation might soon be a thing of the past. One way or another, we'll find out. But it is fun to ruminate on these things.
Although it would be even more fun to ruminate on more of Gio's ideas!
The following 2 users say Thank You to cclsys for this post:
you certainly are a good writer. wish I could do that.
have to agree about gio's ideas. and it's not just his astro stuff (which is amazing btw), he has a few other "tricks" as well.
as for the markets, I don't see it as black. us markets are not more corrupt than european and certainly not more than asian markets. us dollar might have a hard time against the euro. but as far as financial markets I don't think we'll see a big shifting. at least not in our life span. I'm more worried about getting taxed to death.
well, the point about the dollar is: there is huge liquidity in the Crude, Soybean, interest rate and index markets, all of which are priced in dollars. Of course the Forex market, which is a very pure market, dwarfs all futures markets put together, and it is a fine speculative market for small or large players. But in terms of futures markets per se, esp. those based in Chicago and NYC that we tend to trade, the end of US $ hegemony might also spell the and of liquidity in those markets since pricing Soybeans and Crude in dollars might become totally redundant. So I wasn't considering the US futures markets as being more or less corrupt, just more or less relevant/dominant as medium of international exchange, i.e. with sufficient liquidity.
Frankly, I am a little baffled at the volumes in the ES emini and would like to see a breakdown of who is trading these still enormous volumes. I doubt it's mainly small traders, though if it is that is very good news for us.
Then again, one has to wonder at the humongous volumes in forex. Clearly there is still oodles of speculative froth in the world system and unless fiat currencies suddenly all collapse together (which is the way it will happen if it does), at least those markets will remain for a long time to come whether or not US$ futures remain viable/liquid.
As to corruption: I have absolutely no doubt whatsoever that the US is just about as corrupt a situation as any other, either in the past or the present. Some periods are slightly better than others, some slightly worse, but generally they remain pretty much the same from one century to the next, with occasional little hiccoughs like major war, famine, revolutions etc., most of which only last a few years. I'll try to dig up a quote from 12th century Moroccan/Muslim economist on the nature of trading. Apart from a couple of quaint notions (like regarding gold as the only ultimate measure of value), it could have been written today. Not to mention the great seashell-currency bubble of - what was it now - 7000 BC?
The following user says Thank You to cclsys for this post:
From Gunter Frank's 'ReOrient' which traces the world economic system - and Asia/Eurasia's dominant role therein - the past 5000 years:
" One of the extensions of Indian trade for centuries- indeed millen*ia - had been westward to Central Asia, Persia, Mesopo*tania, Anatolia, the Levant, Arabia, Egypt, and East Africa. Of course, analogous -and related- productive, commercial, and financial institutions had been operative there as well. Arab and Muslim trade had flowered during the European Dark Ages, and continued to do so in early modem times, even though the Arab traders themselves were sub*ject to increasing competition from both the west and the east. For instance, Ibn Khaldun, already quoted regarding merchants and com*merce from non-Arabic lands, also wrote about Muslim and other trade in the fourteenth century:
“when goods are few and rare, their prices go up. On the other hand, when . . . they will be found in large quantities, the prices will go down. . . . Commerce means the attempt to make a profit by increasing capital, through buying goods at a lower price and selling them at a higher price, whether these goods consist of slaves, grain, animals, weapons, or clothing. The accrued amount is called profit. . . . It has thus become clear that gains and profits, in their entirety or for the most part, are realized from human labour… Furthermore, God created two minerals, gold and sil*ver, as a measure of value for all capital accumulations. These the inhabitants of the world, by preference, consider treasure and property. Even if, under certain circumstances, other things are acquired, it is only for the purpose of ultimately obtaining (gold and silver). All other things are subject to market fluctuations. . . . '
Profit may come from merchandise and its use in barter; merchants can make such profit either by travelling around with
(merchandise) or by hoarding it and observing the market fluctuations that effect it. This is called commerce. . . . Commerce is a natural way of making profits. However, most of the practices and methods are tricky and designed to obtain the (profit) margin between purchase price and sale prices. This surplus makes it possible to earn profit. Therefore, the law permits cunning in commerce, since (commerce) contains an element of gambling. (Ibn Khaldun 1969: 298-300)
Ibn Khaldun, Tunisian statesman and historian 1332 - 1406
(fourteenth century, not twelfth as cited above, but still.. and this guy was in Tunisia which was a small backwater fringe area of the Muslim Empire of the time, i.e. he was not a big player in the center of things. Obviously, not much as changed in any fundamental terms; theorists in China in 500 AD could have written almost identical stuff, not to mention senior administrators in Rome, or ancient Egypt etc.)
The following 3 users say Thank You to cclsys for this post: