Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I apologize to interject here but since we are both Online I wanted to get better understanding here as I'm rather slow in these math things.
What is your basic strategy that you are testing with data?
On first page you discussed about couple of things and mentioned about "money management" as key. Where are we going with this? Are we testing random bets? Or are you considering all bets as random? If yes, why?
Is there anything like specific objective here? Just data testing can be quite misleading and can be manipulated in representing lot of things. Rather than that is it not better to test specified strategy details across sample data? And from the amount of data you are testing, it seems there is not any time constraint on strategy of trading, or tbh any strategy.
Just curiosity, I've subbed the thread, since I can't really do lot of what can I want to get understanding of things here, sorry for silly questions
Can you help answer these questions from other members on NexusFi?
the money management part has to do about when you are making your purchase/bet...
its based on the idea that the market is random (which it sort of is, lets say its a coin toss with a unbalanced coin which can change its balance as its tossed... however, whatever skew in balance it gets cant be determined)
It was trying to make the larger point that what is going on isnt JUST about guessing moves well/better
that the system is two things, the bet and the outcome, not just the outcome.
Most beginner day trades and people are all focused mostly on the outcome
they do NOT focus on the odds of the outcome and how the bet structure changes it even if its 50:50
The point of a Monte Carlo simulation is the difference between theory and practice..
THTHTHTHTHTHTHTHTHTHTHTHTH
and
TTTTTTTTTTTTTHHHHHHHHHHHHH
and
THHTHHTTTHTTTHHHHTTTHTHHHT
All of the above would accurately be described as 50:50
But only the last one fits more closely to what actual coin toss would be
Your ability to change the prediction outcome of the stock your locking at is limited
however your ability to change the dynamic and ratio of the way the bet is placed is in your control!!
so a person with 50:50 ability can get ahead... IF the win lose ratio is asymmetrical in their favor
however, if BOTH are in the favor of the better... ie. they do 70:30 in choosing AND manage the asymmetry
they can become very profitable..
There is nothing new in the above... its what you would learn working for a fund or in the business
AS to the neural nets..
that is an attempt to find moments in which making a choice favors one outcome over another
we know that can be done given that the "trend is your friend" (until it isnt a trend that way any more)
the trend is the bias in the coin toss i mentioned above
Is there anything like specific objective here? Just data testing can be quite misleading and can be manipulated in representing lot of things. Rather than that is it not better to test specified strategy details across sample data? And from the amount of data you are testing, it seems there is not any time constraint on strategy of trading, or tbh any strategy.
well.. that depends... it depends on whether you want to make up these strategies and then test them endlessly with no way of knowing your efforts are not wasted... OR if you want to black box them and let machine learning ability to figure out what is important and what isn't important and do that kind of work for you
but the advantage of AI is you dont need to know the procedural methods or instructions to teach a neural net to accomplish a task or goal. in this latter way, it all depends on what your training the net to do and if its possible..
most efforts to train nets try to get them to act as oracles... ie. tell the exact price later..
they do surprisingly well which is why they are being used by large companies to trade or inform
however.. that all depends on whether you have a large enough dataset...
and what kinds of questions you are asking to train the system to be expert in..
does this help?
feel free to ask... (but you cant ask for the essence of the secret recipe)
Just curiosity, I've subbed the thread, since I can't really do lot of what can I want to get understanding of things here, sorry for silly questions
That is a very interesting chart...
could you clue me in on it?
basically i am looking to make several neural nets to create a consensus system
each trained to some different point, each trained using a different net type and structure
looking to get a decent short term edge..
however, nothing will help when things like this corona virus crap hits
its a blood bath out there... and in the larger picture, its not that bad
totally blown out of proportion... its not the red death...
Its a chart from book called "Fortunes Formula- The Untold Story of the Scientific Betting System That Beat the Casinos and Wall Street"
Its not necessarily detailed enough but is in fact light read, I guess there was a desire to expand target audience.
I do suggest to read through though, you might find more clues than I can.
I guess, this corona virus is one of those black-swans, I wont comment on its deadliness though. There is steady 1% rise of infected everyday for last couple of days.
I guess, this corona virus is one of those black-swans, I wont comment on its deadliness though. There is steady 1% rise of infected everyday for last couple of days.
The influenza pandemic of 1918-1919 killed more people than the Great War, known today as World War I (WWI), at somewhere between 20 and 40 million people. It has been cited as the most devastating epidemic in recorded world history. More people died of influenza in a single year than in four-years of the Black Death Bubonic Plague from 1347 to 1351. Known as "Spanish Flu" or "La Grippe" the influenza of 1918-1919 was a global disaster.
The effect of the influenza epidemic was so severe that the average life span in the US was depressed by 10 years. The influenza virus had a profound virulence, with a mortality rate at 2.5% compared to the previous influenza epidemics, which were less than 0.1%. The death rate for 15 to 34-year-olds of influenza and pneumonia were 20 times higher in 1918 than in previous years (Taubenberger). People were struck with illness on the street and died rapid deaths. One anectode shared of 1918 was of four women playing bridge together late into the night. Overnight, three of the women died from influenza (Hoagg). Others told stories of people on their way to work suddenly developing the flu and dying within hours (Henig). One physician writes that patients with seemingly ordinary influenza would rapidly "develop the most viscous type of pneumonia that has ever been seen" and later when cyanosis appeared in the patients, "it is simply a struggle for air until they suffocate," (Grist, 1979). Another physician recalls that the influenza patients "died struggling to clear their airways of a blood-tinged froth that sometimes gushed from their nose and mouth," (Starr, 1976). The physicians of the time were helpless against this powerful agent of influenza. In 1918 children would skip rope to the rhyme (Crawford):
I had a little bird,
Its name was Enza.
I opened the window,
And in-flu-enza.
Well... now that i finished a NN challenge and succeeded, i can kind of sort of get back to this...
first... lets show what we as investors are up against
these figures are for a two day skip... ie. Wednesday close to Thursday close
my database has 1,048,575 samples in this series...
first i converted the results into a percentage...
then i converted the values to integers to simplify things
then i boxed them in categories...
if the percentage is less than 2% assign a 0
if the percentage is less than 3% assign a 1
if the percentage is less than 5% assign a 2
if the percentage is less than 8% assign a 3
if the percentage is less than 13% assign a 4
if the percentage is less than 21% assign a 5
if the percentage is less than 34% assign a 6
if the percentage is less than 55% assign a 7
if the percentage is less than 89% assign a 8
if the percentage is greater than 88% assign a 9
The hopes were to put the data into quantile.
Quantile: each of any set of values of a variate which divide a frequency distribution into equal groups, each containing the same fraction of the total population.
the idea was to try to get the whole to break down into more equal groupings than percentages
knowing that the group of moves that were more than 89% in up or down direction, were small compared to those moves that were 2% or less...
but did it work? nope...
Remember, i have over 1 million samples (in this set) - so here are the results...
Out of one million, over 700,000 entries fall between -2% and +2%
Lets just say that i am going to have to "re-balance" these numbers otherwise a neural net guessing every number being between -2% and +2% would be right most of the time, without actually computing anything!!!!!
for those who trade, this will give an idea of what your up against in terms of moving the needle..
The quantile divisions were wrong...
but in the effort to secure 'right' ones i made a discovery, that others discovered..
that price moves follow a Laplace Double exponential distribution...
To quote another (name wasnt included so i dont know who):
“why should stock market trading, stock prices, stock indexes lead, after logarithmic transformation and first differencing to the Laplace distribution?”
almost balanced... though if one looks carefully...
i have not checked if these price moves happen as a fractal...
but i would bet they do... meaning whether you check one day, or many days, your going to get the same outcome even with different numbers... the same 'shape'..
this means predicting will be very difficult.. at least across the whole..
Ok... with the easy training of (in this case) a tabular version of the Neural Net not doing well (it was abysmal)
the iterative process continues.... first you try to remove data, then you decide to normalize it yourself...
well, one of the most GLARING problems in the structure i had was the asymmetry of the different outcomes.
that is... a zero movement in price, had over 21000 samples... that being the center of the orange spike in the prior posts graph. as you can see... as the values deviated from zero, there were fewer of them
what this is basically indicating in simpler terms is that the farther you get from zero, the more unlikely statistically the move... there are many many times the number of 2 penny moves compared to one dollar moves...
so there were about 13,000 one penny moves, and 21000 zero moves... thats close to a 2:1 ratio
and that can make a net go sour... if you go accross the whole distribution, the net just cant learn
there are ways of handling this... one thing is to filter the data so that if there are 6258 0.30 cent moves
then just chuck away the number of records in penny moves so that the different moves have similar numbers
alternatively... if your happy with the idea of grouping, you can use quintiles to put several values together
personally, i went this route since it would let the net have more data, and that my goal (unlike most others) is not to get an exact value as output... i would be happy with a good enough value...
this has certain benefits... one is that rather than try to hit a value to the 100th of a number 00.00
it now is going to try to predict a range...
for those who want a bit more understanding, here is a small sample..
in the chart above, each number on the far right indicates the quintile the values all the way to the left belong in
the large integers in the middle are the number of values in the data set.
0 has 21000 values..
a penny move and a 2 cent move combined is 26,000
a 3 cent and 4 cent move combined is 25,000
as the change in value goes higher, there are fewer samples and each group gets larger...
the 1 cent and 2 cent could be combined..
but by the time you get to 13 cents, you need three values to establish the 6th quintile
by the time you get over a dollar, it could take dozens...
personally? in my looking for value from the Neural Net, i would be happy with a solid prediction that the price will be .13, .14, or .15 i do NOT need to know the exact value to earn... in fact, for me, knowing its in the 7th quntile i can set my sell price to the top of the 6th and be happy with that...
wouldn't you?
this first stab of using this, may or may not work... it might not solve the problem
it may need more kinds of such divisions... like avg stock price broken into similar quintiles
as there are a lot more companies whose price is between 10-13 dollars than 100 and 130..
also, the higher the value of the stock, the more likely that it may move a larger amount...
so this could be another way of working the problem...
