NexusFi: Find Your Edge


Home Menu

 





What a long, strange trip its been...


Discussion in Trading Journals

Updated
      Top Posters
    1. looks_one Artfldgr with 25 posts (23 thanks)
    2. looks_two AllSeeker with 3 posts (0 thanks)
    3. looks_3 Quick Summary with 1 posts (0 thanks)
    4. looks_4 ninjus with 1 posts (1 thanks)
    1. trending_up 4,541 views
    2. thumb_up 24 thanks given
    3. group 4 followers
    1. forum 29 posts
    2. attach_file 6 attachments




 
Search this Thread

What a long, strange trip its been...

  #11 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

Its such a bear working with large data sets... this one is pretty large.
2,616,997 rows so far, and not quite 2/3rds the way..

there is Sooooo much one can do with this much data once its prepped..
heck, this is just putting the sets together the way i want it...

how about using the raw data to classify candles against their next iterations results?
people claim lots of things about candles, and patterns.. be interesting to see if they are right
or whether it was wishful thinking on their parts..

anyway... this is about halfway through..
and there are at least four more sets to go..

going to take into the weekend probably....

thats the issue with deep learning or expert systems or ANN machine learning
whatever you want to call it, the biggest time is spent in data prepp, and learning
both let you get up and watch TV and eat and take naps...

Reply With Quote

Can you help answer these questions
from other members on NexusFi?
Better Renko Gaps
The Elite Circle
Futures True Range Report
The Elite Circle
Deepmoney LLM
Elite Quantitative GenAI/LLM
ZombieSqueeze
Platforms and Indicators
My NT8 Volume Profile Split by Asian/Euro/Open
NinjaTrader
 
Best Threads (Most Thanked)
in the last 7 days on NexusFi
Get funded firms 2023/2024 - Any recommendations or word …
60 thanks
Funded Trader platforms
43 thanks
NexusFi site changelog and issues/problem reporting
24 thanks
GFIs1 1 DAX trade per day journal
22 thanks
The Program
19 thanks
  #12 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

took ALL day...
3,744,034 rows generated in this stage..
making for a total 4,444,216 rows...

This ought to be an interesting project even if it dont work..
pytorch to the rescue...

Reply With Quote
  #13 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90


Today a good day...
yesterday i lost about $50...
today i made $676.50
cant complain..

but i can biatch that one of the stock i bought if i waited, would have earned my day trade in options a lot more
it went up 12 points... dont you hate that?

tomorrow my credit spreads will expire profitable... so cant complain on that...
AND if things work out on the ACHN shares..
they will deposite 70k into my account...
a 47 times profit!!!!

thats probably the first and last time that will happen in my life...
but here is to hoping.. (and i wont believe it till i get the actual cash in my account!!!)

either way, the stocks and options play should net another two payments of 20k each if the CVR conditions are met
i do believe one of them will be met... but unsure about the 2nd... but thats all icing i guess...

The stock data has been rolling all night long...
and it has three more passes to go through after this one finishes...
working with large datasets are a PAIN...
but i will say that they are the only way to go if your going to program Neural Nets with Pytorch..

Reply With Quote
Thanked by:
  #14 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

Well did good yesterday..
up 1160.50
today i am up..
but will wait till i have a final total given how things in life can be so changeable...

On another note...
I have to rebuild the database... (i found a new source of data)
The new one will be even bigger, and go back farther in time!!!
Thats the good news..

The bad news is how hard it is to work with that much data
I mean its REALLY hard... even though i have made myself a monster box
its not a Cray...

will update with stats later as to how many tickers and how many years...
if 37 million days was a lot before... this will be even larger...

time to put the old database to rest...


[isnt it nice that my updates are all written to myself... ]

Reply With Quote
Thanked by:
  #15 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

I just finished acquiring the data..
HUGE pool... but uneven quality given its 'adjusted prices'
what do they mean by that?

Well, from what i can tell, is they adjust the old prices downwards on splits..
either that or stock that was sold in 1970 had a zero price...

this is going to be an issue of cleaning things up AFTER i get things put into the MSSQL database
Created a new DB.. now will write the software to put the data in..
then i will compare it to the old database as many companies i have in that one, are now gone
and gone companies tend not to be kept in new databases...
but their trading data (for CNN neural nets) is still good...

then i will have to see if its possible to merge some of this data..

how much do i have?
well... 19,000 companies now... almost double what i had before..
and prices in some cases go back to 1960's

so i am unsure whether i will bother loading ALL the data..
after all, how relevant is it to see what prices and price action there was before online discount trading?
and before the normal use of HFT machines

i will PROBABLY load it all..
then i will back it up
then i will delete everything prior to a certain date..

just have to figure out what that date should be..
i would ask for ideas, but i dont think anyone is really reading my journal...


Reply With Quote
  #16 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

oy.. its as my title says... i have reached the processing limit of Python on my machine...
sort of..

what i needed was a histogram... but ever make a histogram out of 19,954,035 datapoints?
ah.. its clear instantly, isnt it...

well...what python cant do, MSSQL can...

where there is a will there is a way, with a large enough hammer, you can crack anything

Reply With Quote
  #17 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

here is the histogram of 11,000 companies since jan 2007 w prices between $10 and $500
[my other database has 19,000 companies going back to the 1960s]

as you can see...
normalizing this data would not work that well...


Doing this kind of data analysis really necessary or else what efforts are put in, are probably going to fail.

Reply With Quote
  #18 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90

This has been a pain.. but its a necessary pain...
when python failed to be able to handle the data..
MSSQL to the rescue! actually... if anyone here is doing neural nets
i would advise loading data into SQL server if you have large datasets
SQL server can handle huge amounts in record times without dragging your system

in this case, i took the original tables, sorted and got the High values and low values for each column
then built query that would report all the old fields, and then add the normalization across the whole data set
and so i would not have to do it again, i did log(x), and zscore, as well as linear range
and clipped the data to stock prices between 10 and 71 because this is the lions share of stocks over history

yes there are stocks above 71, of course... but the number of records of prices they have are dwarfed by others
between 10 and 71 represents 11,230,434 rows going back to Jan 2017 to sometime in 2019
while the number of records for stock prices above 71 is 1,481,529 - an obvious big difference..

for those curious, this is what the data looks like...

Reply With Quote
Thanked by:
  #19 (permalink)
 
ninjus's Avatar
 ninjus 
Chiang Mai Thailand
 
Experience: Beginner
Platform: Seirra Chart
Broker: Stage5 + Denali
Trading: MNQ
Frequency: Many times daily
Duration: Minutes
Posts: 735 since Jun 2017
Thanks Given: 1,223
Thanks Received: 2,336


Artfldgr View Post
just have to figure out what that date should be..
i would ask for ideas, but i dont think anyone is really reading my journal...


Just wanted to say Im reading along.

It's over my head for the most part but you sound like a smart guy and Im interested in seeing where you take things.

Cheers


Marcus Aurelius
"Not to assume it's impossible because you find it hard. But to recognize that if it's humanly possible, you can do it too"

Follow me on Twitter Visit my NexusFi Trade Journal Reply With Quote
Thanked by:
  #20 (permalink)
Artfldgr
New York + New York / USA
 
Posts: 76 since Jan 2020
Thanks Given: 2
Thanks Received: 90


Well, what do you think is over your head? while doing the actual work in detail maybe, understanding things conceptually should not be. for instance... you probably cant build a nuclear power plant, but you can understand how the most important parts work in the abstract, even down to the nuclear reactions! there have even been great videos which started using walt disneys example of filling up a gymnasium of mousetraps and ping pong balls to illustrate how neutrons can cause an unstable atom of uranium, to fall apart, and emit energy and more neutrons that create a chain reaction.

So, if you want to understand, you can just ask...
I only bite when people are insulting...
and honest curiosity is never insulting

Reply With Quote




Last Updated on March 6, 2020


© 2024 NexusFi™, s.a., All Rights Reserved.
Av Ricardo J. Alfaro, Century Tower, Panama City, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada)
All information is for educational use only and is not investment advice. There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
About Us - Contact Us - Site Rules, Acceptable Use, and Terms and Conditions - Privacy Policy - Downloads - Top
no new posts