Welcome to NexusFi: the best trading community on the planet, with over 150,000 members Sign Up Now for Free
Genuine reviews from real traders, not fake reviews from stealth vendors
Quality education from leading professional traders
We are a friendly, helpful, and positive community
We do not tolerate rude behavior, trolling, or vendors advertising in posts
We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community. It's free for basic access, or support us by becoming an Elite Member -- see if you qualify for a discount below.
-- Big Mike, Site Administrator
(If you already have an account, login at the top of the page)
I haven't heard of streambase before but looking at the aurora project which it spawned from it appears to be just a very optimized database oriented towards realtime processing of streams.
Wikipedia says that as of Q2 2009 they had 75 customers, I take that to mean that their software is very expensive =)
One of the cardinal rules of machine learning could be summed up as 'work smart, not hard'.. if you follow that rule then you will not need something like streambase.
Can you help answer these questions from other members on NexusFi?
This line from the intro is particularly valuable:
traditional volatility measures have shown major drawbacks
since they perform their calculations for the standard deviation
in fixed times intervals, in contrast to the real life situation
where volatility should be calculated from an event based
perspective and observations made after a certain event has
occurred.
Before you start thinking about AI, first consider what you will feed the AI. The hardest part is coming up with the inputs and figuring out the pre-processing. Simply feeding in the tick data or OHLCV data ain't gonna work. My first experiment with AI was feeding in several of the traditional technical indicators into a genetic algorithm / genetic programming framework (ECJ). I found nothing of value with that approach, but perhaps others here can.
I own NeuroShell and have a lot of time & $ invested in it. I've made money with the platform, but not by using neural nets (neural nets are only a small portion of their hundreds of indicators). There are a number of things that they got right; for example they have a great genetic optimizer (which of course can be very dangerous and easily lead to overfit systems if you don't know what you're doing). But their wizard-based interface gets really tedious, especially for rules that are time-sequential in nature. You can put custom code in a DLL, and I wrote a bunch of them, but it's not a very productive development methodology.
With neural nets it's extremely easy to make overfit systems that backtest great, but perform poorly in the future. It takes a fair amount of experience to be able to build a neural net system that has a chance of surviving in the real world. I know how to test a non-AI system to build confidence that it's not overfit, but some of the techniques (like varying parameters to test sensitivity) can't really be used on neural nets. So it's hard to build confidence that your great backtesting results aren't an accident. You can test on out-of-sample data, but some percentage of overfit systems will look great on the out-of-sample data, so even that doesn't really protect you.
Then you run into another problem. Every system, even really good ones, goes through periods of losing money. When that happens, you need to really understand what kind of behavior is expected from your system, so you can either confidently continue trading it through the drawdown, or confidently stop trading it because it has gone outside its expected boundaries. I had trouble getting to the required confidence level. I built plenty of models that backtested great, performed great on out of sample data, paper traded great for a while, then totally fell apart.
And then what do you do? You can retrain the net, but then you have a completely different system, so you have to start characterizing it all over again. You can add or tweak some additional rules, but then you are probably overfitting even more.
There are people out there who claim to be getting great results with neural nets, and I don't doubt that they are. I just wasn't able to get there myself, and I found it easier to make money with systems that use non-AI techniques where I understand what they are doing and I can characterize them and and build confidence in them.
Something else to think about is there is the entire world of classification when it comes to AI as opposed to prediction. I've always thought this would be a much more worth while experiment to generate trade ideas from as opposed to trying to get the computer to sift through all this data without finding nonsense in its prediction, but worse yet nonsense that might be past your ability to see that its nonsense until it slams your bankroll.
There is also an entire machine learning course on youtube from Stanford.
I watched the first 3 and realized I was in way over my head with this and have put ML far on the backburner.
I would also think you are really cutting yourself short by thinking of doing this stuff in something other than matlab or R without a degree in the field.
Anyone done any more work on this stuff? That good paper linked to a couple of posts back is no longer available, wondering if anyone can upload if they have it?
Am very interested in this stuff, and most info that I'm finding is from 2008 - 2010, so surely there are some updates and progress in this field?
There are several threads that cover this in detail from various aspects, but they are all only available to Elite Members, so you won't be able to find them or search for them. The author controls where to place the thread.
steve do have or know a simple sugeno example for java or c#?
$100 just to see some threads on theories/ideas.