if I recall correctly from Big Mikes cutom development platform thread, he reviewed the use of H5 versus MySql with Toku engine and found that Toku was far more performant. But would have to check.
Currently I am just using MySql with the default engine which is good enough for me as I am only just collecting daily data. But when I start to think about collecting lower level data I will crtainly investigate different storage technologies.
I would certainly be interested in discussing concepts and ideas. Given we all have various implementations snd are at differing stages, sharing code might too difficult to align.
The following user says Thank You to diverdan for this post:
Regarding alternative storage technologies - I started a thread exploring one of such possibilities: https://futures.io/platforms-indicators/35558-time-series-database-tick-trade-data.html
My logic behind this choice is that a database designed with time series in mind should provide the best performance for tick data. Didn't find anyone with experience in these on this forum - so I will have to wait for some free time on my hands to experiment.
The following user says Thank You to gregid for this post:
I guess for me the bottleneck is between my ears anyway so pytables is perfectly fine speed wise. I just like that you can store things like a dictionary and be done with it.
If performance really became an issue I would probably start thinking of using something other than python too.
I think there is an interesting intellectual challenge with storing large amounts of tick data in and of itself but there is also a time decay to it's usefulness. If you have 10 years of tick data are you ever going to go back and look at all the ticks on 5/1/2006 between 1:00pm and 1:15pm? All that work makes no sense if you just end up re-sampling to 5 minute bars anyway.
The following 2 users say Thank You to Darthtrader4beta for this post:
If you are testing level 1 or level 2 tick data in python. My biggest question is how are you going to process the backtest or any optimization in parallel? Using only a single core is going to be highly inefficient. You must be able to deal with the GIL.
I think that storage wise. You can try mongoDB for non relational database. It is very fast at writing but slow on the query side. Although due to its structure you can save a lot of meta data, news events or other info along with the prices. But most people will probably want to go with a relational or SQL varient.
Hi, first post ... I saw my platform github page having been visited from this thread.
I am the author of backtrader and really doubt anyone has experience with my platform since I only decided to put a link in reddit a couple of days ago, motivated by my internal belief the platform (and associated API) is stable enough for public usage and open to suggestions (the project started on some boring evenings after long skiing sessions at the beginning of 2015)
The motivation to write it, as pointed out by a message in this same thread, was to have something I would have full control of and therefore understand what was going on, whilst also giving me (and later users) "unparalleled" of use (left to the users to judge that) I had toyed with PyAlgoTrade (CANNOT POST LINKS YET) which I found useful but too cumbersome for my own taste.
Being previously unaware of "bt" (will get added to my README.rst) I see that many choices have been the same at least when it comes down to the naming of methods ("run", "plot") but I believe the platforms differ on the approach.
Looking at "bt" Examples section (CANNOT POST LINKS YET) I see that for example the Moving Average is taken from "Pandas" and the approach is kind of a "filter" and execute. Had I to quickly say something I would put "bt" in a more "functional" realm than backtrader.
Without trying to compare the platforms (which I think doesn't make much sense) I can quickly comment on what backtrader can offer to anyone. And like I said ... development is open to suggestions.
But first ... "bt" (and PyAlgoTrade too) seem to excel in statistics. This is a weak point in backtrader, simply because it is a very weak point in my own trading too. A lot of room for improvement there.
Data type agnostic and timeframe agnostic (unless you mix/operate on the original timeframes)
Mixing different timeframes
Data Resampling Data Replaying
Multiple strategies simultaneously
An already long list (and will grow) list of Indicators
A broker simulation which I deem to be very realistic
Commission schemes for stocks, futures (whether any can be applied to Forex ... it's not my cup of tea)
Vectorized (in the form of single for loops) and step by step execution modes
Strategy optimization (single core ... but multicore is on my head)
Note: PyAlgoTrade can use multiple cores and multiple MACHINES for optimization.
Data loading is now restricted to "csv" files but I see no problem in creating a Pandas adapter (I have simply never used Pandas before)
One of my major goals was to make development of indicators (and therefore trading ideas) look like a breeze. Following the publication in reddit and with it having gotten some feedback I went for a blog where I will try to describe this ease of use (I have put a first example about how the "Trix" indicator is developed ... just follow the link to the website in the github page)
Having said much I am open to answer any question and take any suggestions which may make the platform useful to anyone besides myself.
The following 5 users say Thank You to mementum for this post: