IQFeed: Suggestions to filter bad ticks - Matlab, R project and Python | futures io social day trading
futures io futures trading


IQFeed: Suggestions to filter bad ticks
Updated: Views / Replies:191 / 6
Created: by phaser Attachments:1

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 100,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

 
 
Thread Tools
 

IQFeed: Suggestions to filter bad ticks

 
Singapore
 
 
Posts: 3 since Mar 2019
Thanks: 1 given, 1 received

IQFeed: Suggestions to filter bad ticks

I've been trying out IQFeed data streaming via Python and Amibroker. As IQFeed gives unfiltered data (From IQFeed's website: "...IQFeed provides a TRUE, tick-by-tick datafeed. IQFeed feed is completely unfiltered, allowing you to see EVERY TRADE..."), bad ticks are actually quite frequent.

When I streamed 1-min data into Amibroker, those bad ticks appear as very long 'tails' in certain bars that, to human beings, seem obviously wrong. An obvious way to filter would be to use some sort of standard deviation or ATR to determine if the bar's high or low is correct. But I feel that this is rather arbitrary.

I have 2 very preliminary ideas that I wish to bounce off the community. They are technically possible because IQFeed provides a lot more information than just bar-data. They also provide bid-ask spreads, tick-level trades, among other data.

Preliminary Idea #1
Using Python, I stream bars of a much finer timescale (e.g. 1sec) than what I really need (i.e. 1-min bars). At the finer timescale, trades which are out of whack with the rest can be easily identified and ignored. I will then programmatically aggregate the 1sec bars into 1minute bars.

Preliminary Idea #2
Another idea is to monitor in real-time the bid-ask spreads and when a trade appears that are way different from the bid-ask spreads, ignore that trade.


I apologize if my 2 ideas above sound hazy and lack detail - as mentioned, they're very preliminary ideas. Thus, please feel free to comment on them.

I would also be happy to hear from anyone who has other methods of filtering bad ticks from streaming data.

The following user says Thank You to phaser for this post:
 
 
Quick Summary
Quick Summary Post

Quick Summary is created and edited by users like you... Add FAQ's, Links and other Relevant Information by clicking the edit button in the lower right hand corner of this message.

 
 
south africa
 
 
Posts: 170 since Dec 2018


That sounds cool. What API are you using for this?

First, I should say real time data analysis is not my thing. I would think about first how real time you really need to be.

So much is going to depend on how this is used. A ridiculous situation for example would be if your system sees a tick of zero as a trigger to buy at a discount and places a market order. That is a different problem.

Your eye can ignore a bad tick because you don't really care about the high and low so much of the bar.

I think just taking care of this when you sample is best. If you use the median of a volume bar or something like that as your value you can ignore this 99% of the time.

 
 
Singapore
 
 
Posts: 3 since Mar 2019
Thanks: 1 given, 1 received


centaurer View Post
That sounds cool. What API are you using for this?

For Python, I use pyiqfeed to connect to the IQFeed client.



centaurer View Post
First, I should say real time data analysis is not my thing. I would think about first how real time you really need to be.

I should have mentioned that mine is a fully-automated system. So bad ticks are simply going to screw everything up. When I streamed the 1-min data directly into Amibroker, the bad ticks really showed up as very long tails (unfortunately, I don't have enough posts to include the URL of a post I wrote in another forum describing the issue).

 
 
south africa
 
 
Posts: 170 since Dec 2018

That is what I mean though , one bad tick shouldn't ruin the system. It is showing up as a tale because it is seen in a candle as a very high or low wick. You don't care because you aren't really using the wick. What is the system then doing that it cares about such a wick?

Like what is the value based on of price array[0], time now, T0 however you want to say it?

Like for me this is exactly why I like the median of whatever I am sampling. So 1 minute bar for example, the median is never going to be the outlier. The outlier will be the first or last value when sorted and tossed out. Discounting outliers is exactly the reason to use the median.

I mean this is also a reason I prefer a quantamental/mixed type approach as opposed to full automation. I want the machine to give me the buy/sell signal but I still want to place the order myself as a sanity check.

pyiqfeed looks interesting. I have been thinking about trying IQ feed and work on shorter term strategies but I still don't see myself streaming data directly into python.


Last edited by centaurer; April 21st, 2019 at 10:52 AM.
 
 
Singapore
 
 
Posts: 3 since Mar 2019
Thanks: 1 given, 1 received

I see what you mean, @centaurer. Unfortunately, the highs and lows of my bars do get factored in my ML model so I can't run away from having to deal with them.

In fact, even among common indicators like ATR, the wicks of the bar do matter too, so would I be right to say that you don't use such indicators at all? I mean, if your indicators take Hi/Lo into their computation, how does your 'median-sampling' mitigate this bad-tick issue?

On another note, which data vendor do you use? I'm trying out IQFeed due to their affordability and supposed reliability. For a flat 'low' price, they do pack quite a lot of value, from tick-level bars to bid-ask spreads, news, and company fundamentals. Of course, to tap all these information on an automated system, one has to use pyiqfeed or similar plug-ins.

P.s. I finally realize that, even with less than 5 posts, I can attach images! Below is a screenshot of 5 consecutive bad ticks (those long green tails on the right of the chart) in my minute-bar chart. This is what caused me to sit up.

Please register on futures.io to view futures trading content such as post attachment(s), image(s), and screenshot(s).

 
 
south africa
 
 
Posts: 170 since Dec 2018

I just use IB data, quandl and FRED. Keep in mind I am not trading intraday.

I mean everyone has to use some kind of calculated predictive features but I don't use the old trading indicators.

For volatility I just use the VIX.

I am sure part of the price of DTN is no one is cleaning the data for you. The past two years I have really been trying to learn more about general data science and cleaning/pre-processing data is part of the work of any data scientist.

Taking a moving average and then filtering based off some std deviation then replacing the bad tick with the moving average value is probably the way to go.

The median is just taking prices over some fixed period, putting them in order and then taking the middle value.


 



futures io > > > > IQFeed: Suggestions to filter bad ticks
Thread Tools



Upcoming Webinars and Events (4:30PM ET unless noted)
 

futures io is celebrating 10-years w/ over $18,000 in prizes!

Right now
 

$1,000 in Amazon Gift Cards being given away right now from GFF Brokers

Right now
 

$250 Amazon Gift Cards with our "Thanks Contest" challenge!

Right now
 

Show us your trading desks and win over $5,000 in prizes w/Jigsaw Trading

August
 

Webinar: Suri Duddella (TBA)

Elite only
 

Webinar: Richard Bailey (TBA)

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Trade statistics spreadsheet - suggestions? Gary Psychology and Money Management 37 October 8th, 2013 11:57 AM
Sierra Chart filter bad ticks Big Mike Sierra Chart 7 August 1st, 2012 09:34 PM
Filter bad Trades for RSI Bot jkpdon Elite Automated NinjaTrader Trading 4 February 29th, 2012 01:14 PM
Bad January, Bad First Half: A Gloomy Omen for Stocks Quick Summary News and Current Events 0 July 19th, 2010 04:00 PM


All times are GMT -4. The time now is 06:18 PM. (this page content is cached, log in for real-time version)

Copyright © 2019 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts