there may be some data error in TS - futures io
futures io



there may be some data error in TS


Discussion in TradeStation

Updated
    1. trending_up 3,099 views
    2. thumb_up 13 thanks given
    3. group 3 followers
    1. forum 21 posts
    2. attach_file 9 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

there may be some data error in TS

(login for full post details)
  #1 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received

please look at the difference close price on 6/10 between the two pics below...
the first is captured on the open of 6/11.
and the second is captured on 6/12...
can TS compensate my loss due to the error?
Or what can I do??


Reply With Quote

Journal Challenge April 2021 results (now extended!):
Competing for $1800 in prizes from Jigsaw
looks_oneMaking a Living with the Microsby sstheo
(385 thanks from 39 posts)
looks_twoSalao's Journalby Salao
(85 thanks from 14 posts)
looks_3Learning to Profit - A journey in algorithms and optionsby Syntax
(59 thanks from 18 posts)
looks_4Deetee’s DAX Trading Journal (time based)by Deetee
(51 thanks from 23 posts)
looks_5Maybe a little bit different journalby Malykubo
(37 thanks from 23 posts)
 
Best Threads (Most Thanked)
in the last 7 days on futures io
The Crude Dude Oil Trading System
98 thanks
Help improve the FIO community
38 thanks
futures io site changelog and issues/problem reporting
33 thanks
Seasonal Trades
25 thanks
Spoo-nalysis ES e-mini futures S&P 500
22 thanks
 
(login for full post details)
  #2 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received


slicent View Post
please look at the difference close price on 6/10 between the two pics below...
the first is captured on the open of 6/11.
and the second is captured on 6/12...

I'm not a heavy Tradestation user and I've discovered several cases of bad data. When notified they quietly fix it behind the scenes with no notice or record there was ever an issue. I believe this is why when you rerun a backtest months or years later your can't replicate exactly. If I've found that many others must have found a lot more. As for your screenshots the bars definitely look different but without more information it's impossible to tell.

slicent View Post
can TS compensate my loss due to the error?

That's an easy question. No.

slicent View Post
Or what can I do??

Well first you'd need to verify it was bad data and not something happening you don't understand. (Like LAST vs SETTLE on a daily bar). If it is a continual problem though the only thing you can do is switch software.

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #3 (permalink)
 vmodus 
Legendary Systematic Algo Trader
Somewhere, Delaware, USA
 
Experience: Intermediate
Platform: TradeStation, MultiCharts
Broker: TS Securities, OANDA
Trading: Energies, Grains, Fixed Income
 
vmodus's Avatar
 
Posts: 1,057 since Feb 2017
Thanks: 2,292 given, 2,206 received


I don't think this is an issue. The close you see is not always the actually daily close. The daily closing price almost always changes due to settlements.

Example:
  • Market closes at 3901.5
  • Settlements occur by the exchange during market close time and the close is actually determined to be 3902
It is just the nature of the markets.

~vmodus

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to vmodus for this post:
 
(login for full post details)
  #4 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


vmodus View Post
I don't think this is an issue. The close you see is not always the actually daily close. The daily closing price almost always changes due to settlements.

Example:
  • Market closes at 3901.5
  • Settlements occur by the exchange during market close time and the close is actually determined to be 3902
It is just the nature of the markets.

~vmodus

the first pic is captured "after" the open of 6/11 and the close price on 6/10 is changed on 6/12!
the same things has happened several times and cause additional loss. But this time I have left the screen, leaving evidence.

Reply With Quote
 
(login for full post details)
  #5 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


SMCJB View Post
I'm not a heavy Tradestation user and I've discovered several cases of bad data. When notified they quietly fix it behind the scenes with no notice or record there was ever an issue. I believe this is why when you rerun a backtest months or years later your can't replicate exactly. If I've found that many others must have found a lot more. As for your screenshots the bars definitely look different but without more information it's impossible to tell.

That's an easy question. No.

Well first you'd need to verify it was bad data and not something happening you don't understand. (Like LAST vs SETTLE on a daily bar). If it is a continual problem though the only thing you can do is switch software.

After the open of 6/11, the close price of 6/10 I see is SETTLE?

the data of rithmic will be more reliable?

Reply With Quote
 
(login for full post details)
  #6 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

You understand what @vmodus is saying. The settlement price of say Crude ("CL") is determined based upon trading between 1:28 and 1:30 Central Time. The exchange doesn't close or reset for the day until 4pm Central Time. So if you look at your bar at say 3:59pm the bar is going to show the last print NOT the settlement price. At some point after 4pm Tradestation updates your previous daily bar from the last price to the settlement price. This "should" happen in the market down time, which in CL's case is between 4pm and 5pm, but in my not-extensive experience of Tradestation, this does not always happen in a timely fashion. This could well be one of those situations where that did not happen.

To avoid this problem some people don't use daily bars and instead use 1440 min bars (1 day = 24*60 = 1440 mins). This bar will have the same open, high and low as a daily bar, but will always use the last trade and not the settlement as the close. The problem with this is that the last print is not nearly as important from a volume perspective as the settlement print.

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #7 (permalink)
 vmodus 
Legendary Systematic Algo Trader
Somewhere, Delaware, USA
 
Experience: Intermediate
Platform: TradeStation, MultiCharts
Broker: TS Securities, OANDA
Trading: Energies, Grains, Fixed Income
 
vmodus's Avatar
 
Posts: 1,057 since Feb 2017
Thanks: 2,292 given, 2,206 received


SMCJB View Post
You understand what @vmodus is saying. The settlement price of say Crude ("CL") is determined based upon trading between 1:28 and 1:30 Central Time. The exchange doesn't close or reset for the day until 4pm Central Time. So if you look at your bar at say 3:59pm the bar is going to show the last print NOT the settlement price. At some point after 4pm Tradestation updates your previous daily bar from the last price to the settlement price. This "should" happen in the market down time, which in CL's case is between 4pm and 5pm, but in my not-extensive experience of Tradestation, this does not always happen in a timely fashion. This could well be one of those situations where that did not happen.

To avoid this problem some people don't use daily bars and instead use 1440 min bars (1 day = 24*60 = 1440 mins). This bar will have the same open, high and low as a daily bar, but will always use the last trade and not the settlement as the close. The problem with this is that the last print is not nearly as important from a volume perspective as the settlement print.

Using the 1440 bars (for data2) does solve that issue. There are other solutions, but SMCJB's is probably the most elegant.

~vmodus

Follow me on Twitter Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to vmodus for this post:
 
(login for full post details)
  #8 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received


vmodus View Post
Using the 1440 bars (for data2) does solve that issue. There are other solutions, but SMCJB's is probably the most elegant.

Makes me think - you could have Daily as data1, 1440 as data2 and check them to see if they are different!

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #9 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


SMCJB View Post
You understand what @vmodus is saying. The settlement price of say Crude ("CL") is determined based upon trading between 1:28 and 1:30 Central Time. The exchange doesn't close or reset for the day until 4pm Central Time. So if you look at your bar at say 3:59pm the bar is going to show the last print NOT the settlement price. At some point after 4pm Tradestation updates your previous daily bar from the last price to the settlement price. This "should" happen in the market down time, which in CL's case is between 4pm and 5pm, but in my not-extensive experience of Tradestation, this does not always happen in a timely fashion. This could well be one of those situations where that did not happen.

To avoid this problem some people don't use daily bars and instead use 1440 min bars (1 day = 24*60 = 1440 mins). This bar will have the same open, high and low as a daily bar, but will always use the last trade and not the settlement as the close. The problem with this is that the last print is not nearly as important from a volume perspective as the settlement print.

This "should" happen in the market down time...
How about other data feeds like rithmic?

Reply With Quote
 
(login for full post details)
  #10 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received



slicent View Post
This "should" happen in the market down time...
How about other data feeds like rithmic?

Don't know everybody probably does it differently. I do know Trading Technologies push the price well before the 4pm close but they are not your traditional 'bar based trading software'

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #11 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


SMCJB View Post
Don't know everybody probably does it differently. I do know Trading Technologies push the price well before the 4pm close but they are not your traditional 'bar based trading software'

I think its problem about TS's data feed.
I will change it to more reliable one...

When I try 1440min bar just I found a more data error of TS...and other problems...shock me...
It miss the bar of 7/4(In my time zone the close time of Fri. is 7/4)

2020-07-10 094749

Reply With Quote
 
(login for full post details)
  #12 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received


slicent View Post
When I try 1440min bar just I found a more data error of TS...and other problems...shock me...
It miss the bar of 7/4(In my time zone the close time of Fri. is 7/4)

Most of the data issues I found and reported involved Holidays. Some were real data issues and others were code not processing properly on Holidays.

Reply With Quote
 
(login for full post details)
  #13 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


SMCJB View Post
Most of the data issues I found and reported involved Holidays. Some were real data issues and others were code not processing properly on Holidays.

...Last Friday is not a holiday...

Reply With Quote
 
(login for full post details)
  #14 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received


slicent View Post
...Last Friday is not a holiday...

Last Friday, the 3rd of July was a US Holiday. The Holiday is the 4th of July but since that was a Saturday it's celebrated on the Friday.

I don't know what product you trade, but talking about Crude "CL"

NORMALLY
Thu 2nd. 1:28-1:30 Settlement Window.
Thu 2nd. 4pm Last Trade. Start of Maintenance Window.
Thu 2nd. 5pm Reopen. Start of Trading for Fri 3rd.
Fri 3rd. 1:28-1:30 Settlement Window.
Fri 3rd. 4pm Last Trade. Start of Maintenance Window until Sunday.

Sun 5th. 5pm Reopen. Start of Trading for Mon 6th.

LAST WEEK, JULY 4th WEEKEND
Thu 2nd. 1:28-1:30 Settlement Window.
Thu 2nd. 4pm Last Trade. Start of Maintenance Window.
Thu 2nd. 5pm Reopen. Trades done after this point count as trades for Monday 6th since there is no Friday 3rd trade day.
Fri 3rd. 12pm Last Trade. Market Halt until Sunday. No Settlement Window.

Sun 5th. 5pm Reopen. Continuation of Trading for Mon 6th.

So the problem becomes Friday 3rd was officially not a trading day, hence has no high, low, open or close, BUT there was actually trading activity for 19 hours. So theoretically Monday 6th was a 42 hour trading day instead of the normal 23 but the software struggles to handle that. You can understand the underlying reasons why the software struggles with this, but given that this is a billion dollar industry you think they would have worked out how to handle it!

Years ago the market closed on Thursday and didn't reopen to Sunday night. But with the global nature of markets and the shift from exchanges being member/local focused to shareholder/revenue focused this is no longer the case.

Reply With Quote
The following 2 users say Thank You to SMCJB for this post:
 
(login for full post details)
  #15 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received


SMCJB View Post
Last Friday, the 3rd of July was a US Holiday. The Holiday is the 4th of July but since that was a Saturday it's celebrated on the Friday.

I don't know what product you trade, but talking about Crude "CL"

NORMALLY
Thu 2nd. 1:28-1:30 Settlement Window.
Thu 2nd. 4pm Last Trade. Start of Maintenance Window.
Thu 2nd. 5pm Reopen. Start of Trading for Fri 3rd.
Fri 3rd. 1:28-1:30 Settlement Window.
Fri 3rd. 4pm Last Trade. Start of Maintenance Window until Sunday.

Sun 5th. 5pm Reopen. Start of Trading for Mon 6th.

LAST WEEK, JULY 4th WEEKEND
Thu 2nd. 1:28-1:30 Settlement Window.
Thu 2nd. 4pm Last Trade. Start of Maintenance Window.
Thu 2nd. 5pm Reopen. Trades done after this point count as trades for Monday 6th since there is no Friday 3rd trade day.
Fri 3rd. 12pm Last Trade. Market Halt until Sunday. No Settlement Window.

Sun 5th. 5pm Reopen. Continuation of Trading for Mon 6th.

So the problem becomes Friday 3rd was officially not a trading day, hence has no high, low, open or close, BUT there was actually trading activity for 19 hours. So theoretically Monday 6th was a 42 hour trading day instead of the normal 23 but the software struggles to handle that. You can understand the underlying reasons why the software struggles with this, but given that this is a billion dollar industry you think they would have worked out how to handle it!

Years ago the market closed on Thursday and didn't reopen to Sunday night. But with the global nature of markets and the shift from exchanges being member/local focused to shareholder/revenue focused this is no longer the case.


2020-07-11 123255


2020-07-11 123342


Thank you for solving my doubt.
I found another problem...
It's so odd that the open & high of daily and 1440min of the same day is different....

Reply With Quote
 
(login for full post details)
  #16 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

@slicent

Smells funny....
@CL
6/21 and 6/22/01 same open high low and close
6/25, 6/26 and 6/27/01 no prices at all!


Reply With Quote
 
(login for full post details)
  #17 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

And another. Really not difficult to find....


Reply With Quote
 
(login for full post details)
  #18 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

And another. That's 7 bad data points in a 6 month period.


Reply With Quote
 
(login for full post details)
  #19 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

Sure you don't want me to post charts of them all but just downloaded 20 years of @CL data and there's 16 obvious data errors all in 2001-2003. Must but not all are associated with US Holidays.

Reply With Quote
 
(login for full post details)
  #20 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

Last one. I promise....


Reply With Quote
The following user says Thank You to SMCJB for this post:
 
(login for full post details)
  #21 (permalink)
slicent
Taipei City
 
 
Posts: 8 since Jul 2020
Thanks: 7 given, 0 received

So It's obvious that the data of TS is so poor!
Will eSignal be better?

Reply With Quote
 
(login for full post details)
  #22 (permalink)
 SMCJB 
Legendary Market Wizard
Houston, TX
 
Experience: Advanced
Platform: Trading Technologies
Broker: Primary Advantage Futures. Also ED&F and Tradestation
Trading: Primarily Energy but also a little GE, GC, SI & Bitcoin
 
Posts: 4,064 since Dec 2013
Thanks: 3,371 given, 8,030 received

Only thing I would say is that these were all in 2001-2003 data so 17+ years old.
No idea on eSignal.
I would point out that this is probably not caused by the underlying data itself but probably by how Tradestation are building the continuous contract themselves.

Reply With Quote


futures io Trading Community Platforms and Indicators TradeStation > there may be some data error in TS


Last Updated on July 25, 2020


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1,800 in prizes!

April

An Introduction to CME Group Micro Bitcoin Futures w/David Lerman

Elite only
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, Ph: +507 833-9432 (Panama and Intl), +1 888-312-3001 (USA and Canada), info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts