Accuracy of SC Denali data, and a data reconciliation request - futures io
futures io



Accuracy of SC Denali data, and a data reconciliation request


Discussion in Sierra Chart

Updated
      Top Posters
    1. looks_one Schnook with 13 posts (12 thanks)
    2. looks_two bobwest with 5 posts (5 thanks)
    3. looks_3 tomgilb with 5 posts (8 thanks)
    4. looks_4 TraderMich with 3 posts (4 thanks)
      Best Posters
    1. looks_one tomgilb with 1.6 thanks per post
    2. looks_two TraderMich with 1.3 thanks per post
    3. looks_3 bobwest with 1 thanks per post
    4. looks_4 Schnook with 0.9 thanks per post
    1. trending_up 1,148 views
    2. thumb_up 33 thanks given
    3. group 9 followers
    1. forum 28 posts
    2. attach_file 8 attachments




Welcome to futures io: the largest futures trading community on the planet, with well over 125,000 members
  • Genuine reviews from real traders, not fake reviews from stealth vendors
  • Quality education from leading professional traders
  • We are a friendly, helpful, and positive community
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts
  • We are here to help, just let us know what you need
You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

(If you already have an account, login at the top of the page)

 
Search this Thread
 

Accuracy of SC Denali data, and a data reconciliation request

(login for full post details)
  #1 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Sierra Chart has been telling customers for months that CQG data is terrible, and recommending that users switch to their Denali data feed.

Based on recent problems with CQG I followed their advice and made the switch last week.

Meanwhile, I've been downloading some price data to update some spreadsheet studies, and while doing so I noticed an anomaly that made me question the accuracy of the data.

The specific instance I'm referring to is in the daily historical data for ZN. I selected the continuous contract - volume based rollover option. But for Dec. 4, 2018, a few days after rollover, Denali data shows extremely low volume, and for the following day, Dec. 5 2018, the exact same low volume number and an open, high, low, close of 120 0.5/32, implying no range for the entire session. This was not a holiday.

See screenshot:


I've asked Sierra Chart support to look into this, in case there's something I did wrong with my settings, but they have not yet responded. I will update here when they do.

But I wanted to ask other users if they've had any issues with Denali data accuracy. Because data integrity is obviously a critical issue. If you can't trust your data, then your charts, backtests etc. are all worthless.

Also, would someone with non Denali data be willing to download or screeshot some daily price data for ZN so I can compare the OHLC and volume numbers with the Denali data?

Any help in resolving this would be very much appreciated.

Started this thread Reply With Quote
The following 3 users say Thank You to Schnook for this post:

Journal Challenge April 2021 results (now extended!):
Competing for $1800 in prizes from Jigsaw
looks_oneMaking a Living with the Microsby sstheo
(267 thanks from 25 posts)
looks_twoSalao's Journalby Salao
(67 thanks from 12 posts)
looks_3Deetee’s DAX Trading Journal (time based)by Deetee
(48 thanks from 18 posts)
looks_4Learning to Profit - A journey in algorithms and optionsby Syntax
(37 thanks from 15 posts)
looks_5Maybe a little bit different journalby Malykubo
(17 thanks from 17 posts)
 
Best Threads (Most Thanked)
in the last 7 days on futures io
The Crude Dude Oil Trading System
87 thanks
Big Mike in Ecuador
76 thanks
Help improve the FIO community
41 thanks
The New Micro Contract - MICRO BITCOIN coming May 2021
30 thanks
futures io site changelog and issues/problem reporting
25 thanks
 
(login for full post details)
  #2 (permalink)
 TraderMich 
Copenhagen + Denmark
 
Experience: Advanced
Platform: Sierra Chart
Broker: Broker: Trade Future 4 Less. Data Feed: CQG
Trading: ES Mini
 
TraderMich's Avatar
 
Posts: 70 since Oct 2018
Thanks: 34 given, 55 received


Schnook View Post
Sierra Chart has been telling customers for months that CQG data is terrible, and recommending that users switch to their Denali data feed.

Based on recent problems with CQG I followed their advice and made the switch last week.

Meanwhile, I've been downloading some price data to update some spreadsheet studies, and while doing so I noticed an anomaly that made me question the accuracy of the data.

The specific instance I'm referring to is in the daily historical data for ZN. I selected the continuous contract - volume based rollover option. But for Dec. 4, 2018, a few days after rollover, Denali data shows extremely low volume, and for the following day, Dec. 5 2018, the exact same low volume number and an open, high, low, close of 120 0.5/32, implying no range for the entire session. This was not a holiday.

See screenshot:


I've asked Sierra Chart support to look into this, in case there's something I did wrong with my settings, but they have not yet responded. I will update here when they do.

But I wanted to ask other users if they've had any issues with Denali data accuracy. Because data integrity is obviously a critical issue. If you can't trust your data, then your charts, backtests etc. are all worthless.

Also, would someone with non Denali data be willing to download or screeshot some daily price data for ZN so I can compare the OHLC and volume numbers with the Denali data?

Any help in resolving this would be very much appreciated.

Schnook - I has the same idea and are also, like you were on CQG, but I have not yet reacted on it, so you are faster than me

I have checked on the CQG data and in fact the volume is extremely low on the two days in question 2018 Dec. 4th. and 5 th. - I do not know why.

On 4th.: O= 119 5/32 H =119 22/32 L= 1193/32 C= 119 13,5/32. Vol approx. 18.600

On 5 th. O H L C all 119 13,5/32. same as close on 4 th. Vol more or less the same.

This is all reading from the chart so could be 1 tick or so wrong, but confirms the dare you get. - I do not know if you can find date on-line from CBOT that far back, or ask them what the problem was those day.
I checked same dates for 30 years bond and this is the same - very little volume on both day and 5th. no range on one price and the same as the close of the 4th.

2 yr. note the same -

Ultra Bonds the same

So seems like a general problem in the bonds at CBOT

Hope this helps

Reply With Quote
The following user says Thank You to TraderMich for this post:
 
(login for full post details)
  #3 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received


Yes, that is helpful @TraderMich - thank you

It does seem odd though. Perhaps the problem is not with Denali, but elsewhere. Still, I'd still really like to get to the bottom of this.

Started this thread Reply With Quote
 
(login for full post details)
  #4 (permalink)
 TraderMich 
Copenhagen + Denmark
 
Experience: Advanced
Platform: Sierra Chart
Broker: Broker: Trade Future 4 Less. Data Feed: CQG
Trading: ES Mini
 
TraderMich's Avatar
 
Posts: 70 since Oct 2018
Thanks: 34 given, 55 received


Schnook View Post
Yes, that is helpful @TraderMich - thank you

It does seem odd though. Perhaps the problem is not with Denali, but elsewhere. Still, I'd still really like to get to the bottom of this.

I do not think it is with Denali since I used CQG as data source - Most likely it was a CBOT problem in the bond sections as all bonds and T-note futures had same problem in the same way.

But I would be interested in learning about your experience the this Denali/TT data feed and trading routing with Sierra Charts, as I am considering the same.

Reply With Quote
The following user says Thank You to TraderMich for this post:
 
(login for full post details)
  #5 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

@TraderMich I am going to reserve judgment until I've had a chance to get more familiar with Denali. Again, data integrity is VERY important to me, which is why I'm so intent on figuring out what's going on with these bond prices from Dec. 5 2018

Here is some data I pulled off the internet (investing.com)



This source at least shows some movement in price on Dec. 5 but reports zero volume. Note also the declining volumes until a large bounce on Dec. 20. This suggests to me that the contract roll might be a factor.

Still, the roll shouldn't be a factor in the Denali data, because I am using the volume based rollover setting, and the Denali chart shows the roll had occurred a few days prior.

Does anyone else have different data, perhaps from a different source that they could share? Does anyone else have any ideas or guesses as to what the issue might be?

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #6 (permalink)
 ocpb 
Hartford CT, USA
 
Experience: Beginner
Platform: None
Trading: ES
 
Posts: 121 since Jan 2015
Thanks: 126 given, 117 received


Schnook View Post
Does anyone else have any ideas or guesses as to what the issue might be?

US Stock Markets were closed on Dec. 5, 2018, as a national day of remembrance for George HW Bush.

Futures were on holiday schedules, Interest rate products were completely shut down.

https://www.cmegroup.com/notices/clearing/2018/12/Chadv18-474.html

Reply With Quote
The following 3 users say Thank You to ocpb for this post:
 
(login for full post details)
  #7 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received


ocpb View Post
US Stock Markets were closed on Dec. 5, 2018, as a national day of remembrance for George HW Bush.

Futures were on holiday schedules, Interest rate products were completely shut down.

https://www.cmegroup.com/notices/clearing/2018/12/Chadv18-474.html


Thank you! Very glad to have this cleared up. When I looked at an online calendar all I found was "National Cookie Day" on the 4th

Started this thread Reply With Quote
The following 2 users say Thank You to Schnook for this post:
 
(login for full post details)
  #8 (permalink)
 TraderMich 
Copenhagen + Denmark
 
Experience: Advanced
Platform: Sierra Chart
Broker: Broker: Trade Future 4 Less. Data Feed: CQG
Trading: ES Mini
 
TraderMich's Avatar
 
Posts: 70 since Oct 2018
Thanks: 34 given, 55 received


ocpb View Post
US Stock Markets were closed on Dec. 5, 2018, as a national day of remembrance for George HW Bush.

Futures were on holiday schedules, Interest rate products were completely shut down.

https://www.cmegroup.com/notices/clearing/2018/12/Chadv18-474.html

Thank you very much

Reply With Quote
The following 2 users say Thank You to TraderMich for this post:
 
(login for full post details)
  #9 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Just to follow up on this, SC Support never provided an adequate response to my questions and the above noted issue remains unresolved (inaccurate volume and questionable price data for Dec. 4 and 5, 2018).

Second, there are other issues with the data. For example, this past Friday (Good Friday holiday), rates and FX were open for an abbreviated session, but none of that data is shown in the historical (daily) data feed.

In TY, for example, after the employment number came out on April 2, the market traded a 130-29/32 to 131-20/32 range, but on the daily, historical chart none of that is shown.

When I asked SC support about the missing TY data from Friday, I received the following response: "contact the exchange about this. There is not going to be data for Friday. There is nothing wrong. Contact the exchange. We expect you to understand, how the exchanges provide data. We are not going to explain that. That is their responsibility. Instead you post here, thinking there is something wrong. There is not! Do not take more time from us on this."

Indeed, the daily charts on the CME website also omit the April 2 data, which I find a bit frustrating. However, Yahoo Finance is reporting complete data, Investing.com is reporting complete data, Bloomberg is reporting complete data. But the CME, and the Denali data that I pay for, are not.

So do I just need to accept it, and make a mental note that when studying historical data, or backtesting on daily data, I will have to contend with incomplete data around holidays or shortened trading sessions?

If so, I fear that I will never be able to trust my data.

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #10 (permalink)
 bobwest 
Site Moderator
Sarasota FL
 
Experience: Advanced
Platform: Sierra Chart
Trading: ES, YM
 
bobwest's Avatar
 
Posts: 6,453 since Jan 2013
Thanks: 48,921 given, 21,581 received



Schnook View Post
Just to follow up on this, I continue to have issues with Sierra Chart's Denali data.

First, SC Support never provided an adequate response to my questions and the above noted issue remains unresolved.

Second, there were other issues with the data. For example, this past Friday (Good Friday holiday), rates and FX were open for an abbreviated session, but none of that data is shown in the historical (daily) data feed.

In TY, for example, after the employment number came out on April 2, the market traded a 130-29/32 to 131-20/32 range, but on the daily, historical chart none of that is shown.

When I asked SC support about the missing TY data from Friday, I received the following response: "contact the exchange about this. There is not going to be data for Friday. There is nothing wrong. Contact the exchange. We expect you to understand, how the exchanges provide data. We are not going to explain that. That is their responsibility. Instead you post here, thinking there is something wrong. There is not! Do not take more time from us on this."

What am I missing here? What is it about the exchange that I'm not understanding? Yahoo Finance is reporting complete data. Investing.com is reporting complete data. Bloomberg is reporting complete data. So why can't Denali? All I want is for my historical data to be accurate and complete. Am I wrong to ask this?

Edit: Oops, I just read your edited post, which includes this:

Schnook View Post
Indeed, the daily charts on the CME website also omit the April 2 data, which I find a bit frustrating. However, Yahoo Finance is reporting complete data, Investing.com is reporting complete data, Bloomberg is reporting complete data. But the CME, and the Denali data that I pay for, are not.

So do I just need to accept it, and make a mental note that when studying historical data, or backtesting on daily data, I will have to contend with incomplete data around holidays or shortened trading sessions?

If so, I fear that I will never be able to trust my data.

This changes things, but I'll leave what I originally wrote intact. I don't know what is going on, but perhaps SC has a point this time. Perhaps it has to do with what trade date holiday data is reported as....?

That would suggest that SC might be right after all, if CME is reporting as they are. It would mean that there was actual trading for that calendar date, but it is not reported as belonging to that trade date in the daily numbers. (I often get notices from my broker about how trades during partial sessions on holidays are handled, but I'm afraid I ignore them, because I'm not going to trade on a partial seesion on a holiday anyway. Maybe I should read them.)

So maybe unexpected, but not incorrect by SC, if this is so.

So take the rest of my post with a certain amount of salt.... I"m starting to think I was wrong.

Here's my original post, below the line. I still think they have an attitude problem.

-------------------------

I don't know the particular issue with this data and so I can't address it directly. I assume that the fact that other sources are giving quotes means that SC is incorrect here.

This response from SC sounds like one of their knee-jerk "you're wrong, we aren't, how dare you question us" responses. Someone at SC has an unfortunate quick trigger finger and fires off defensive counters like this if they feel they are being questioned.

A simple reply would be to list the data reported by other sources. Also, CME does provide data on its website, which would counter the "contact the exchange about this. There is not going to be dara for Friday" statement. (I am assuming that the exchange website will show they are wrong.... if not, well, that's another resolution. Either way, it's a simple way to settle things.)

My guess is that someone snapped off this quick response without knowing what they were talking about. I have seen them give up their quick-response rants in the past when actually given specifics, but sometimes it requires a hammer to get them to do it.

As a fan of SC I am always unhappy with their defensive type mentality that makes them so sensitive to any suggestion they are not perfect. But at least they usually do snap out of it when given some facts. It's too bad that their customers may have to do it.

Bob.

When one door closes, another opens.
-- Cervantes, Don Quixote
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to bobwest for this post:
 
(login for full post details)
  #11 (permalink)
 bobwest 
Site Moderator
Sarasota FL
 
Experience: Advanced
Platform: Sierra Chart
Trading: ES, YM
 
bobwest's Avatar
 
Posts: 6,453 since Jan 2013
Thanks: 48,921 given, 21,581 received

I'm thinking I may have spoken too soon in by last post about SC and the CME data. I don't honestly have time to look at this right now, but I think it just may be that there were actual trades on Friday, but the official daily data doesn't include them because of the way CME reports their trade date, which I think is not the same as their calendar date. I believe I've run into this before. I'll edit some of that post, and hope this is the resolution.

If it is, then yes, there will always be an anomaly with partial sessions on holidays, because that is the CME stance on how they report it. I think this has to be lived with.

Bob.

When one door closes, another opens.
-- Cervantes, Don Quixote
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to bobwest for this post:
 
(login for full post details)
  #12 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Thanks, Bob.

I'll be the first to admit that I am not an expert on how exchanges report data and how the plumbing ultimately works with these data feeds, but if I'm paying for data I should have a reasonable expectation of accuracy and completeness. I know I'm repeating myself when I say this, but I do consider it important to have complete and accurate data when conducting historical analysis and strategy backtests.

The fact is that FX and rates markets were open and any open positions or orders on those days would have been live.

And even if the exchange sees fit to omit those data, the other (free) historical data providers include it. I wish that Denali did as well.

SC support, meanwhile, has once again lived up to its well-earned reputation of being rude, dismissive, and unprofessional.

Out of curiosity, are there any traders here using different (non Denali) data that do show April 2 price and volumes in their historical daily feed?

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #13 (permalink)
 bobwest 
Site Moderator
Sarasota FL
 
Experience: Advanced
Platform: Sierra Chart
Trading: ES, YM
 
bobwest's Avatar
 
Posts: 6,453 since Jan 2013
Thanks: 48,921 given, 21,581 received


Schnook View Post
Thanks, Bob.

I'll be the first to admit that I am not an expert on how exchanges report data and how the plumbing ultimately works with these data feeds, but if I'm paying for data I should have a reasonable expectation of accuracy and completeness. I know I'm repeating myself when I say this, but I do consider it important to have complete and accurate data when conducting historical analysis and strategy backtests.

The fact is that FX and rates markets were open and any open positions or orders on those days would have been live.

And even if the exchange sees fit to omit those data, the other (free) historical data providers include it. I wish that Denali did as well.

SC support, meanwhile, has once again lived up to its well-earned reputation of being rude, dismissive, and unprofessional.

Out of curiosity, are there any traders here using different (non Denali) data that do show April 2 price and volumes in their historical daily feed?

At this point, I think this is a good question, and I'd like to see other responses. I am coming around to the idea that CME is going to report the data the same as SC does (actually, that SC is reporting it the same as CME, right?) and that nothing can be done when dealing with historical daily data with partial sessions due to holidays.

See the post I just made up above, after my first one, which is now somewhat edited.

I do agree about the SC attitude problem. It would have taken them no more time to explain why their mumbers are what they are than it took to blast you.

If there's another answer to the data question, I assume someone will post about it.

Bob.

When one door closes, another opens.
-- Cervantes, Don Quixote
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to bobwest for this post:
 
(login for full post details)
  #14 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

And to add yet another wrinkle...

CME data is showing and reporting historical data for RTY on April 2, but SC / Denali is not


Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #15 (permalink)
 bobwest 
Site Moderator
Sarasota FL
 
Experience: Advanced
Platform: Sierra Chart
Trading: ES, YM
 
bobwest's Avatar
 
Posts: 6,453 since Jan 2013
Thanks: 48,921 given, 21,581 received

This is all a little puzzling, but I wonder how important it actually is.

There are perhaps a dozen holidays, give or take, in a full year, probably fewer. Unless each individual bar is very important to whatever you are backtesting, having a few that are anomalies probably will not matter in the sense of throwing you off. This may not always be the case, but I think it usually will be.

For example, when I look at daily data I am mainly just interested in trends over multiple days, usually quite a few (weeks and months.) I don't know that one day being off actually matters that much. If it does, I will be looking at intra-day data, which should cover all the trading on that calendar day.

There is also the probable reason that CME treats these partial days this way, at least what I assume (guess) is the reason. Typically these are very low-participation days, very often either US holidays (very few US traders active) or international holidays (very few traders active from anywhere.)

As a rule, trading ends on these semi-holidays in the early afternoon US time, which I have always assumed is mainly to accommodate European traders, who will be out of the markets in the 11:00 or 11:30-ish (Eastern Time) period due to the time difference. Most US traders will stay away entirely, at least for US holidays. An example would be the US July 4 holiday, if it isn't on a weekend. No Americans are going to be trading, and the US stock markets will be closed. So there is usually just a small amount of light non-institutional trading, which would drop off to near-zero as Europe closed down. (I believe that equity contracts closed at 08:15 US CT on Good Friday. Not much point to even recording that data point.)

Other markets will have other characteristics, but in general I think that trading is going to be light on the partial holidays. If there's not much trading, you could ask whether it's actually that important, in terms of identifying what is happening in whatever market it's in.

So the question basically is, how much do a few odd days a year actually matter? It would be good to know the reason for the oddness, but I don't think it will usually make a great difference.

Bob.

When one door closes, another opens.
-- Cervantes, Don Quixote
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to bobwest for this post:
 
(login for full post details)
  #16 (permalink)
 tr8er 
Europe
 
Experience: Advanced
Platform: TradeNavigator, BookMap
Trading: ES, CL, 6E, 6B
 
Posts: 471 since Jan 2017
Thanks: 29 given, 412 received

Bob is absolutly right, these short days (with early close) are not important, special for backtesting.

Most platforms show these days, but maybe there is a setting that excludes these days. Perhaps you should watch for it, I don't know it, because it is years ago, that I have used SC.

Reply With Quote
The following user says Thank You to tr8er for this post:
 
(login for full post details)
  #17 (permalink)
 tomgilb 
Nashville TN USA
 
Experience: Advanced
Platform: SierraChart
Broker: Sierra Futures/SC Denali
Trading: NQ ES YM
 
Posts: 431 since Nov 2009
Thanks: 168 given, 535 received

Denali Daily data (.dly) did not included Good Friday.
Denali intraday data (.scid) has data for Good Friday.

A 1Day 1-0-0 intraday chart will look similar to Schnook's post from the CME, so its comparison to Denali is from two different data. Even so, I was not able to create a chart that matched CME's. See pic.

Reconciling Daily and 1Day-intraday charts is not straigthforward, and usually not exact.
https://www.sierrachart.com/index.php?page=doc/SessionTimes.php

Specific dates can be excluded from displaying by editing the Symbol Settings:
sierrachart.com/index.php?page=doc/ChartSettings.html#DatesToExclude
(Copy/Paste the above because this forum doesn't allow # in the url.)

Attached Thumbnails
Click image for larger version

Name:	RTY Easter weekend Daily vs 1Day.PNG
Views:	21
Size:	47.6 KB
ID:	311238  
Reply With Quote
The following 2 users say Thank You to tomgilb for this post:
 
(login for full post details)
  #18 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

I appreciate what you guys are saying, but for my purposes I actually do require this holiday data. To wit, I am doing a lot of seasonal, calendar-based analysis right now that specifically requires these holiday sessions to be included. And to a bond trader, this past Friday - when the employment report was released - is in fact an extremely important data set.

The other part of this whole issue is that as a paying customer of the new Denali data feed I simply wish to conduct some due diligence, and in doing so, I have now noticed a few errors, which frankly makes me question the accuracy of the data in its entirety.

Here's yet another example:



Thursday, April 1 was NOT a holiday, yet the Denali daily historical data shows volume in June RTY of just 9216 contracts, while the intraday data shows total volume of well over 100,000 contracts. So the data actually contradicts itself. This does not inspire confidence. Again, I need my data to be accurate and reliable, and my due diligence thus far shows that not to be the case with Denali.

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #19 (permalink)
 tomgilb 
Nashville TN USA
 
Experience: Advanced
Platform: SierraChart
Broker: Sierra Futures/SC Denali
Trading: NQ ES YM
 
Posts: 431 since Nov 2009
Thanks: 168 given, 535 received


Quoting 
the Denali daily historical data shows volume in June RTY of just 9216 contracts, while the intraday data shows total volume of well over 100,000 contracts

The reason for the discrepancy between a Daily chart and a 1Day intraday chart is explained in the documentation.
It has to do with when the day starts and ends, and the date stamp of the bar:
https://www.sierrachart.com/index.php?page=doc/SessionTimes.php

You'll also find that they will rarely be exactly the same.
You might be chasing a phantom issue.

Reply With Quote
 
(login for full post details)
  #20 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received


tomgilb View Post
The reason for the discrepancy between a Daily chart and a 1Day intraday chart is explained in the documentation.
It has to do with when the day starts and ends, and the date stamp of the bar:
https://www.sierrachart.com/index.php?page=doc/SessionTimes.php

You'll also find that they will rarely be exactly the same.
You might be chasing a phantom issue.

No this is real. RTY traded way more than just 9216 lots on Thursday. Sierra Chart has also just now acknowledged that it's incorrect and have stated that they will ask their data provider about it.

If you look at the screenshot I attached to my last post you will see the obvious error. And anyone who uses volume as an input in a trading strategy should care about this.


tr8er
these short days (with early close) are not important

Perhaps they are not important to you, @tr8er, but for me they are critical.

Again, perhaps I'm in the minority here, but I need my data to be both accurate and complete. At least among professionals I know that I am not unique in this regard.

Started this thread Reply With Quote
 
(login for full post details)
  #21 (permalink)
 tomgilb 
Nashville TN USA
 
Experience: Advanced
Platform: SierraChart
Broker: Sierra Futures/SC Denali
Trading: NQ ES YM
 
Posts: 431 since Nov 2009
Thanks: 168 given, 535 received


Quoting 
If you look at the screenshot I attached to my last post you will see the obvious error

I see your point. Obviously the Daily and the Intraday do not have the same volume data.

FWIW, ES, NQ, and YM also all had very low volume on their 4-1 Daily bar. Maybe others also.

Reply With Quote
The following user says Thank You to tomgilb for this post:
 
(login for full post details)
  #22 (permalink)
 tomgilb 
Nashville TN USA
 
Experience: Advanced
Platform: SierraChart
Broker: Sierra Futures/SC Denali
Trading: NQ ES YM
 
Posts: 431 since Nov 2009
Thanks: 168 given, 535 received

It seems SC has repaired the Daily volume data for Apr 1.

Attached Thumbnails
Click image for larger version

Name:	RTY 4-1 vol.PNG
Views:	24
Size:	23.4 KB
ID:	311274  
Reply With Quote
The following user says Thank You to tomgilb for this post:
 
(login for full post details)
  #23 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Thanks, it does indeed appear as though this has been corrected. Without any followup comment to me from SC support, I might add. Meanwhile, the incorrect volume figures from Dec. 5, at least in TY, still have not been corrected. I presume this would be the case in other products as well.



A few questions remain
What caused these errors in the first place?
Would anything have been corrected it if I hadn't brought it to their attention?
What other errors remain unnoticed and therefore uncorrected?
What protocols are in place to prevent future errors from occurring?

This is a data integrity issue. One that concerns me still. These incorrect volumes were obvious, and thus easy to detect, but what about less obvious errors? I don't have the time and resources to check them all, and frankly it isn't my job to do so. I should be able to trust the data that I'm paying for, and right now, unfortunately, I don't.

Conclusions:
Denali data may not be accurate and reliable
Sierra Chart Support staff remain rude and unprofessional

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #24 (permalink)
 tomgilb 
Nashville TN USA
 
Experience: Advanced
Platform: SierraChart
Broker: Sierra Futures/SC Denali
Trading: NQ ES YM
 
Posts: 431 since Nov 2009
Thanks: 168 given, 535 received


Quoting 
Conclusions:
Denali data may not be accurate and reliable
Sierra Chart Support staff remain rude and unprofessional

I don't think the Historical Daily data is from the same provider as the Denali data.
However, I'm not yet able to confirm this in the documentation.

There is at least one SC Support staff that is professional (John). We know who he is because he's the only one that includes his name. IMO, they should all include their names.
There is also one that should be removed from forum duty.

Reply With Quote
The following 4 users say Thank You to tomgilb for this post:
 
(login for full post details)
  #25 (permalink)
Average Joe
Singapore
 
 
Posts: 28 since Apr 2017
Thanks: 15 given, 11 received


Schnook View Post

Conclusions:
Denali data may not be accurate and reliable
Sierra Chart Support staff remain rude and unprofessional

LOL, that is their usual "take it or leave it" fk up attitude. But hey, look at it from the bright side at least they replied you. 🤣

For back testing perhaps you could consider downloading data from elsewhere and upload to SC? Yahoo Finance for example.

Reply With Quote
 
(login for full post details)
  #26 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Thanks guys. Appreciate your input.

@tomgilb I'm pretty sure that you're correct about the historical data coming from a different source than the intraday data. In fact yesterday I saw somewhere in the documentation that the historical data comes directly from the exchange. While trying to reconcile the daily to the intraday I also noticed that it appears the CME data for Treasuries uses the start of the 18:00 evening session as the open and 15:00 as the close. But, strangely, sometimes the daily close matches the OPEN of the period beginning at 15:00, and other times it matches the CLOSE of the period ending at 15:00 (they don't always match). This obviously makes reconciliation of daily and intraday a bit squirrely since the sources are different and don't always agree. Moreover, what happens between 15:00 and 17:00 is not captured in the historical daily data and instead is shown as a gap. Just something to be aware of when analyzing the historical data.

When running backtests, I believe the Denali intrada data is used, so hopefully there would be no gaps or errors there. I say hopefully because I haven't been able to independently confirm this.

Started this thread Reply With Quote
 
(login for full post details)
  #27 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Here's a great example of the daily / intraday data reconciliation issue noted above



This is the bond contract from yesterday to today. The historical daily chart shows a closing price of 159-00 for April 15. The 30 minute chart shows an OPEN price of 159-00 at 15:00. But then, starting almost precisely at 15:00, the market puked by more than a point on heavy volume, so that by 17:00 it ended way down at 157-29

Meanwhile, the volume reported on the historical daily chart for ZB is 614,615, which EXCEEDS the volumes shown on the intraday chart between 15:00 on April 14 and 18:00 on April 15. Perhaps this is due to block trades conducted off exchange and thus not included in the intraday volumes but then added to the daily total after the close of the session. I really can't say for sure. But the point is, I can't seem to get the volumes to match, and the daily data perhaps includes post 15:00 volumes but NOT post 15:00 price.

Again, just something to be aware of. Personally I think it's weird and also a bit annoying but what can I do

Started this thread Reply With Quote
The following user says Thank You to Schnook for this post:
 
(login for full post details)
  #28 (permalink)
 bobwest 
Site Moderator
Sarasota FL
 
Experience: Advanced
Platform: Sierra Chart
Trading: ES, YM
 
bobwest's Avatar
 
Posts: 6,453 since Jan 2013
Thanks: 48,921 given, 21,581 received

I just remembered something that may only create more confusion, but CME has a procedure to determine the daily "settlement" price that is almost always not going to be the same as the actual intra-day last price. I don't know if the CME data uses this settlement price as the daily close or not.

Here's what they do:

"Equity Futures: For S&P and NASDAQ, the settlement price of the lead* month contract is the midpoint of the
closing range determined based on pit trading activity between 15:14:30-15:15:00 Central Time (“CT”). For all other
equity indices, the Volume Weighted Average Price (VWAP) of trades executed on Globex between 15:14:30-
15:15:00 CT is used to determine the settlement prices for the lead month contracts. Back month contract months are
settled to traded or quoted spread relationships. E-mini S&P and Nasdaq are settled to the value derived from the Big
S&P and Nasdaq. "

(Link: https://www.cmegroup.com/market-data/files/CME_Group_Settlement_Procedures.pdf )

Is this likely to be relevant to your issue? I would say, probably not. But here it is, just in case it might.

Bob.

-----------------

Edit: I just noticed that this pdf refers to the "pit trading activity," which no longer exists. So this procedure may be an old one that Google served up, that is no longer in effect.

But I do think that they use the short-term (final x seconds) VWAP method, and probably for ES and NQ now.

When one door closes, another opens.
-- Cervantes, Don Quixote
Visit my futures io Trade Journal Reply With Quote
The following user says Thank You to bobwest for this post:
 
(login for full post details)
  #29 (permalink)
 Schnook 
Munich, Germany
 
Experience: Advanced
Platform: Sierra Chart
Broker: Interactive Brokers
Trading: liquid products
 
Schnook's Avatar
 
Posts: 459 since Jul 2016
Thanks: 877 given, 1,552 received

Thanks, Bob. Yes, you're right about the settlement price, but thankfully that shouldn't affect these data files, as the settlement price is more of a procedural thing used for mark-to-market purposes. With regards to the price data I'm studying - the raw data that underlies the charts we see on our screens - the only thing that should be reported there is actual traded prices and volumes.

Started this thread Reply With Quote


futures io Trading Community Platforms and Indicators Sierra Chart > Accuracy of SC Denali data, and a data reconciliation request


Last Updated on April 16, 2021


Upcoming Webinars and Events
 

NinjaTrader Indicator Challenge!

Ongoing
 

Journal Challenge w/$1,800 in prizes!

April
     



Copyright © 2021 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts