How to measure data feed latencies between continents? - Traders Hideout | futures io social day trading
futures io futures trading


How to measure data feed latencies between continents?
Updated: Views / Replies:5,991 / 66
Created: by Koepisch Attachments:18

Welcome to futures io.

(If you already have an account, login at the top of the page)

futures io is the largest futures trading community on the planet, with over 90,000 members. At futures io, our goal has always been and always will be to create a friendly, positive, forward-thinking community where members can openly share and discuss everything the world of trading has to offer. The community is one of the friendliest you will find on any subject, with members going out of their way to help others. Some of the primary differences between futures io and other trading sites revolve around the standards of our community. Those standards include a code of conduct for our members, as well as extremely high standards that govern which partners we do business with, and which products or services we recommend to our members.

At futures io, our focus is on quality education. No hype, gimmicks, or secret sauce. The truth is: trading is hard. To succeed, you need to surround yourself with the right support system, educational content, and trading mentors Ė all of which you can find on futures io, utilizing our social trading environment.

With futures io, you can find honest trading reviews on brokers, trading rooms, indicator packages, trading strategies, and much more. Our trading review process is highly moderated to ensure that only genuine users are allowed, so you donít need to worry about fake reviews.

We are fundamentally different than most other trading sites:
  • We are here to help. Just let us know what you need.
  • We work extremely hard to keep things positive in our community.
  • We do not tolerate rude behavior, trolling, or vendors advertising in posts.
  • We firmly believe in and encourage sharing. The holy grail is within you, we can help you find it.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.

You'll need to register in order to view the content of the threads and start contributing to our community.  It's free and simple.

-- Big Mike, Site Administrator

Reply
 18  
 
Thread Tools Search this Thread
 

How to measure data feed latencies between continents?

  #11 (permalink)
Elite Member
@ Germany
 
Futures Experience: Beginner
Platform: NinjaTrader
Broker/Data: Mirus Futures/Zen-Fire
Favorite Futures: FDAX
 
Posts: 440 since Nov 2011
Thanks: 254 given, 369 received


Big Mike View Post
Unless you are wanting microsecond precision, I think using NTP is sufficient instead of GPS.

You need a server in Chicago. Depending on your budget I would say either @sam028 which uses Steadfast backbone at 350 E. Cemak, or go with an Aurora server from your execution broker next to CME. The difference between these will be less than 1ms, but depending on your app that could be an eternity.

Sent from my LG Optimus G Pro

NTP was my first idea ("Internet clock adjustment"), but Fat Tails mentioned that there will be still an 100ms latency. But i think if i use the nearest NTP server the time is pretty precise. I don't know if the protocol implementation take the trip time to and from the NTP Server into account. If yes, everything is fine - i will check that too. And of course i will be utilizing SAM's systems for that test. For my strategy controlling a latency of 100-500 ms is sufficient, the real strategy execution will be aligned to latency vs. costs considerations. So for the running systems there will also be SAM in my boat.

But i'm really surprised that no one has some valid, proofable statistics about that topic. As an european trader i want to know if the data feed currently lagging 10, 100, 1000 or 30000 MS behind the exchange time. It's an relevant information, because if the feed lags, my system lags and i don't trade until the lag is gone. Many threads at BigMike are talking about high latency, but now longer term statistics can be found.

Koepisch

Reply With Quote
 
  #12 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received


Koepisch View Post
NTP was my first idea ("Internet clock adjustment"), but Fat Tails mentioned that there will be still an 100ms latency. But i think if i use the nearest NTP server the time is pretty precise. I don't know if the protocol implementation take the trip time to and from the NTP Server into account. If yes, everything is fine - i will check that too. And of course i will be utilizing SAM's systems for that test. For my strategy controlling a latency of 100-500 ms is sufficient, the real strategy execution will be aligned to latency vs. costs considerations. So for the running systems there will also be SAM in my boat.

But i'm really surprised that no one has some valid, proofable statistics about that topic. As an european trader i want to know if the data feed currently lagging 10, 100, 1000 or 30000 MS behind the exchange time. It's an relevant information, because if the feed lags, my system lags and i don't trade until the lag is gone. Many threads at BigMike are talking about high latency, but now longer term statistics can be found.

Koepisch

I strongly disagree on 100ms latency with NTP. Perhaps he was talking about Windows Time, which is not very efficient and loosely based on NTP.

NTP can easily obtain 1ms precision, usually in fact much less.

The reason most people don't have the info you are requesting is because most people on the forum are using retail platforms which don't provide that info. And if you are working with an API it is trivial to compute the latency, no real need for a discussion I don't think on "how" to do it.

As for a discussion on the benefits of lower latency, I think it's been discussed many times. It's all relevant to your application. If you aren't already co-located at Aurora then I don't believe you are anywhere near the "serious" level of most people that want to get into specifics about microsecond (or better) latency like @artemiso. Just guessing of course.

For the "professional" home trader concerned about latency, at the very least you would use a Chicago based server, like at the Equinix data center at 350 E Cermak. It is relatively cheap. As mentioned previously @sam028 offers a few options of VPS or dedicated at that location for reasonable price, or you could go direct to Steadfast if you have the manpower to manage everything.

I have several servers at Equinix. I do not care about gaining that last mile 1ms advantage and therefore do not spend the extra money to be at Aurora. As I spend more and more time developing my custom platform, that may change and it's a simple cost/benefit analysis. There are some latency and real world $$ benefits of being inside that 1ms barrier even for non-HFT, but how much of a benefit will again depend on your strategy.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #13 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received



Koepisch View Post
NTP was my first idea ("Internet clock adjustment"), but Fat Tails mentioned that there will be still an 100ms latency.

If you are referring to post #5 in this thread, the 100ms he is talking about is the speed of light delay between your location and Chicago. Nothing to do with NTP.

NTP will use server pools nearby your physical location (determined based on ping response) automatically, and it also automatically prioritizes based on the quality of those clock sources (ie GPS or etc) from those servers.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #14 (permalink)
Administrator: Retired Backtester
 Vendor: speedytradingservers.com 
Rennes France
 
Futures Experience: Advanced
Platform: NinjaTrader
Broker/Data: IB/Kinetick
Favorite Futures: Futures
 
sam028's Avatar
 
Posts: 3,366 since Jun 2009
Thanks: 3,572 given, 3,982 received


Koepisch View Post
NTP was my first idea ("Internet clock adjustment"), but Fat Tails mentioned that there will be still an 100ms latency. But i think if i use the nearest NTP server the time is pretty precise. I don't know if the protocol implementation take the trip time to and from the NTP Server into account. If yes, everything is fine - i will check that too. And of course i will be utilizing SAM's systems for that test. For my strategy controlling a latency of 100-500 ms is sufficient, the real strategy execution will be aligned to latency vs. costs considerations. So for the running systems there will also be SAM in my boat.

But i'm really surprised that no one has some valid, proofable statistics about that topic. As an european trader i want to know if the data feed currently lagging 10, 100, 1000 or 30000 MS behind the exchange time. It's an relevant information, because if the feed lags, my system lags and i don't trade until the lag is gone. Many threads at BigMike are talking about high latency, but now longer term statistics can be found.

Koepisch

I think you misunderstood @Fat Tails comment, an NTP synchronized clock will be accurate with a few ms precision, far below 100ms. And whatever the NTP servers are, that the magic of NTP .
Depending on the PC hardware clock, the drift can be high, some chips are more reliable than other. There are alos few tricks to know: CPU with power-efficiency frequency adjustment will be less precise than CPU running always at 100% of their nominal frequency.
Virtual Machines are also less precise, as the physical CPUs are shared between multiple VMs. On my VPS the NTP sync is done every minute, which is not useful on a physical server which an accurate clock.

I think a simple way would be to have a NinjaTrader strategy, comparing using OnMarketData(), and compare times.

Success requires no deodorant! (Sun Tzu)
Reply With Quote
 
  #15 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received

High Frequency Trading and the True Value of Time - Victor Yodaiken's blog - The Trading Mesh

That article also mentions PTP (Precision Time Protocol) as an alternative to NTP.

Anyway I fear I may have got off topic with your thread. The easiest answer to your question remains to simply use a data feed supplied exchange time marker and compare it to local time.

You should remove NT7 from the equation as it doesn't support better than 1 second resolution anyway.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #16 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received


Big Mike View Post
The easiest answer

Let me try again

The easiest answer is to just use ping. Ping your broker/data feed. That will probably get you 98% there in terms of latency and takes about 1 second of your time to implement it.

This would work best in the case of a broker supplied data feed and not IQFeed which is in a very different physical location to CME, whereas most brokers are co-located next to CME. IQFeed is not a latency sensitive feed but instead is focused on accuracy, whereas broker feeds are focused on latency and thus give up accuracy (such as dropping L2 events not important to execution) to get there.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
The following user says Thank You to Big Mike for this post:
 
  #17 (permalink)
Elite Member
Berlin, Europe
 
Futures Experience: Advanced
Platform: NinjaTrader, MultiCharts
Broker/Data: Interactive Brokers
Favorite Futures: Keyboard
 
Fat Tails's Avatar
 
Posts: 9,651 since Mar 2010
Thanks: 4,226 given, 25,601 received
Forum Reputation: Legendary


Koepisch View Post
@Fat Tails
Perhaps it is possible to use this method to get the Delta from exchange to europe:

1. Use R|API or CQG to get the EXCHANGE timestamp of the tick tuple.
2. Install a GPS-based time module at your PC
3. Match the incoming tick with the current time of your GPS time

@Koepisch: It is not possible to synchronize your PC to the level of milliseconds, because the data exchange and transmission itself takes several milliseconds. There are a few problems you need to be aware of if you wish to compare the time stamps of the data feed with your PC time.

Reference time being used - UTC, GPS or TAI

There are three available reference times, UTC, GPS and TAI (Temps Atomique International). TAI is about 19 seconds (=19000 milliseconds) ahead of GPS and GPS is about 16 sec (16000 milliseconds) ahead of UTC. This suggests that you better use the same reference time, which is used by exchange and data provider, if you want to find out about a lag which is less than 16000 milliseconds.

I think that the CME trade engine is synchronized to NTP, an interesting discussion of time-stamps can be found here:

http://www.nanex.net/aqck2/4436/RevisitingFedRobbery.pdf


Synchronisation of Trading PC with NTP

The correct synchronization time is NTP. To avoid any issues with lag of the time signal, you should use

- the nearest available NTP time server (which is PTB)
- not connect to that time server via internet (approx. lag 25 msec) but via radio clock receiver (approx. lag 3 msec)

There are some products available from Meinberg that can be connected via USB and that will allow accurate synchronization of your PC. I do not know the accuracy level though.

USB Radio Clocks


Measuring the lag of the Data Feed

You can now compare the time stamps of the data feed to the synchronized time of your PC. The result will still suffer from a few errors

-> due to the lag of the radio signal from the PTB atomic clock
-> due to data processing
-> due to inaccurate synchronization of your PC
-> due inaccurate synchronization of the internal clock of the CME maching engine

However, these errors should only add up to something like 10 - 20 msec, which is good enough for monitoring the data feed lag, which is about 50 msec (100 msec referred to the round trip) or larger. You should even be able to show the current lag on your NinjaTrader tick-built chart as an indicator.

Reply With Quote
 
  #18 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received


Fat Tails View Post
@Koepisch: It is not possible to synchronize your PC to the level of milliseconds, because the data exchange and transmission itself takes several milliseconds. There are a few problems you need to be aware of if you wish to compare the time stamps of the data feed with your PC time.

Reference time being used - UTC, GPS or TAI

There are three available reference times, UTC, GPS and TAI (Temps Atomique International). TAI is about 19 seconds (=19000 milliseconds) ahead of GPS and GPS is about 16 sec (16000 milliseconds) ahead of UTC. This suggests that you better use the same reference time, which is used by exchange and data provider, if you want to find out about a lag which is less than 16000 milliseconds.

I think that the CME trade engine is synchronized to NTP, an interesting discussion of time-stamps can be found here:

http://www.nanex.net/aqck2/4436/RevisitingFedRobbery.pdf


Synchronisation of Trading PC with NTP

The correct synchronization time is NTP. To avoid any issues with lag of the time signal, you should use

- the nearest available NTP time server (which is PTB)
- not connect to that time server via internet (approx. lag 25 msec) but via radio clock receiver (approx. lag 3 msec)

There are some products available from Meinberg that can be connected via USB and that will allow accurate synchronization of your PC. I do not know the accuracy level though.

USB Radio Clocks


Measuring the lag of the Data Feed

You can now compare the time stamps of the data feed to the synchronized time of your PC. The result will still suffer from a few errors

-> due to the lag of the radio signal from the PTB atomic clock
-> due to data processing
-> due to inaccurate synchronization of your PC
-> due inaccurate synchronization of the internal clock of the CME maching engine

However, these errors should only add up to something like 10 - 20 msec, which is good enough for monitoring the data feed lag, which is about 50 msec (100 msec referred to the round trip) or larger. You should even be able to show the current lag on your NinjaTrader tick-built chart as an indicator.

I am not one to usually disagree with @Fat Tails... but...

1) Yes you can measure latency with millisecond and even microsecond accuracy. It is trivial when using any API that sends the Exchange timestamp. The difference between your PC time and the Exchange time is your latency, or how old that update is when you receive it. The transmission time does not distort the results, it will simply be part of the results.

2) If you are in a data center, you will find NTP <1ms away. If you aren't in a data center, why bother with latency at all?

3) You don't need a hardware device when in a data center. Steadfast already has an NTP server and there are many others in Chicago that do as well. NTP will sync with any of them and the precision can be measured as less than 1ms.

Pretty much all of this is irrelevant, I think the simplest answer to the original question is just to use ping.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #19 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received

Steadfast is a stratum 1 source (GPS):

https://support.steadfast.net/Knowledgebase/Article/View/103/0/ntp-time-servers

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
 
  #20 (permalink)
Site Administrator
Manta, Ecuador
 
Futures Experience: Advanced
Platform: My own custom solution
Favorite Futures: E-mini ES S&P 500
 
Big Mike's Avatar
 
Posts: 46,238 since Jun 2009
Thanks: 29,350 given, 83,220 received


Output from ntpq peers, using Steadfast NTP, from one of my Equinix datacenter servers:

 
Code
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*lookingglass.ch 216.86.146.46    2 u    3   64  377    0.189    0.062   0.047
+ip46.216-86-146 .GPS.            1 u    5   64  377    0.310    0.011   0.021
+ntp.your.org    .CDMA.           1 u    6   64  377    1.379   -0.009   0.082
Shows 189 microseconds and 310 microseconds respectively (<1ms), this is round trip time. Also shows accuracy/precision of 6 microseconds and 1 microsecond.

The third entry is another stratum 1 server in Chicago that is public.

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.


Last edited by Big Mike; February 1st, 2014 at 11:36 AM.
Reply With Quote

Reply



futures io > > > How to measure data feed latencies between continents?

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Upcoming Webinars and Events (4:30PM ET unless noted)

Linda Bradford Raschke: Reading The Tape

Elite only

Adam Grimes: TBA

Elite only

NinjaTrader: TBA

January

Ran Aroussi: TBA

Elite only
     

Similar Threads
Thread Thread Starter Forum Replies Last Post
Best data feed for stocks Day Trading Fool Reviews of Brokers and Data Feeds 10 August 7th, 2016 10:54 AM
Data Feed: How do you confirm Accuracy vs Historical tick data provided. FAUGH45568 Reviews of Brokers and Data Feeds 2 February 14th, 2013 07:04 PM
WHat data feed for ICE US Data? samadams Reviews of Brokers and Data Feeds 7 September 20th, 2012 04:08 PM
A better retail data feed LittleCatJones Elite Automated Trading 6 April 27th, 2012 11:52 AM
Data feed comparison - tick data Big Mike Reviews of Brokers and Data Feeds 6 March 18th, 2010 01:52 PM


All times are GMT -4. The time now is 01:49 AM.

Copyright © 2017 by futures io, s.a., Av Ricardo J. Alfaro, Century Tower, Panama, +507 833-9432, info@futures.io
All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
no new posts
Page generated 2017-12-13 in 0.17 seconds with 40 queries on phoenix via your IP 54.92.201.232