CQG feed on Steadfast box dropping

October 2nd, 2012, 02:36 PM

Anyone running CQG on a steadfast box and seeing periodic problems?

I'm running Multicharts 8 w/ CQG and am experiencing time outs per the logs that are shutting down the CQG datafeed without warning. Running a ping to the CQG servers in the command line shows consistent sub 1MS pings while the CQG API continues to print warnings about latency.

Last night was the 3rd time in the last 4 weeks I've lost the datafeed while in trades. I was actively RDP'd into the box when the feed stopped, so I know the box never lost full connectivity. Luckily its been in slow markets to date, but I fear getting caught in a fast market.

Would love to hear feedback of other members who may have seen similar issues. My setup is relatively new - I've only been running for 3 months.

Neither CQG or Multicharts support has been able to provide any answers other than suggesting that my box is losing connectivity.

October 2nd, 2012, 03:06 PM

I'm also running on Steadfast with MC/CQG and I looked in my logs and found the following:

Code

N [2012-10-02 08:13:28,408] - NETS: [01] A1001: ping reply is being delayed for 2028 ms
 WARN [2012-10-02 08:13:30,467] - NETS: [01] A1001: ping reply is being delayed for 4087 ms
 WARN [2012-10-02 08:13:34,554] - NETS: [01] A1001: ping reply is being delayed for 8175 ms
 INFO [2012-10-02 08:13:34,554] - Connection status changed: 
 Previous Status: Connection to 'SecBuf' at 'cqginsfc003i.cqgrdnet.com' is 'normal'.
 New status: Connection to 'SecBuf' at 'cqginsfc003i.cqgrdnet.com' is 'slow'.

Looks like I'm also having delays that I wasn't aware of. So far I haven't had CQG go belly up on me but I'd really hate to be delayed for 8 seconds in fast moving market...

Hope others can chime in on this thread as well.

October 2nd, 2012, 04:02 PM

Interesting, that is the exact time my logs show the long pings:

WARN [2012-10-02 08:13:39,561] - NETS: [01] A1001: ping reply is being delayed for 16240 ms
WARN [2012-10-02 08:13:55,832] - NETS: [01] A1001: ping reply is being delayed for 32510 ms
WARN [2012-10-02 08:14:28,358] - NETS: [01] A1001: ping reply is being delayed for 65035 ms
WARN [2012-10-02 08:15:33,410] - NETS: [01] A1001: ping reply is being delayed for 130085 ms
WARN [2012-10-02 08:17:43,499] - NETS: [01] A1001: ping reply is being delayed for 260171 ms
INFO [2012-10-02 08:20:52,696] - ASLU: ReportClientActivity(140877,*****) method calling at auth service
INFO [2012-10-02 08:20:52,712] - ASLU: ReportClientActivity method at returned request status: Success
WARN [2012-10-02 08:22:03,692] - NETS: [01] A1001: ping reply is being delayed for 520358 ms
WARN [2012-10-02 08:24:05,949] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or hung: no response to test message for 1029 msec, queue: 32 bytes
WARN [2012-10-02 08:24:06,979] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or hung: no response to test message for 2059 msec, queue: 64 bytes
WARN [2012-10-02 08:24:09,022] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or hung: no response to test message for 4102 msec, queue: 304 bytes
WARN [2012-10-02 08:24:13,078] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or hung: no response to test message for 8158 msec, queue: 432 bytes
WARN [2012-10-02 08:24:20,753] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or overloaded: core->subsystem message delivery took 15845 msec, queue: 1008 bytes
WARN [2012-10-02 08:24:20,753] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or overloaded: core->subsystem message delivery took 7686 msec, queue: 576 bytes
WARN [2012-10-02 08:24:20,753] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or overloaded: core->subsystem message delivery took 3584 msec, queue: 272 bytes
WARN [2012-10-02 08:24:20,753] - NETS: [01] SUBS2(811ff002-edfc-4d84-83ee-0e2aea8c6d1d - Subsystem #02)-pro: subsystem is slow or overloaded: core->subsystem message delivery took 1525 msec, queue: 32 bytes
WARN [2012-10-02 08:30:44,062] - NETS: [01] A1001: ping reply is being delayed for 1040717 ms
ERROR [2012-10-02 08:30:44,062] - NETS: [01] A1001: pings are not answered, stopping pinging

17 minutes later is when the datafeed cut out entirely. So the question is was it Steadfast or CQG? Curious if anyone not on Steadfast has logs from the same time (timestamps are weird - it actually corresponds to around 3AM CST this morning)?

October 2nd, 2012, 04:17 PM

It might be CQG, no problem today with Steadfast (at least in chi18).

October 2nd, 2012, 04:21 PM

sam028

It might be CQG, no problem today with Steadfast (at least in chi18).

Hi Sam, how do you find out 'chi18'? I assume that's some sort of logical node or domain you're on.

October 2nd, 2012, 05:24 PM

Thanks Sam.

Curious to hear from any CQG/Multicharts users outside of Steadfast. Any issues or suspicious log entries around 3AM CST?

You'd know for sure if you went down - requires a restart to get reconnected. Looks like I went down while Nick did not even though we both suffered similar issues.

October 2nd, 2012, 05:54 PM

illusivenick

Hi Sam, how do you find out 'chi18'? I assume that's some sort of logical node or domain you're on.

They have few facilities in Chicago, chi01/chi02/chi18, so if you have equipments hosted by them, you know where they are (which facility, row & rack, switches ports used, ...).
Maybe both of you were in chi01/chi02, but they didn't talk about any network issue today, so I think it's on CQG side.

October 3rd, 2012, 12:28 PM

I found out I'm also on chi18.

Steadfast is having some network issues.

When I ping CQG servers it's taking about 27ms when it should be sub 1ms.

Code

C:\Users\Administrator>ping cqginsfc003i.cqgrdnet.com

Pinging cqginsfc003i.cqgrdnet.com [64.208.51.213] with 32 bytes of data:
Reply from 64.208.51.213: bytes=32 time=27ms TTL=119
Reply from 64.208.51.213: bytes=32 time=27ms TTL=119
Reply from 64.208.51.213: bytes=32 time=27ms TTL=119
Reply from 64.208.51.213: bytes=32 time=27ms TTL=119

Ping statistics for 64.208.51.213:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 27ms, Maximum = 27ms, Average = 27ms

Here is a tracert

Code

C:\Users\Administrator>tracert 64.208.51.213

Tracing route to 64.208.51.213 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  ip3.50-31-26.static.steadfastdns.net [50.31.26.3
]
  2    <1 ms    <1 ms    <1 ms  ip77.208-100-32.static.steadfastdns.net [208.100
.32.77]
  3    <1 ms    <1 ms    <1 ms  xe-7-3-0-383.chi12.ip4.tinet.net [216.221.156.17
7]
  4    23 ms    23 ms    23 ms  xe-2-0-0.dal33.ip4.tinet.net [89.149.185.85]
  5    39 ms    39 ms    63 ms  as6461.ip4.tinet.net [77.67.71.166]
  6    25 ms    43 ms    25 ms  xe-3-0-0.cr2.dfw2.us.above.net [64.125.30.94]
  7    27 ms    27 ms    27 ms  xe-2-0-0.cr2.ord2.us.above.net [64.125.25.137]
  8    64 ms    27 ms    27 ms  xe-0-1-0.er2.ord2.us.above.net [64.125.25.110]
  9    27 ms    27 ms    27 ms  64.124.202.146.t00610.above.net [64.124.202.146]

 10    27 ms    27 ms    27 ms  64.208.51.213

Trace complete.

Steadfast is aware of the problem and is trying to engineer around it. They have been very responsive and helpful on this issue I just hope they can solve it quickly.

Here is what they said:

Quoting

It does seem that Inteliquent/Tinet does not peer with Abovenet in Chicago, thus the traffic going out them needs to go to Dallas to reach Abovenet's network. This route is not being corrected by the FCP due to some asymmetric routes that are causing some confusion, but we will look into engineering around this issue tomorrow.

I'm definitely not happy paying for a server in Chicago and my traffic is whizzing by my house in Colorado to Dallas and back to Chicago... but I'm hopeful steadfast will fix this soon. I'm curious if others on chi18 are having the same issue.

October 3rd, 2012, 03:41 PM

Things look like they are working good now. Hope it stays this way.

Code

C:\Users\Administrator>tracert 64.208.51.213

Tracing route to 64.208.51.213 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  ip3.50-31-26.static.steadfastdns.net [50.31.26.3
]
  2    <1 ms    <1 ms    <1 ms  ip77.208-100-32.static.steadfastdns.net [208.100
.32.77]
  3    <1 ms    <1 ms    <1 ms  xe-10-1-2.edge1.Chicago2.Level3.net [4.71.248.20
1]
  4    <1 ms    <1 ms    <1 ms  4.69.158.234
  5    <1 ms    <1 ms    <1 ms  ae-2-52.edge3.Chicago3.Level3.net [4.69.138.168]

  6    <1 ms    <1 ms    <1 ms  xe-1-2-0.er1.ord7.us.above.net [64.125.13.237]
  7     2 ms    <1 ms    <1 ms  xe-2-2-0.cr1.ord2.us.above.net [64.125.26.249]
  8    <1 ms    <1 ms    <1 ms  xe-1-0-0.cr2.ord2.us.above.net [64.125.28.238]
  9    <1 ms    <1 ms    <1 ms  xe-1-1-0.er2.ord2.us.above.net [64.125.26.190]
 10     1 ms    <1 ms    <1 ms  64.124.202.146.t00610.above.net [64.124.202.146]

 11    <1 ms    <1 ms    <1 ms  64.208.51.213

Trace complete.

C:\Users\Administrator>ping cqginsfc003i.cqgrdnet.com

Pinging cqginsfc003i.cqgrdnet.com [64.208.51.213] with 32 bytes of data:
Reply from 64.208.51.213: bytes=32 time<1ms TTL=119
Reply from 64.208.51.213: bytes=32 time<1ms TTL=119
Reply from 64.208.51.213: bytes=32 time<1ms TTL=119
Reply from 64.208.51.213: bytes=32 time<1ms TTL=119

Ping statistics for 64.208.51.213:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

CQG feed on Steadfast box dropping

Discussion in Brokers

CQG feed on Steadfast box dropping