How do I interpret the time delay and mismatched ack numbers in this tcpdump output?

by flyingL123   Last Updated January 13, 2018 01:00 AM

I'm trying to troubleshoot an issue I am seeing between a client server and a 3rd party server that the client is making http request to. I was able to get a tcpdump of one of the troubled calls, and something does look off, but I'm not an expert on this stuff and am unsure how to interpret what's happening. The tcpdump data for the single API request is shown below. The issue with this request is it took 58 seconds to respond, but the provider claims there is no issue on their side and they responded immediately. They see no evidence of a slow response. I'm trying to use the tcpdump data to see who is actually at fault here.

From the data it seems there was definitely a 58 second delay between tcp packets. Also, after the delay, you can see that a packet was sent, but it was out of sequence with the previous one sent before the delay. Can anyone offer any insight into what might be happening here? Why would I see such a long delay here, but the provider doesn't. Did my server miss part of the response, or did their server not send it. Basically, I'm just looking for some new ideas as to how to figure out what the issue is here.

I annotated the output below with what I believe each line means. This is my first time trying to interpret tcpdump data, so please feel free to let me know anything I've got wrong.

Also, I have some questions in the annotations. If anyone knows the answers to those, I would really appreciate the help.

CLIENT HTTP REQUEST TO SERVER
14:50:12.381402 IP (tos 0x0, ttl 64, id 65389, offset 0, flags [DF], proto TCP (6), length 816)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [P.], cksum 0x33e5 (incorrect -> 0x3ff9), seq 2273:3037, ack 13622, win 232, options [nop,nop,TS val 89028398 ecr 3124036660], length 764

SERVER ACKNOWLEDGES REQUEST
14:50:12.391546 IP (tos 0x18, ttl 56, id 32086, offset 0, flags [DF], proto TCP (6), length 52)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [.], cksum 0x5239 (correct), seq 13622, ack 3037, win 282, options [nop,nop,TS val 3124036667 ecr 89028398], length 0

SERVER SENDS DATA 13622:15070
14:50:15.736787 IP (tos 0x18, ttl 56, id 32093, offset 0, flags [DF], proto TCP (6), length 1500)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [.], cksum 0x0515 (correct), seq 13622:15070, ack 3037, win 282, options [nop,nop,TS val 3124037504 ecr 89028398], length 1448

CLIENT ACKNOWLEDGES 15070
14:50:15.773793 IP (tos 0x0, ttl 64, id 65390, offset 0, flags [DF], proto TCP (6), length 52)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30e9 (incorrect -> 0x4622), seq 3037, ack 15070, win 243, options [nop,nop,TS val 89029247 ecr 3124037504], length 0

SERVER SENDS DATA 15070:16518
14:50:19.088686 IP (tos 0x18, ttl 56, id 32099, offset 0, flags [DF], proto TCP (6), length 1500)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [.], cksum 0x819b (correct), seq 15070:16518, ack 3037, win 282, options [nop,nop,TS val 3124038342 ecr 89029247], length 1448

CLIENT ACKNOWLEDGES 16518
14:50:19.088725 IP (tos 0x0, ttl 64, id 65391, offset 0, flags [DF], proto TCP (6), length 52)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30e9 (incorrect -> 0x39ed), seq 3037, ack 16518, win 254, options [nop,nop,TS val 89030075 ecr 3124038342], length 0

SERVER SENDS DATA 16518:17966    
14:50:19.316699 IP (tos 0x18, ttl 56, id 32101, offset 0, flags [DF], proto TCP (6), length 1500)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [.], cksum 0x5872 (correct), seq 16518:17966, ack 3037, win 282, options [nop,nop,TS val 3124038399 ecr 89030075], length 1448

CLIENT ACKNOWLEDGES 17966    
14:50:19.316734 IP (tos 0x0, ttl 64, id 65392, offset 0, flags [DF], proto TCP (6), length 52)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30e9 (incorrect -> 0x33c7), seq 3037, ack 17966, win 266, options [nop,nop,TS val 89030132 ecr 3124038399], length 0

SERVER SENDS DATA 17966:19414    
14:50:19.326770 IP (tos 0x18, ttl 56, id 32102, offset 0, flags [DF], proto TCP (6), length 1500)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [.], cksum 0x6dcc (correct), seq 17966:19414, ack 3037, win 282, options [nop,nop,TS val 3124038401 ecr 89030132], length 1448

CLIENT ACKNOWLEDGES 19414    
14:50:19.326801 IP (tos 0x0, ttl 64, id 65393, offset 0, flags [DF], proto TCP (6), length 52)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30e9 (incorrect -> 0x2e0f), seq 3037, ack 19414, win 277, options [nop,nop,TS val 89030135 ecr 3124038401], length 0

-------58 SECONDS ELAPSE-----------

SERVER SENDS DATA 21608:21639 AND SAYS ITS FINISHED (BUT WHERE IS 19414:21607 ???)
14:51:17.461417 IP (tos 0x18, ttl 56, id 32113, offset 0, flags [DF], proto TCP (6), length 83)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [FP.], cksum 0x1a94 (correct), seq 21608:21639, ack 3037, win 282, options [nop,nop,TS val 3124052935 ecr 89030135], length 31

I ACKNOWLEDGE 19414 AGAIN
14:51:17.461451 IP (tos 0x0, ttl 64, id 65394, offset 0, flags [DF], proto TCP (6), length 64)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30f5 (incorrect -> 0xba1a), seq 3037, ack 19414, win 288, options [nop,nop,TS val 89044668 ecr 3124038401,nop,nop,sack 1 {21608:21640}], length 0

ODOO SENDS 19414:21608 (BECAUSE MY PREVIOUS ACKNOWLEDGEMENT ONLY WENT TO 19414? BUT WHY DID IT SEND TO 21608 INSTEAD OF 21607? IT ALREADY SENT 21608 PREVIOUSLY.)
14:51:17.471741 IP (tos 0x18, ttl 56, id 32114, offset 0, flags [DF], proto TCP (6), length 2246)
    158.69.226.30.443 > 104.236.71.114.36771: Flags [P.], cksum 0x397b (incorrect -> 0xf4a1), seq 19414:21608, ack 3037, win 282, options [nop,nop,TS val 3124052937 ecr 89044668], length 2194

I ACKNOWLEDGE 21640 (SHOULDN'T THIS BE 21639 SINCE THAT WAS THE LARGEST NUMBER RECEIVED? WHERE DOES 21640 COME FROM?)
14:51:17.471774 IP (tos 0x0, ttl 64, id 65395, offset 0, flags [DF], proto TCP (6), length 52)
    104.236.71.114.36771 > 158.69.226.30.443: Flags [.], cksum 0x30e9 (incorrect -> 0xb3af), seq 3037, ack 21640, win 306, options [nop,nop,TS val 89044671 ecr 3124052937], length 0
Tags : http tcp tcpdump


Related Questions


On-the-fly decoding HTTP bodies on Linux?

Updated October 05, 2015 07:00 AM

Human readable format for http headers with tcpdump

Updated September 12, 2015 04:00 AM

Where is the client ACK in this tcpdump output?

Updated January 12, 2018 21:00 PM

Broken page on only one server

Updated March 15, 2016 08:00 AM

How to capture http requests headers and body

Updated March 23, 2017 00:00 AM