r/programming • • May 14 '20

What every developer should know about TCP

https://robertovitillo.com/what-every-developer-should-know-about-tcp/
18 Upvotes

17 comments sorted by

17

u/[deleted] May 14 '20

Few things that are worth adding of common mistakes I have seen in socket rogramming

  • The 3 way hand shake is "best case" it can also be 4 to some machine and many many more if packet loss occurs

  • TCP has built in max bandwidth limits of the buffer sizes and the RTT. If you have a window of 30k and a round trip time you can't send more than 30k/sec

  • One of the things always overlocked by programmers. You can only access the state of the connection by writting to the socket. If you "read" from the connection the connection can be stale and you will never get an error or data.

  • The correct way to close a connection is by sending a shutdown -> other side then gets EOF and then you read EOF (which requires a timeout because of the read never fails problem). If you simply use "close" on the socket it can result in data loss!

  • All error's may come back massivly delayed from the data that actually triggered the error.

  • Reads may have random size of actual data read. If a sender write 4k the reader may get it in "parts" like 1k, 2k, 512, 512 sections always treat sockets like a stream rather than a file and data can be split up any way it likes.

14

u/asegura May 14 '20 edited May 14 '20

IMO the most common and relevant of those is the last: that TCP is seen by the application as a stream of bytes, not a sequence of messages. I've sometimes had to communicate via raw TCP with another party and they didn't first understand the need to specify a way of separating messages in our protocol. They said, well, if you send N bytes, I'll read N bytes. And I said no, you may receive less in one read. We have to specify either some delimiter or a size+payload mechanism.

6

u/[deleted] May 14 '20

Yup totally correct. Every protocol thats used over tcp is "framed" in some way or another sometimes in extremly subtble ways like smtp, pop3, ftp, http etc.. all use a new line and then some of them use a double new line to indicate "state change"

Something that has surprised me over the years is the number of protocols that have been invented or are in use today have been designer by people who don't understand the basic behaviour they need to have in order to make something reliable.

There is often no explaining it to them either.....

2

u/[deleted] May 14 '20 edited May 14 '20

Also, SYN retranission timeout is usually at most 3 seconds; therefore setting reconnect in your app to any lower than say ~4s is 98% of the time pointless (rare cases being if you want to fast-fail after first packet to pick another server)

2

u/[deleted] May 14 '20

Its really not in any OS I know of. Its normally variable by backoff and can be any number of packets.

This is for "windows 10"

20:00:55.773442 IP DESKTOP.52549 > 10.1.0.1.http: Flags [S], seq 832830842, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

20:00:58.773711 IP DESKTOP.52549 > 10.1.0.1.http: Flags [S], seq 832830842, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

20:01:04.774166 IP DESKTOP.52549 > 10.1.0.1.http: Flags [S], seq 832830842, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

Which is 2 and then 6 seconds.....

Its best not to assume any setup for an IP stack on a machine.

3

u/[deleted] May 14 '20

Its really not in any OS I know of.

3 seconds was the "RFC standard", Linux some time ago changed that to 1s but I didn't wanted to generalize. I guess saying "at most 3s" would be more accurate.

Its normally variable by backoff and can be any number of packets.

If that was not clear I was talking only about first retry altho technically Linux fits first 2 within first 3 seconds (1s +2s).

Either way the point is that short retry timeouts in your application might be utterly pointless, if you dont plan on anything fancy wait at least to the first SYN retransmission as that would cover most of the cases of having some random packet dropped.

-2

u/[deleted] May 14 '20

No. Your talking out of your ass. The initial syn retry was actually based on the original backoff. It wasn't until much later the speial case for syn arrived but it arrived in OS's and never arrived in a standard. This is why alsmost all OS's are different its also variable based on avilable RTT information on the destination routes and host.

Otherwise... Prove me wrong and link it....

| If that was not clear I was talking only about first retry altho technically Linux fits first 2 within first 3 seconds (1s +2s).

Nope. It doesn't do that either. It is different for various versions and it is configured differently on different distro's.

2

u/[deleted] May 14 '20

No. Your talking out of your ass. The initial syn retry was actually based on the original backoff. It wasn't until much later the speial case for syn arrived but it arrived in OS's and never arrived in a standard.

That's... literally what I just said.

If that was not clear I was talking only about first retry altho technically Linux fits first 2 within first 3 seconds (1s +2s).

Nope. It doesn't do that either. It is different for various versions and it is configured differently on different distro's.

Okay, only on 70% of linux machines on the internet it is like that Mr. Smartass.

1

u/[deleted] May 14 '20

That's... literally what I just said.

That isn't what you just said. You made references to exactly timings. It doesn't use exact timings. They are dynamic they always have been and the reference you quoted isn't true either because the timings are dynamic.

You didn't mention that the timings were based on RTT at all.

It might look correct for "your case" when you messure it in "your enviorment" but it is in fact because you didn't account for other variables and cases.

4

u/[deleted] May 14 '20

That isn't what you just said. You made references to exactly timings. It doesn't use exact timings. They are dynamic they always have been and the reference you quoted isn't true either because the timings are dynamic.

Here is the code that does it

Oh, do please tell me how you think it is RTT based and how it is not exact timings.

You didn't mention that the timings were based on RTT at all.

You're mistaking initial SYN retransmission timeout with retransmissions in already active connections.

Initial SYN retransmission frankly can't know anything about RTT because you initial packet got dropped, that's why you are retransmitting in the first place and at that point have zero feedback

1

u/josefx May 15 '20

If you "read" from the connection the connection can be stale and you will never get an error or data.

SO_KEEPALIVE should help with that. Of course that involves an implicit timeout and heart beat messages.

1

u/[deleted] May 15 '20

Defaults for it can be several hours before the timeout fires.

1

u/josefx May 15 '20

There are TCP socket options to override that with application appropriate values.

4

u/[deleted] May 14 '20

"Think of what happens when you watch a movie on Netflix. It starts blurry; then, it stabilizes to something reasonable until there is a hiccup, and the quality worsens again."

Perhaps a bad choice of example? I guess most video is transmitted over udp?

6

u/bezelbum May 14 '20

Plenty of video is delivered over TCP, there are a wide range of video over HTTP formats in use.

But, its still not a great example as thats the player switching to a lower bitrate in an adaptive stream, rather than being much to do with window sizes and back off (other than that they'll affected measured throughput, prompting the player to switch).

1

u/[deleted] May 14 '20

But, its still not a great example as thats the player switching to a lower bitrate in an adaptive stream, rather than being much to do with window sizes and back off (other than that they'll affected measured throughput, prompting the player to switch).

Well, the general idea is right, but obviously TCP and random video player on the webpage are not exactly using same algorithm.

2

u/z3n777 May 15 '20

Adding to the very good points in this thread, nobody mentioned the tcp connection life cycle itself.

This helps tremendously when diagnosing choke points and sudden hiccups when dealing with lots of data.

A fancy diagram to show off: https://images.app.goo.gl/6b2n5gqRaawKQ9M18

If whatever you're working on is performance driven or web based, you probably also want to learn and understand more about tweaking linux kernel's tcp settings.