How does the TCP Retransmission? Which formula is determined by how many packets will be re-sent? I understand it is set somewhere in the cubic tcp, but where?
Interested in how it works in Linux. I use Debian 8 and just looking dumps. I set up a connection using netcat to 27000 port. I usually do on the server that makes iptables drop all packets on port 27000 and send a packet(And look how many times the package has been re-sent.).
This is a very broad question.
No, this is not basically nor necessarily CUBIC.
Retransmission is firstly specified in TCP "basic" RFC 793 (1981), section 3.7 Data Communication, paragraph "Retransmission Timeout".
Since then, there's been many (really many[*]) enhancements. A very noticeable one is "slow-start", last specified by RFC 5681, but which roots go back to 1997 RFC 2001, "TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms".
There's no "one size fits all" in this domain, there's always trade-off. Plus "smart" algorithms will have more footprint (software + CPU use) so they may or may not be used or simply even available depending on the application (think embedded devices). And as these things are in the implementation (i.e. not seen in the exchanged data between host), you can never know for sure which host use which. You'll see the TCP window size and scale in the segments for instance, but you will not know by which algorithm it is managed.
As for Linux, it's supposed to default to PRR since 3.2. Prior to that was CUBIC, and prior BIC.
Though, default does not mean it's the only one available. On my debian stock 4.4.0 kernel, it's CUBIC:
jbmaillet@sumo:~$ cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
Though Reno is available too:
jbmaillet@sumo:~$ cat /proc/sys/net/ipv4/tcp_allowed_congestion_control
cubic reno
...and there's a dozen available in the "TCP: advanced congestion control" section of the kernel configuration.
*: https://en.wikipedia.org/wiki/TCP_congestion-avoidance_algorithm