f26efb65f2
The conn.Bind UDP sockets' send and receive buffers are now being sized to 7MB, whereas they were previously inheriting the system defaults. The system defaults are considerably small and can result in dropped packets on high speed links. By increasing the size of these buffers we are able to achieve higher throughput in the aforementioned case. The iperf3 results below demonstrate the effect of this commit between two Linux computers with 32-core Xeon Platinum CPUs @ 2.9Ghz. There is roughly ~125us of round trip latency between them. The first result is from commit 792b49c which uses the system defaults, e.g. net.core.{r,w}mem_max = 212992. The TCP retransmits are correlated with buffer full drops on both sides. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 4.74 GBytes 4.08 Gbits/sec 2742 285 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.74 GBytes 4.08 Gbits/sec 2742 sender [ 5] 0.00-10.04 sec 4.74 GBytes 4.06 Gbits/sec receiver The second result is after increasing SO_{SND,RCV}BUF to 7MB, i.e. applying this commit. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 6.14 GBytes 5.27 Gbits/sec 0 3.15 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.14 GBytes 5.27 Gbits/sec 0 sender [ 5] 0.00-10.04 sec 6.14 GBytes 5.25 Gbits/sec receiver The specific value of 7MB is chosen as it is the max supported by a default configuration of macOS. A value greater than 7MB may further benefit throughput for environments with higher network latency and lower CPU clocks, but will also increase latency under load (bufferbloat). Some platforms will silently clamp the value to other maximums. On Linux, we use SO_{SND,RCV}BUFFORCE in case 7MB is beyond net.core.{r,w}mem_max. Co-authored-by: James Tucker <james@tailscale.com> Signed-off-by: James Tucker <james@tailscale.com> Signed-off-by: Jordan Whited <jordan@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
44 lines
1.3 KiB
Go
44 lines
1.3 KiB
Go
/* SPDX-License-Identifier: MIT
|
|
*
|
|
* Copyright (C) 2017-2023 WireGuard LLC. All Rights Reserved.
|
|
*/
|
|
|
|
package conn
|
|
|
|
import (
|
|
"net"
|
|
"syscall"
|
|
)
|
|
|
|
// UDP socket read/write buffer size (7MB). The value of 7MB is chosen as it is
|
|
// the max supported by a default configuration of macOS. Some platforms will
|
|
// silently clamp the value to other maximums, such as linux clamping to
|
|
// net.core.{r,w}mem_max (see _linux.go for additional implementation that works
|
|
// around this limitation)
|
|
const socketBufferSize = 7 << 20
|
|
|
|
// controlFn is the callback function signature from net.ListenConfig.Control.
|
|
// It is used to apply platform specific configuration to the socket prior to
|
|
// bind.
|
|
type controlFn func(network, address string, c syscall.RawConn) error
|
|
|
|
// controlFns is a list of functions that are called from the listen config
|
|
// that can apply socket options.
|
|
var controlFns = []controlFn{}
|
|
|
|
// listenConfig returns a net.ListenConfig that applies the controlFns to the
|
|
// socket prior to bind. This is used to apply socket buffer sizing and packet
|
|
// information OOB configuration for sticky sockets.
|
|
func listenConfig() *net.ListenConfig {
|
|
return &net.ListenConfig{
|
|
Control: func(network, address string, c syscall.RawConn) error {
|
|
for _, fn := range controlFns {
|
|
if err := fn(network, address, c); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
return nil
|
|
},
|
|
}
|
|
}
|