# OB-UDPST Open Broadband-UDP Speed Test (OB-UDPST) is a client/server software utility to demonstrate one approach of doing IP capacity measurements as described by: - Broadband Forum TR-471 (07/2020): _Maximum IP-Layer Capacity Metric, Related Metrics, and Measurements_, BBF TR-471, https://www.broadband-forum.org/technical/download/TR-471.pdf - ITU-T Recommendation Y.1540 (12/2019): _Internet protocol data communication service - IP packet transfer and availability performance parameters_, ITU-T Y.1540, https://www.itu.int/rec/T-REC-Y.1540-201912-I/en - ITU-T Y-series Supplement 60 (2020): _Interpreting ITU-T Y.1540 maximum IP-layer capacity measurements_, ITU-T Y.Sup60, https://www.itu.int/rec/T-REC-Y.Sup60/en - ETSI Technical Specification 103 222 Part 2 (08/2019): _Reference benchmarking and KPIs for High speed internet_, ETSI TS 103 222-2, V.1.2.1, https://www.etsi.org/deliver/etsi_ts/103200_103299/10322202/01.02.01_60/ts_10322202v010201p.pdf - IETF RFC 9097: _Metrics and Methods for One-way IP Capacity_, https://datatracker.ietf.org/doc/html/rfc9097 - ETSI Technical Report 103 702 (2020-10): _Speech and multimedia Transmission Quality (STQ); QoS parameters and test scenarios for assessing network capabilities in 5G performance measurements_, ETSI TR 103 702, V.0.1.0 https://www.etsi.org/deliver/etsi_tr/103700_103799/103702/01.01.01_60/tr_103702v010101p.pdf ## Overview Utilizing an adaptive transmission rate, via a pre-built table of discreet sending rates (from 0.5 Mbps to 10 Gbps), UDP datagrams are sent from client to server or server to client to determine the maximum available IP-layer capacity between them. The load traffic is only sent in one direction at a time, and status feedback messages are sent periodically in the opposite direction. For upstream tests, the feedback messages from the server instruct the client on how it should adjust its transmission rate based on the presence or absence of sufficient sequence errors or delay variation changes observed by the server. For downstream tests, the feedback messages simply communicate any sequence errors or delay variation changes observed by the client. In either case, the server is the host executing the algorithm that determines the rate adjustments made during the test. This centralized approach allows the rate adjustment algorithm to be more easily enhanced and customized to accommodate diverse network services and protocols. To that end, and to encourage additional testing and experimentation, the software has been structured so that virtually all of the settings and thresholds used by the algorithm are currently available as client-side command-line parameters (allowing modification beyond the current set of default values updated in Release 7.2.1). By default, both IPv4 and IPv6 tests can be serviced simultaneously when acting as a server. When acting as a client, testing is performed in one address family at a time. Also, the default behavior is that jumbo datagram sizes (datagrams that would result in jumbo frames) are utilized for sending rates above 1 Gbps. The expectation is that jumbo frames can be accommodated end-to-end without fragmentation. For sending rates below 1 Gbps, or all sending rates when the `-j` option is used, the default UDP payload size is 1222 bytes. This size was chosen because although it is relatively large it should still avoid any unexpected fragmentation due to intermediate protocols or tunneling. Alternatively, the `-T` option is available to allow slightly larger default datagrams that would create full packets when a traditional 1500 byte MTU is available end-to-end. These larger sizes are optional because several consumer-grade routers have been shown to handle packet fragmentation, from a performance perspective, very poorly. Note that both the `-j` and `-T` options must match between the client and server or the server will reject the test request. All delay variation values, based on One-Way Delay (OWDVar) or Round-Trip Time (RTTVar), indicate the delays measured above the most recent minimum. Although OWDVar is measured for each received datagram, RTTVar is only sampled and is measured using each status feedback message. If specifying the use of OWDVar instead of RTTVar (via the `-o` option), there is no requirement that the clocks between the client and server be synchronized. The only expectation is that the time offset between clocks remains nominally constant during the test. **Usage examples for server mode:** ``` $ udpst Service client requests received on any interface $ udpst -x Service client requests in background (as a daemon) on any interface $ udpst Only service client requests on interface with the specified IP address $ udpst -a Only service requests that utilize a matching authentication key ``` *Note: The server must be reachable on the UDP control port [default **25000**] and all UDP ephemeral ports (**32768 - 60999** as of the Linux 2.4 kernel, available via `cat /proc/sys/net/ipv4/ip_local_port_range`).* **Usage examples for client mode:** ``` $ udpst -u Do upstream test from client to server (as hostname or IP address) $ udpst -d Do downstream test to client from server (as hostname or IP address) $ udpst -s -u Do upstream test and only show the test summary at the end $ udpst -r -d Do downstream test and show loss ratio instead of delivered percentage $ udpst -f json -d Do downstream test and use JSON format for test results ``` *Note: The client can operate behind a firewall (w/NAT) without any port forwarding or DMZ designation because all connections with the server are initiated by the client.* **To show the sending rate table:** ``` $ udpst -S ``` *The table shows dual transmitters for each index (row) because two separate sets of transmission parameters are required to achieve the desired granularity of sending rates. Each of the transmitters has its own interval timer and one also includes an add-on fragment at the end of each burst (again, for granularity). By default the table is shown for IPv4 with jumbo datagram sizes enabled above 1 Gbps. Including `-6` (for IPv6 only) and/or `-j` (to disable all jumbo sizes) shows the table adjusted for those specific scenarios.* **For a list of all options:** ``` $ udpst -? ``` **Default values:** Note that default values have been provided for all essential settings and thresholds, and these values are recommended for use at the beginning of any test campaign. The set of default values are subject to re-consideration with further testing, and may be revised (as were two of the values in Release 7.2.1: the sequence error threshold was increased to 10 and the consecutive feedback interval to 3; this is equivalent to using -q10 and -c3 in earlier releases, and these changes to the default values help to maintain fast ramp-up when non-congestion related packet losses are present). For example, the option to send a fixed rate testing after conducting a search for the Maximum IP-Layer Capacity will provide a stable measurement of loss, delay variation, etc., using -I index: ``` $ udpst -d -I 958 Do downstream test from server (as hostname or IP address) to client at a fixed rate of 958 Mbps (0.99 times the Maximum IP-layer Capacity for udpst on a 1 Gbps access link using one VLAN tag). ``` There are circumstances when changes to the defaults are warranted, such as extremely long paths with unknown cross traffic, high levels of competing traffic, testing over radio links with highly variable loss and delay, and test paths that exhibit bi-modal rate behavior in repeated tests. In these circumstances, the following client-side command-line parameters have been useful: ``` $ udpst -? ... (j) -j Disable jumbo datagram sizes above 1 Gbps -T Use datagram sizes for traditional (1500 byte) MTU -i count Display bimodal maxima (specify initial sub-intervals) (c) -R Ignore Out-of-Order/Duplicate datagrams (m,i) -I [@]index Index of sending rate (see '-S') [Default @0 = ] (m) -t time Test interval time in seconds [Default 10, Max 3600] (c) -U delvar Upper delay variation threshold in ms [Default 90] (c) -c thresh Congestion slow adjustment threshold [Default 3] (c) -h delta High-speed (row adjustment) delta [Default 10] (c) -q seqerr Sequence error threshold [Default 10] ``` See the following publication (which is updated frequently) for more details on testing in the circumstances described above: - ITU-T Y-series Supplement 60 (2020): _Interpreting ITU-T Y.1540 maximum IP-layer capacity measurements_, ITU-T Y.Sup60, https://www.itu.int/rec/T-REC-Y.Sup60/en ## Building OB-UDPST To build OB-UDPST a local installation of CMake is required. Please obtain it for your particular build system either using the locally available packages or consult with [https://cmake.org] for other download options. ``` $ cmake . $ make ``` *Note: Authentication functionality uses a command-line key along with the OpenSSL crypto library to create and validate a HMAC-SHA256 signature (which is used in the setup request to the server). Although the makefile will build even if the expected directory is not present, disabling the key and library dependency, the additional files needed to support authentication should be relatively easy to obtain (e.g., `sudo apt-get install libssl-dev`).* ## Test Processing Walkthrough (all messaging and PDUs use UDP) On the server, the software is run in server mode in either the foreground or background (as a daemon) where it awaits Setup requests on its UDP control port. The client, which always runs in the foreground, requires a direction parameter as well as the hostname or IP address of the server. The client will create a connection and send a Setup request to the server's control port. It will also start a test initiation timer so that if the initiation process fails to complete, the client will display an error message to the user and exit. **Setup Request** When the server receives the Setup request it will validate the request by checking the protocol version, the jumbo datagram support indicator, and the authentication data if utilized. If the Setup request must be rejected, a Setup response will be sent back to the client with a corresponding command response value indicating the reason for the rejection. If the Setup request is accepted, a new test connection is allocated and initialized for the client. This new connection is associated with a new UDP socket allocated from the UDP ephemeral port range. A timer is then set for the new connection as a watchdog (in case the client goes quiet) and a Setup response is sent back to the client. The Setup response includes the new port number associated with the new test connection. Subsequently, if a Test Activation request is not received from the client on this new port number, the watchdog will close the socket and deallocate the connection. **Setup Response** When the client receives the Setup response from the server it first checks the command response value. If it indicates an error it will display a message to the user and exit. If it indicates success it will build a Test Activation request with all the test parameters it desires such as the direction, the duration, etc. It will then send the Test Activation request to the UDP port number the server communicated in the Setup response. **Test Activation request** After the server receives the Test Activation request on the new connection, it can choose to accept, ignore or modify any of the test parameters. When the Test Activation response is sent back, it will include all the test parameters again to make the client aware of any changes. If an upstream test is being requested, the transmission parameters from the first row of the sending rate table are also included. Note that the server additionally has the option of completely rejecting the request and sending back an appropriate command response value. If activation continues, the new connection is prepared for an upstream OR downstream test with either a single timer to send status PDUs at the specified interval OR dual timers to send load PDUs based on the first row of the sending rate table. The server then sends a Test Activation response back to the client, the watchdog timer is updated and a test duration timer is set to eventually stop the test. The new connection is now ready for testing. **Test Activation response** When the Test Activation response is received back at the client it first checks the command response value. If it indicates an error it will display a message to the user and exit. If it indicates success it will update any test parameters modified by the server. It will then prepare its connection for an upstream OR downstream test with either dual timers set to send load PDUs (based on the starting transmission parameters sent by the server) OR a single timer to send status PDUs at the specified interval. The test initiation timer is then stopped and a watchdog timer is started (in case the server goes quiet). The connection is now ready for testing. **Testing** Testing proceeds with one end point sending load PDUs, based on transmission parameters from the sending rate table, and the other end point sending status messages to communicate the traffic conditions at the receiver. Each time a PDU is received the watchdog timer is reset. When the server is sending load PDUs it is using the transmission parameters directly from the sending rate table via the index that is currently selected (which was based on the feedback in its received status messages). However, when the client is sending load PDUs it is not referencing a sending rate table but is instead using the discreet transmission parameters that were communicated by the server in its periodic status messages. This approach allows the server to always control the individual sending rates as well as the algorithm used to decide when and how to adjust them. **Test Stop** When the test duration timer on the server expires it sets the connection test action to STOP and also starts marking all outgoing load or status PDUs with a test action of STOP. When received by the client, this is the indication that it should finalize testing, display the test results, and also mark its connection with a test action of STOP (so that any subsequently received PDUs are ignored). With the test action of the connection set to STOP, the very next expiry of a send timer for either a load or status PDU will cause the client to schedule an immediate end time to exit. It then sends that PDU with a test action of STOP as confirmation to the server. When the server receives this confirmation in the load or status PDU, it schedules an immediate end time for the connection which closes the socket and deallocates it. **More Info** For much more detail on the test protocol, see the ./protocol.md file. Also, an Internet-Draft, https://datatracker.ietf.org/doc/html/draft-morton-ippm-capacity-metric-protocol-02 describes Protocol Version 8 in even more detail. ## Linux Socket Buffer Optimization For very high speeds (typically above 1 Gbps), the socket buffer maximums of the Linux kernel can be increased to reduce possible datagram loss. As an example, the following could be added to the /etc/sysctl.conf file: ``` net.core.rmem_max=16777216 net.core.wmem_max=16777216 ``` To activate the new values without a reboot: ``` $ sudo sysctl -w net.core.rmem_max=16777216 $ sudo sysctl -w net.core.wmem_max=16777216 ``` *The default software settings will automatically take advantage of the increased socket buffering available. However, the command-line option `-b` can be used if even higher buffer levels should be explicitly requested for each socket.* ## Considerations for Older Hardware and Low-End Devices There are two general categories of devices in this area, 1) those that operate normally but lack the horsepower needed to reach a specific sending rate and 2) those that are unable to function properly because they do not support the required clock resolution. In this case, an error message is produced (“ERROR: Clock resolution (xxxxxxx ns) out of range”) and the software exits because without the expected interval timer, sending rates would be skewed and the rate-adaption algorithm would not function properly. Devices in the first category may be helped by the `-T` option when a traditional 1500 byte MTU is available end-to-end between client and server. This option will increase the default datagram payload size so that full 1500 byte packets are generated. This reduces both the socket I/O and network packet rates (see `-S` vs. `-ST` output). Another beneficial option in these cases is `-j`. This is to disable all jumbo datagram sizes and prevent any possible fragmentation. This can happen with a very underpowered device when the server, attempting to drive the client higher and higher, ends up at sending rates above 1 Gbps. One specific device in this category worth mentioning is a Raspberry Pi 4 running Raspberry Pi OS (previously called Raspbian). Testing has shown that to reach a 1 Gbps sending rate, it was necessary to set the CPU affinity of udpst to avoid CPU 0 (the CPU handling network interrupts when *irqbalance* is not used). Additionally, and especially when using a 32-bit Raspbian, both the `-T` and `-j` options were also needed. Although these last two options were not a hard requirement with a 64-bit Raspbian and 64-bit udpst, they are still recommended for ANY device in this general category (along with a 64-bit environment if applicable). The command to utilize these recommendations is: ``` taskset -c 1-3 udpst -u -T -j ``` For devices in the second category (unsupported timer resolution), a compile-time option (DISABLE_INT_TIMER) is available that does not rely on an underlying system interval timer. However, the trade-off for this mode of operation is that it results in high CPU utilization. But, clients running on older or low-capability hosts may be able to execute tests where they otherwise would not. ``` $ cmake -D DISABLE_INT_TIMER=ON . ``` *Note: Because of the increased CPU utilization, this option is not recommended for standard server operation.* ## JSON Output For examples of the JSON output fields see the included sample files named "udpst-*.json". Available JSON output options include `-f json` (unformatted), `-f jsonb` (brief & unformatted), and `-f jsonf` (formatted). To significantly reduce the size of the JSON output, option `-s` (omit sub-interval results) can be combined with `-f jsonb` (omit static input fields). Included in the output is a numeric ErrorStatus field (which corresponds with the software exit status) as well as a text ErrorMessage field. If a test completes normally without incident the error status will be 0 (zero) and the error message will be empty. If a test starts but a soft error or warning occurs, the error status will be 1 (one) and the most recent warning message will be included. If a hard error or failure occurs the error status will be -1 (negative one) and an error message will be included. The file "ob-udpst_output_mapping...pdf" provides a mapping between JSON key names, TR-471 names, TR-181 names, and the ob-udpst STDOUT names for various results. *Note: When stdout is not redirected to a file, JSON may appear clipped due to non-blocking console writes.* ## Local Interface Traffic Rate Where applicable, it is possible to also output the local interface traffic rate (in Mbps) via the `-E intf` option. This can be informative when trying to account for external traffic that may be consuming a non-trivial amount of the interface bandwidth and competing with the measurement traffic. The rate is obtained by querying the specific interface byte counters that correspond with the direction of the test (i.e., `tx_bytes` for upstream tests and `rx_bytes` for downstream tests). These values are obtained from the sysfs path `/sys/class/net//statistics`. An additional associated option `-M` is also available to override normal behavior and use the interface rate instead of the measurement traffic to determine a maximum. When the `-E intf` option is utilized, the console output will show the interface name in square brackets in the header info and the Ethernet rate of the interface in square brackets after the L3/IP measured rate. When JSON output is also enabled, the interface name appears in "Interface" and the interface rate is in "InterfaceEthMbps". When this option is not utilized, these JSON fields will contain an empty string and zero respectively. ## Server Bandwidth Management The `-B mbps` option can be used on a server to designate a maximum available bandwidth. Typically, this would specify the speed of the interface servicing tests and is managed separately for upstream and downstream (i.e., `-B 10000` indicates that 10 Gbps is available in each direction). One scenario to achieve better server utilization is to run multiple software instances on a machine with each one bound to a different physical interface (and this option set for the respective speed of each). Another scenario is to run multiple instances where each handles a portion of the bandwidth of a single physical interface. For example, 10 instances with `-B 10000` for a 100G connection. This can be done by either binding each to a different IP alias or configuring them to use different control ports (e.g., `-p 25000`, `-p 25001`, `-p 25002`,...). And although single threaded, each running instance will support multiple simultaneous overlapping tests (~128 by default). When configured on the server, clients will also need to utilize the `-B mbps` option in their test request to indicate the maximum bandwidth they require from the server for accurate testing. For example, a client connected via a 300 Mbps Internet service would specify `-B 300`. If the server's available bandwidth is unable to accommodate what the client requires, due to tests already in progress, the test request is rejected and the client produces an error indicating this as the cause. The client can then retry the test a little later, when server bandwidth may be available, or immediately try either an alternate software instance (per the scenarios described above) or a different server altogether. *Note: This option does not alter the test methodology or the rate adjustment algorithm in any way. It only provides test admission control to better manage the server's network bandwidth.* ## Increasing the Starting Sending Rate (Considerations) While the `-I index` option designates a fixed sending rate, it is also possible to set the starting rate with load adjustment enabled. Option `-I @index` allows selection of a higher initial sending rate starting at the specified index in the sending rate table (with the default equivalent to `-I @0` and shown as ``). Note that with or without the `@` character prefix, this option relies on an index from the sending rate table (see `-S` for index values). For maximum capacities up to 1 Gbps, the `-h delta` option has been available for some time to allow customization of the ramp-up speed (which is used below 1 Gbps until congestion is detected). In some cases, and especially with maximum capacities above 1 Gbps (where `-h delta` no longer has an impact), it can make sense to start at an initial sending rate above index 0 (zero). For example, specifying `-I @1000` when testing a 10 Gbps service would start with a sending rate of 1 Gbps. One important consideration with the `-I @index` option is that setting too high a value can be counter-productive to finding an accurate maximum. This is because when some devices or communication channels are suddenly overwhelmed by the appearance of very-high sustained traffic, it can result in early congestion and data loss that make it prematurely appear as if a maximum capacity has been reached. As such, it is recommended to use starting rates of only 10-20% of the expected maximum to avoid an early overload condition and false maximum. ## Server Optimization - Particularly When Jumbo Frames Are Unavailable By default, the sending rate table utilizes jumbo size datagrams when testing above 1 Gbps. As expected, maximum performance is obtained when the network also supports a jumbo MTU size (9000+ bytes). However, some environments are restricted to a traditional MTU of 1500 bytes and would be required to fragment the jumbo datagrams into multiple IP packets. One available option in these scenarios is to utilize the `-j` option on both the server and client to restrict datagrams to non-jumbo sizes above 1 Gbps. However, because of the extremely high resulting socket I/O rate at high speeds, it is unlikely that a full 10 Gbps can be achieved with this option. The compromise, to maintain a reasonable socket I/O rate, is to leave jumbo datagrams enabled (the default) and rely on the IP layer to fragment them. An important performance consideration, due to the higher network overhead in these scenarios (although beneficial under any circumstances), is to instantiate the udpst process in the same Non-Uniform Memory Access (NUMA) node as the network interface. This placement will limit cross-NUMA memory access on systems with more than one NUMA node. Testing has shown this commonplace server optimization can significantly increase sending rates while also reducing CPU utilization. The goal is to set the CPU affinity of the process to the same NUMA node as the one handling the network interface used for testing. Follow the steps below to achieve this: First, obtain the NUMA node count (to verify applicability if >1) as well as the CPU listing for each node. ``` $ lscpu | grep NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-13,28-41 NUMA node1 CPU(s): 14-27,42-55 ``` Next, find the NUMA node that corresponds to the test interface (in this case ens1f0 is associated with node 0). ``` $ cat /sys/class/net/ens1f0/device/numa_node 0 ``` Finally, start the process with a CPU affinity that matches the NUMA node of the test interface (node0 = 0-13,28-41). This example also uses a starting sending rate of 1 Gbps. ``` $ taskset -c 0-13,28-41 udpst -u -I @1000 ```