394 lines · plain
1.. SPDX-License-Identifier: GPL-2.02.. include:: <isonum.txt>3 4=============================================5Chelsio N210 10Gb Ethernet Network Controller6=============================================7 8Driver Release Notes for Linux9 10Version 2.1.111 12June 20, 200513 14.. Contents15 16 INTRODUCTION17 FEATURES18 PERFORMANCE19 DRIVER MESSAGES20 KNOWN ISSUES21 SUPPORT22 23 24Introduction25============26 27 This document describes the Linux driver for Chelsio 10Gb Ethernet Network28 Controller. This driver supports the Chelsio N210 NIC and is backward29 compatible with the Chelsio N110 model 10Gb NICs.30 31 32Features33========34 35Adaptive Interrupts (adaptive-rx)36---------------------------------37 38 This feature provides an adaptive algorithm that adjusts the interrupt39 coalescing parameters, allowing the driver to dynamically adapt the latency40 settings to achieve the highest performance during various types of network41 load.42 43 The interface used to control this feature is ethtool. Please see the44 ethtool manpage for additional usage information.45 46 By default, adaptive-rx is disabled.47 To enable adaptive-rx::48 49 ethtool -C <interface> adaptive-rx on50 51 To disable adaptive-rx, use ethtool::52 53 ethtool -C <interface> adaptive-rx off54 55 After disabling adaptive-rx, the timer latency value will be set to 50us.56 You may set the timer latency after disabling adaptive-rx::57 58 ethtool -C <interface> rx-usecs <microseconds>59 60 An example to set the timer latency value to 100us on eth0::61 62 ethtool -C eth0 rx-usecs 10063 64 You may also provide a timer latency value while disabling adaptive-rx::65 66 ethtool -C <interface> adaptive-rx off rx-usecs <microseconds>67 68 If adaptive-rx is disabled and a timer latency value is specified, the timer69 will be set to the specified value until changed by the user or until70 adaptive-rx is enabled.71 72 To view the status of the adaptive-rx and timer latency values::73 74 ethtool -c <interface>75 76 77TCP Segmentation Offloading (TSO) Support78-----------------------------------------79 80 This feature, also known as "large send", enables a system's protocol stack81 to offload portions of outbound TCP processing to a network interface card82 thereby reducing system CPU utilization and enhancing performance.83 84 The interface used to control this feature is ethtool version 1.8 or higher.85 Please see the ethtool manpage for additional usage information.86 87 By default, TSO is enabled.88 To disable TSO::89 90 ethtool -K <interface> tso off91 92 To enable TSO::93 94 ethtool -K <interface> tso on95 96 To view the status of TSO::97 98 ethtool -k <interface>99 100 101Performance102===========103 104 The following information is provided as an example of how to change system105 parameters for "performance tuning" an what value to use. You may or may not106 want to change these system parameters, depending on your server/workstation107 application. Doing so is not warranted in any way by Chelsio Communications,108 and is done at "YOUR OWN RISK". Chelsio will not be held responsible for loss109 of data or damage to equipment.110 111 Your distribution may have a different way of doing things, or you may prefer112 a different method. These commands are shown only to provide an example of113 what to do and are by no means definitive.114 115 Making any of the following system changes will only last until you reboot116 your system. You may want to write a script that runs at boot-up which117 includes the optimal settings for your system.118 119 Setting PCI Latency Timer::120 121 setpci -d 1425::122 123* 0x0c.l=0x0000F800124 125 Disabling TCP timestamp::126 127 sysctl -w net.ipv4.tcp_timestamps=0128 129 Disabling SACK::130 131 sysctl -w net.ipv4.tcp_sack=0132 133 Setting large number of incoming connection requests::134 135 sysctl -w net.ipv4.tcp_max_syn_backlog=3000136 137 Setting maximum receive socket buffer size::138 139 sysctl -w net.core.rmem_max=1024000140 141 Setting maximum send socket buffer size::142 143 sysctl -w net.core.wmem_max=1024000144 145 Set smp_affinity (on a multiprocessor system) to a single CPU::146 147 echo 1 > /proc/irq/<interrupt_number>/smp_affinity148 149 Setting default receive socket buffer size::150 151 sysctl -w net.core.rmem_default=524287152 153 Setting default send socket buffer size::154 155 sysctl -w net.core.wmem_default=524287156 157 Setting maximum option memory buffers::158 159 sysctl -w net.core.optmem_max=524287160 161 Setting maximum backlog (# of unprocessed packets before kernel drops)::162 163 sysctl -w net.core.netdev_max_backlog=300000164 165 Setting TCP read buffers (min/default/max)::166 167 sysctl -w net.ipv4.tcp_rmem="10000000 10000000 10000000"168 169 Setting TCP write buffers (min/pressure/max)::170 171 sysctl -w net.ipv4.tcp_wmem="10000000 10000000 10000000"172 173 Setting TCP buffer space (min/pressure/max)::174 175 sysctl -w net.ipv4.tcp_mem="10000000 10000000 10000000"176 177 TCP window size for single connections:178 179 The receive buffer (RX_WINDOW) size must be at least as large as the180 Bandwidth-Delay Product of the communication link between the sender and181 receiver. Due to the variations of RTT, you may want to increase the buffer182 size up to 2 times the Bandwidth-Delay Product. Reference page 289 of183 "TCP/IP Illustrated, Volume 1, The Protocols" by W. Richard Stevens.184 185 At 10Gb speeds, use the following formula::186 187 RX_WINDOW >= 1.25MBytes * RTT(in milliseconds)188 Example for RTT with 100us: RX_WINDOW = (1,250,000 * 0.1) = 125,000189 190 RX_WINDOW sizes of 256KB - 512KB should be sufficient.191 192 Setting the min, max, and default receive buffer (RX_WINDOW) size::193 194 sysctl -w net.ipv4.tcp_rmem="<min> <default> <max>"195 196 TCP window size for multiple connections:197 The receive buffer (RX_WINDOW) size may be calculated the same as single198 connections, but should be divided by the number of connections. The199 smaller window prevents congestion and facilitates better pacing,200 especially if/when MAC level flow control does not work well or when it is201 not supported on the machine. Experimentation may be necessary to attain202 the correct value. This method is provided as a starting point for the203 correct receive buffer size.204 205 Setting the min, max, and default receive buffer (RX_WINDOW) size is206 performed in the same manner as single connection.207 208 209Driver Messages210===============211 212 The following messages are the most common messages logged by syslog. These213 may be found in /var/log/messages.214 215 Driver up::216 217 Chelsio Network Driver - version 2.1.1218 219 NIC detected::220 221 eth#: Chelsio N210 1x10GBaseX NIC (rev #), PCIX 133MHz/64-bit222 223 Link up::224 225 eth#: link is up at 10 Gbps, full duplex226 227 Link down::228 229 eth#: link is down230 231 232Known Issues233============234 235 These issues have been identified during testing. The following information236 is provided as a workaround to the problem. In some cases, this problem is237 inherent to Linux or to a particular Linux Distribution and/or hardware238 platform.239 240 1. Large number of TCP retransmits on a multiprocessor (SMP) system.241 242 On a system with multiple CPUs, the interrupt (IRQ) for the network243 controller may be bound to more than one CPU. This will cause TCP244 retransmits if the packet data were to be split across different CPUs245 and re-assembled in a different order than expected.246 247 To eliminate the TCP retransmits, set smp_affinity on the particular248 interrupt to a single CPU. You can locate the interrupt (IRQ) used on249 the N110/N210 by using ifconfig::250 251 ifconfig <dev_name> | grep Interrupt252 253 Set the smp_affinity to a single CPU::254 255 echo 1 > /proc/irq/<interrupt_number>/smp_affinity256 257 It is highly suggested that you do not run the irqbalance daemon on your258 system, as this will change any smp_affinity setting you have applied.259 The irqbalance daemon runs on a 10 second interval and binds interrupts260 to the least loaded CPU determined by the daemon. To disable this daemon::261 262 chkconfig --level 2345 irqbalance off263 264 By default, some Linux distributions enable the kernel feature,265 irqbalance, which performs the same function as the daemon. To disable266 this feature, add the following line to your bootloader::267 268 noirqbalance269 270 Example using the Grub bootloader::271 272 title Red Hat Enterprise Linux AS (2.4.21-27.ELsmp)273 root (hd0,0)274 kernel /vmlinuz-2.4.21-27.ELsmp ro root=/dev/hda3 noirqbalance275 initrd /initrd-2.4.21-27.ELsmp.img276 277 2. After running insmod, the driver is loaded and the incorrect network278 interface is brought up without running ifup.279 280 When using 2.4.x kernels, including RHEL kernels, the Linux kernel281 invokes a script named "hotplug". This script is primarily used to282 automatically bring up USB devices when they are plugged in, however,283 the script also attempts to automatically bring up a network interface284 after loading the kernel module. The hotplug script does this by scanning285 the ifcfg-eth# config files in /etc/sysconfig/network-scripts, looking286 for HWADDR=<mac_address>.287 288 If the hotplug script does not find the HWADDRR within any of the289 ifcfg-eth# files, it will bring up the device with the next available290 interface name. If this interface is already configured for a different291 network card, your new interface will have incorrect IP address and292 network settings.293 294 To solve this issue, you can add the HWADDR=<mac_address> key to the295 interface config file of your network controller.296 297 To disable this "hotplug" feature, you may add the driver (module name)298 to the "blacklist" file located in /etc/hotplug. It has been noted that299 this does not work for network devices because the net.agent script300 does not use the blacklist file. Simply remove, or rename, the net.agent301 script located in /etc/hotplug to disable this feature.302 303 3. Transport Protocol (TP) hangs when running heavy multi-connection traffic304 on an AMD Opteron system with HyperTransport PCI-X Tunnel chipset.305 306 If your AMD Opteron system uses the AMD-8131 HyperTransport PCI-X Tunnel307 chipset, you may experience the "133-Mhz Mode Split Completion Data308 Corruption" bug identified by AMD while using a 133Mhz PCI-X card on the309 bus PCI-X bus.310 311 AMD states, "Under highly specific conditions, the AMD-8131 PCI-X Tunnel312 can provide stale data via split completion cycles to a PCI-X card that313 is operating at 133 Mhz", causing data corruption.314 315 AMD's provides three workarounds for this problem, however, Chelsio316 recommends the first option for best performance with this bug:317 318 For 133Mhz secondary bus operation, limit the transaction length and319 the number of outstanding transactions, via BIOS configuration320 programming of the PCI-X card, to the following:321 322 Data Length (bytes): 1k323 324 Total allowed outstanding transactions: 2325 326 Please refer to AMD 8131-HT/PCI-X Errata 26310 Rev 3.08 August 2004,327 section 56, "133-MHz Mode Split Completion Data Corruption" for more328 details with this bug and workarounds suggested by AMD.329 330 It may be possible to work outside AMD's recommended PCI-X settings, try331 increasing the Data Length to 2k bytes for increased performance. If you332 have issues with these settings, please revert to the "safe" settings333 and duplicate the problem before submitting a bug or asking for support.334 335 .. note::336 337 The default setting on most systems is 8 outstanding transactions338 and 2k bytes data length.339 340 4. On multiprocessor systems, it has been noted that an application which341 is handling 10Gb networking can switch between CPUs causing degraded342 and/or unstable performance.343 344 If running on an SMP system and taking performance measurements, it345 is suggested you either run the latest netperf-2.4.0+ or use a binding346 tool such as Tim Hockin's procstate utilities (runon)347 <http://www.hockin.org/~thockin/procstate/>.348 349 Binding netserver and netperf (or other applications) to particular350 CPUs will have a significant difference in performance measurements.351 You may need to experiment which CPU to bind the application to in352 order to achieve the best performance for your system.353 354 If you are developing an application designed for 10Gb networking,355 please keep in mind you may want to look at kernel functions356 sched_setaffinity & sched_getaffinity to bind your application.357 358 If you are just running user-space applications such as ftp, telnet,359 etc., you may want to try the runon tool provided by Tim Hockin's360 procstate utility. You could also try binding the interface to a361 particular CPU: runon 0 ifup eth0362 363 364Support365=======366 367 If you have problems with the software or hardware, please contact our368 customer support team via email at support@chelsio.com or check our website369 at http://www.chelsio.com370 371-------------------------------------------------------------------------------372 373::374 375 Chelsio Communications376 370 San Aleso Ave.377 Suite 100378 Sunnyvale, CA 94085379 http://www.chelsio.com380 381This program is free software; you can redistribute it and/or modify382it under the terms of the GNU General Public License, version 2, as383published by the Free Software Foundation.384 385You should have received a copy of the GNU General Public License along386with this program; if not, write to the Free Software Foundation, Inc.,38759 Temple Place - Suite 330, Boston, MA 02111-1307, USA.388 389THIS SOFTWARE IS PROVIDED ``AS IS`` AND WITHOUT ANY EXPRESS OR IMPLIED390WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF391MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.392 393Copyright |copy| 2003-2005 Chelsio Communications. All rights reserved.394