-
Notifications
You must be signed in to change notification settings - Fork 0
/
MLNX_EN_Linux_README.txt
411 lines (333 loc) · 15.5 KB
/
MLNX_EN_Linux_README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
Mellanox Technologies
=====================
===============================================================================
MLNX_EN for Linux README
Driver Kit for Mellanox Adapter Cards with 10GigE Support
Version 1.5.10 January 2013
Document No. 2950
===============================================================================
-------------------------------------------------------------
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (PRODUCT(S))
AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX
TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY
FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS
THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S
MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET
BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S)
AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES
CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS
WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL
MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY
DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED
DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
--------------------------------------------------------------
© Copyright 2013. Mellanox Technologies. All rights reserved.
Mellanox®, Mellanox Logo®, BridgeX®, ConnectX®, CORE-Direct®,
InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®, SwitchX®,
Virtual Protocol Interconnect® and Voltaire®
are registered trademarks of Mellanox Technologies, Ltd.
Connect-IB™, FabricIT™, MLNX-OS™, MetroX™, ScalableHPC™,
Unbreakable-Link™, UFM™ and Unified Fabric Manager™
are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
Contents:
=========
1. Overview
1.1 Package Contents
2. Supported Platforms, Operating Systems and Firmware
3. Software Dependencies
4. Driver Installation
4.1 Installing the Driver
4.2 Loading the Driver
4.3 Unloading the Driver
4.4 Uninstalling the Driver
5. Ethernet Driver Usage and Configuration
6. Firmware Programming
6.1 Installing Firmware Tools
6.2 Downloading the Firmware Image of the Adapter Card
6.3 Updating Adapter Card Firmware
7. Perfromance Tuning
8. Revision History
===============================================================================
1. Overview
===============================================================================
This document provides information on the MLNX_EN Linux driver and
instructions for installing the driver on Mellanox ConnectX adapter cards
supporting 10Gb/s Ethernet.
The driver is intended for adapter cards that identify on the PCI bus as with
one of the following PCI Device IDs (decimal): 25408, 25418, 25448, 26418,
26448, 26428, 25458, 26458, 26468, 26438, 26478, 26488 4099.
The MLNX_EN driver release exposes the following capabilities:
- Single/Dual port
- Up to 16 Rx queues per port
- 16 Tx queues per port
- Rx steering mode: Receive Core Affinity (RCA)
- MSI-X or INTx
- Adaptive interrupt moderation
- HW Tx/Rx checksum calculation
- Large Send Offload (i.e., TCP Segmentation Offload)
- Large Receive Offload
- Multi-core NAPI support
- VLAN Tx/Rx acceleration (HW VLAN stripping/insertion)
- Ethtool support
- Net device statistics
1.1 Package Contents
--------------------
This driver kit contains the following:
- Kernel modules
* mlx4 driver
mlx4 is the low level driver implementation for the ConnectX adapters
designed by Mellanox Technologies. The ConnectX can operate as an
InfiniBand adapter and as an Ethernet NIC.
To accommodate the two flavors, the driver is split into modules:
mlx4_core, mlx4_en, and mlx4_ib.
Note: mlx4_ib is not part of this package
- mlx4_core
Handles low-level functions like device initialization and firmware
commands processing. Also controls resource allocation so that the
InfiniBand, Ethernet and FC functions can share a device without
interfering with each other.
- mlx4_en
Handles Ethernet specific functions and plugs into the netdev
mid-layer.
- mstflint
An application to burn a firmware binary image
- Sources of all software modules (under conditions mentioned in the modules'
LICENSE files)
- Documentation
===============================================================================
2. Supported Platforms, Operating Systems and Firmware
===============================================================================
o CPU architectures:
- x86_64
o HW:
- Winterfell servers
- Dragonstone servers
o Cards:
- Harrier 2G
- Harrier 3G
o Linux Operating Systems:
- CentOS5.2 + 2.6.38fbk32
- CentOS5.2 + 3.2.18fbk11
o Supported Firmware
- 2.10.3898 or higher verified for Facebook
===============================================================================
3. Software Dependencies
===============================================================================
- To install the driver software, kernel sources must be installed on the
machine.
- MLNX_EN driver cannot coexist with OFED software on the same machine.
Hence when installing MLNX_EN all OFED packages should be removed (done by
the mlnx_en install script).
===============================================================================
4. Driver Installation
===============================================================================
4.1 Installing the Driver
-------------------------
Step 1: Download Driver Package
Please download the current driver package from
http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=27&menu_section=35
Step 2: Install Driver
Run the following commands to install the driver:
#> tar xzvf mlnx_en-1.5.10.tgz file
#> cd mlnx_en-1.5.10
#> ./install.sh
The package consists of several source RPMs. The install script rebuilds the
source RPMs and then installs the created binary RPMs. The created kernel
module binaries are placed under /lib/modules/<kernel-ver>/updates/kernel/drivers/net/mlx4.
mlnx_en installer supports 2 modes of installation.
The install scripts selects the mode of driver installation depending of
the running OS/kernel version.
1. Kernel Module Packaging (KMP) mode, where the source rpm is
rebuilt for each installed flavor of the kernel. This mode is used
for RedHat and SUSE distributions.
2. Non KMP installation mode, where the sources are rebuilt with
the running kernel. This mode is used for vanilla kernels.
NOTE: If the Vanilla kernel is installed as rpm, please use the "--disable-kmp"
flag when installing the driver.
The kernel module sources are placed under /usr/src/mellanox-mlnx-en-1.5.10/.
Run the following commands to recompile the driver:
#> cd /usr/src/mellanox-mlnx-en-1.5.10/
#> scripts/mlnx_en_patch.sh
#> make
#> make install
The uninstall and performance tuning scripts are installed.
NOTE: If the driver was installed without kmp support,
the sources would be located under /usr/srs/mlnx_en-1.5.10/
4.2 Loading the Driver
----------------------
Step 1: Make sure no previous driver version is currently loaded
Run:
#> modprobe -r mlx4_en
Step 2: Load the new driver version
Run:
#> modprobe mlx4_en
The result is a new net-device appearing in 'ifconfig -a' output.
See "Ethernet Driver Usage" for details on driver usage and configuration.
4.3 Unloading the Driver
------------------------
To unload the Ethernet driver run:
#> modprobe -r mlx4_en
4.4 Uninstalling the Driver
--------------------------
To uninstall the mlnx_en driver run:
#> /sbin/mlnx_en_uninstall.sh
===============================================================================
5. Ethernet Driver Usage and Configuration
===============================================================================
- To assign an IP address to the interface run:
#> ifconfig eth<x> <ip>
where 'x' is the OS assigned interface number.
- To check driver and device information run:
#> ethtool -i eth<x>
Example:
#> ethtool -i eth2
driver: mlx4_en (MT_0DD0120009_CX3)
version: 1.5.10 (Jan 2013)
firmware-version: 2.10.3898
bus-info: 0000:1a:00.0
- To query stateless offload status run:
#> ethtool -k eth<x>
- To set stateless offload status run:
#> ethtool -K eth<x> [rx on|off] [tx on|off] [sg on|off] [tso on|off] [lro on|off]
- To query interrupt coalescing settings run:
#> ethtool -c eth<x>
- By default, the driver uses adaptive interrupt moderation for the receive
path, which adjusts the moderation time to the traffic pattern.
To enable/disable adaptive interrupt moderation use the following command:
#>ethtool -C eth<x> adaptive-rx on|off
- Above an upper limit of packet rate, adaptive moderation will set the
moderation time to its highest value. Below a lower limit of packet rate,
the moderation time will be set to its lowest value.
To set the values for packet rate limits and for moderation time high and low
values, use the following command:
#> ethtool -C eth<x> [pkt-rate-low N] [pkt-rate-high N] [rx-usecs-low N] [rx-usecs-high N]
- To set interrupt coalescing settings when adaptive moderation is disabled, use:
#> ethtool -C eth<x> [rx-usecs N] [rx-frames N]
Note: usec settings correspond to the time to wait after the *last* packet is
sent/received before triggering an interrupt.
- To query pause frame settings run:
#> ethtool -a eth<x>
- To set pause frame settings run:
#> ethtool -A eth<x> [rx on|off] [tx on|off]
- To query ring size values run:
#> ethtool -g eth<x>
- To modify rings size run:
#> ethtool -G eth<x> [rx <N>] [tx <N>]
- To obtain additional device statistics, run:
#> ethtool -S eth<x>
- To perform a self diagnostics test, run:
#> ethtool -t eth<x>
The driver defaults to the following parameters:
- Both ports are activated (i.e., a net device is created for each port)
- The number of Rx rings for each port is the nearest power of 2 of number of cpu cores, limited by 16.
- LRO is enabled with 32 concurrent sessions per Rx ring
Some of these values can be changed using module parameters, which can be
displayed by running:
#> modinfo mlx4_en
To set non-default values to module parameters, the following line should be
added to /etc/modprobe.conf file:
"options mlx4_en <param_name>=<value> <param_name>=<value> ..."
Values of all parameters can be observed in /sys/module/mlx4_en/parameters/.
===============================================================================
6. Firmware Programming
===============================================================================
The adapter card was shipped with the most current firmware available. This
section is intended for future firmware upgrades, and provides instructions for
(1) installing Mellanox firmware update tools (MFT), (2) downloading FW, and
(3) updating adapter card firmware.
6.1 Installing Firmware Tools
-----------------------------
The driver package compiles and installs the Mellanox 'mstflint' utility under
/usr/local/bin/. You may also use this tool to burn a card-specific firmware
binary image. See the file /tmp/mlnx_en/src/utils/mstflint/README file for
details.
Alternatively, you can download the current Mellanox Firmware Tools package
(MFT) from http://www.mellanox.com > Downloads > Firmare Tools. The tools
package to download is "MFT_SW for Linux" (tarball name is mft-X.X.X.tgz).
6.2 Downloading the Firmware Image of the Adapter Card
------------------------------------------------------
To download the correct card firmware image, please visit
http://www.mellanox.com > Downloads > Firmware.
For help in identifying your adapter card, please visit
http://www.mellanox.com/content/pages.php?pg=firmware_HCA_FW_identification.
6.3 Updating Adapter Card Firmware
---------------------------------
Using a card specific binary firmware image file, enter the following command:
#> mstflint -d mtXXXXX_pci_cr0 -i <image_name.bin> b
where XXXXX stands for the PCI Device ID (decimal).
For burning firmware using the MFT package, please check the MFT user's manual
under http://www.mellanox.com > Downloads > Firmare Tools.
*** IMPORTANT NOTE ***
After burning new firmware to an adapter card, reboot the machine so that the
new firmware can take effect.
===============================================================================
7. Perfromance Tuning
===============================================================================
7.1 Increasing packet rate
--------------------------
To increase packet rate (especially for small packets), set the value of
"high_rate_steer" module parameter in mlx4_module to 1 (default 0).
*** Note: Enabling this mode will cause the following chassis management features
to stop working:
NCSI
RoL
7.2 Managing IRQ Core Affinity
------------------------------
IRQ Affinity scripts are installed as part of driver installation.
o For Intel Sandy Bridge systems set the irq affinity to the adapter's NUMA
node:
- For optimizing single-port traffic, run:
# set_irq_affinity_bynode.sh <numa node> <interface>
- For optimizing dual-port traffic, run:
# set_irq_affinity_bynode.sh <numa node> <interface1> <interface2>
- To show the current irq affinity settings, run:
# show_irq_affinity.sh <interface>
o AMD systems hase 2 NUMA nodes which are closer to the PCIe adapter.
- In 2-socket system the adapter is connected to socket 0 (NUMA 0 + 1)
# set_irq_affinity_cpulist.sh <numa 0 and numa 1 cores> <interface>
- In 4-socket system the adapter is connected either to socket 0
(NUMA 0 + 1) of socket 3 (NUMA 6 + 7)
# set_irq_affinity_cpulist.sh <numa 6 and numa 7 cores> <interface>
Example:
# set_irq_affinity_cpulist.sh 0-12 eth5
For Intel Nehalem/Westmere platforms set the irq affinity balanced across all
the cores in the system.
o For all other systems:
- For optimizing single-port traffic, run:
# set_irq_affinity.sh <interface>
- For optimizing dual-port traffic, run:
# set_irq_affinity.sh <interface1> <interface2>
- To show the current irq affinity settings, run:
# show_irq_affinity.sh <interface>
Example:
Tuning for RFC2544 throughput benchmark where adapter's node is '0':
# /etc/init.d/irqbalancer stop
# set_irq_affinity_bynode.sh 0 eth3 eth4
===============================================================================
8. Revision History
===============================================================================
* Rev 1.5.10 (January 2013)
* Rev 1.5.9 (September 2012)
* Rev 1.5.8.3 (June 2012)
* Rev 1.5.8.2 (June 2012)
* Rev 1.5.8.1 (May 2012)
* Rev 1.5.7.2 (March 2012)
* Rev 1.5.7 (November 2011)
* Rev 1.5.6 (January 2011)
* Rev 1.5.1.3 (July 2010)
* Rev 1.4.2 (September 2009)
* Rev 1.4.1 (April 2009)
* Rev 1.3.0 (Sep 2008)
* Rev 1.0 (July 2008)