Jumbo frames allows you to set a MTU of up to 9000 – large enough for a single default size Oracle block to transmit in a single packet. However, to increase the MTU you may need to apply changes at the switch, the network card and the Operating System level. Setting jumbo frames at the operating system level, but failing to do so at the switch level is a common mistake. Note too, that not all switches support jumbo frames.
To set jumbo frames at the host level, you use a utility such as ifconfig. On Linux, you can use the ifconfig command to check the MTU size:
#ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0D:56:18:2C:60
inet addr:10.20.12.154 Bcast:10.20.12.255 Mask:255.255.255.0
inet6 addr: fe80::20d:56ff:fe18:2c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:951407 errors:0 dropped:0 overruns:0 frame:0
TX packets:544283 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:119973568 (114.4 MiB) TX bytes:124883921 (119.0 MiB)
Base address:0xdf40 Memory:feae0000-feb00000
The MTU is at the default of 1500. To set it – at the Operating system level – to 9000 we again use the ifconfig command:
# ifconfig eth0 mtu 9000
# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0D:56:18:2C:60
inet addr:10.20.12.154 Bcast:10.20.12.255 Mask:255.255.255.0
inet6 addr: fe80::20d:56ff:fe18:2c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:951649 errors:0 dropped:0 overruns:0 frame:0
TX packets:544386 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:119996835 (114.4 MiB) TX bytes:124902029 (119.1 MiB)
Base address:0xdf40 Memory:feae0000-feb00000
Setting jumbo frames at the OS level will achieve nothing if the NICs or switches are not configured to support an MTU of 9000 – you may need to consult with your MIS department or consult the hardware documentation. You also need to make sure that every host in the cluster has the same MTU setting.
When enabling Jumbo Frames, be certain that you enable it on all hosts in the cluster and also at the hardware level (switches and NICs).
Each OS supports a different mechanism for enabling Jumbo Frames. On Solaris, you need to edit the interface configuration file (/kernel/drv/e1000g.conf for instance).
See Oracle support note Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (341788.1) for more information.
If your interconnect is configured using Ethernet (GBe or 10GBe) then enabling Jumbo Frames will reduce the packet rate, increasing reliability and throughput.
UDP buffer size
When a RAC host receives a network packet over the interconnect, the packet is held in a small memory buffer until Operating System gets around to processing the buffer. If the system is particularly busy and/or the interconnect is overloaded, then packets might be lost. Increasing the UDP receive buffer size can help.
The OS default value for the UDP receive buffer is generally in the vicinity of 128-256K which can be inadequate for the bursts of load that can occur across the interconnect. If the buffe