Optimising Xen

Xenserver network performance can be a little unoptimised out of the box compared to products such as VMware, but there are a few easy steps you can take to bump up the speed.

Disable TCP Checksum offload

If you notice a drop in network performance you can do a dump of the network traffic to see if you are getting checksum mismatching/checksum errors then you can manually disable TCP checksum offloading for the hosts network adapter.

To create a dump run this command:

tcpdump -i eth0 -v -nn | grep incorrect

The main line here you are looking out for is “incorrect (-> 0x6e35)”. This error is showing that checksum is failing to receive, which means that the server is having TCP offloading issues. This can be easily fixed by identifying what has been marked as active on the network interface card for offloading. If you have multiple NIC’s you will need to do this for each one:

ethtool -k ethX
Output:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
  • Scatter-Gather I/O – Rather than passing one large buffer, small buffers are passed which makes up large buffers. This provides more efficiency than large buffers passed.
  • TCP Segmentation Offload – It is the ability to frame data according to MTU size & same IP header with all packets. Useful when buffer is much larger than MTU on the link. The segmentation into smaller size is offloaded to NIC.
  • Generic Segmentation Offload – This is used to postpone the segmentation as long as possible. This performs the segmentation just before the entry into the driver’s xmit routine. GSO & TSO are only significantly effective only when MTU is much less than buffer size.
  • Generic Receive Offload – GSO only works for transmission of packets. This allows the packets to be re-fragmented at output. Unlike LRO which merges every packets, GRO merges with restriction keeping important fields in packet intact. NAPI API polls for new packets and process packets in batches before passing it to OS.
  • Large Receive Offload – This is used for combining multiple incoming packets into single buffer before passing it up to OS stack. Benefits of this is OS sees fewer packets & uses less CPU time.

Depending upon your NIC vendor, names of these processes may vary. Some vendors do provide additional offload processes.

Now you can run below script to disable TCP offload:

if [[ "$1" == "--local" || "$1" == "-l" ]]; then
    echo -n "disabling checksum offloading for local devices... "
    for iface in $(ifconfig | awk '$0 ~ /Ethernet/ { print $1 }'); do
        for if_mode in ${if_modes}; do
          ethtool -K $iface $if_mode off 2>/dev/null
        done
    done
    echo "done."
else
    echo -n "disabling checksum offloading in xapi settings... "
    for VIF in $(xe vif-list --minimal | sed -e 's/,/ /g')
    do
        ###xe vif-param-clear uuid=$VIF param-name=other-config
        for if_mode in ${if_modes}; do
            xe vif-param-set uuid=$VIF other-config:ethtool-${if_mode}="off"
        done
    done
    for PIF in $(xe pif-list --minimal | sed -e 's/,/ /g')
    do
        ###xe pif-param-clear uuid=$PIF param-name=other-config
        for if_mode in ${if_modes}; do
            xe pif-param-set uuid=$PIF other-config:ethtool-${if_mode}="off"
        done
    done
    echo "done."
fi