Introduction

Very often when IPSec tunnel is used, throughput is affected or users are experiencing fragmentation issues. This is caused by incorrect MTU size and encapsulation overhead. Common example is when icmp ping works both way without any issues, or manual telnet to www port is open but the actual page won’t open or opens intermittently.

Cause

MTU size for Ethernet is 1500 (1514 if we count 802.1 Ethernet header).When original IP packet gets encrypted by IPSec, there’s an overall increase in packet size. Any encapsulation that takes place, adds overhead to the original packet size:

  • IPSec encryption performed by the DMVPN adds 73 bytes for ESP-AES-256 and ESP-SHA-HMAC overhead (overhead depends on transport or tunnel mode and the encryption/authentication algorithm and HMAC).
  • MPLS adds 4 bytes for each label in the stack.
  • IEEE 802.1Q tag adds 4 bytes (Q-in-Q would add 8 bytes)

Most networking devices use Path MTU to calculate proper MTU size on the entire path. This works by setting DF-bit to 1 and forcing MTU size. If MTU size along the way to destination is too small, router/firewall will inform the host and drops the packet and sends an ICMP Fragmentation Needed Type 3 Code 4 packet back to the sending device with its MTU size.

Most of the common causes that break PMTUD are blocked icmp, asymmetric routing or not enough bytes sent from the client side to trigger PMTDU. Hence most of the firewall vendors clamp MSS connections to e.g. 1380 bytes (Cisco ASA).

Another workaround (not fix! and it should be used as a last resort!) is to get edge routing device to clear DF-bit so fragmentation is allowed.

Resolution

To fix the issue, we need to determine our MTU size in non-VPN enivorment:

[root@smtp ~]# ping -M do -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1472(1500) bytes of data.
76 bytes from 8.8.8.8: icmp_seq=1 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=2 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=3 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=4 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=5 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=6 ttl=111 (truncated)
76 bytes from 8.8.8.8: icmp_seq=7 ttl=111 (truncated)

As we can see in test above, our interface MTU is default 1500 bytes (1500-28bytes for IP+ICMP overhead = 1472). Now, let’s calculate the IPSec overhead based on encryption used:

IPSec Transform SetIPSec Overhead, Maximum Bytes
esp-AES-(256 or 192 or 128) esp-SHA-hmac or md5  73
esp-AES (256 or 192 or 128)61
esp-3des, esp-DES45
esp-(DES or 3des) esp-SHA-hmac or md5 57
esp-null esp-SHA-hmac or md5 45
ah-SHA-hmac or md5 44

Let’s calculate our proper MTU size using the formula:

MTU size - encapsulation overhead = interface MTU

1500 – 61 = 1439 definitive MTU

Depending on the vendor used, we can update our MTU size to calculated value.