Tunneling and MTU

Tunneling is when lower layer traffic is encapsulated into upper or equal layer PDU. Although somewhat similar it is fundamentally different than Proxy. But now the original payload is encapsulated with additional header, the MTU of the tunnel interface must be adjusted to accommodate this additional header to avoid IP Fragmentation. For example the MTU for GRE/IPv4 tunnel should be brought down to 1476 bytes (1500 bytes Ethernet MTU - 4 bytes GRE header - 20 bytes outer IPv4 header).

This adjustment will be made on the router/firewall that created the tunnel. In the below scenario, I have created a GRE tunnel between R1 and R3 so that the subnet 10.10.10.0/24 and 10.10.20.0/24 are reachable from either router. However, the end hosts PC-1 and PC-2 does not know about this adjustment and they still see their Ethernet MTU as 1500 bytes. Hence, they can still transmit packet with size of 1500 bytes which when added with additional GRE/IPv4 headers will result to 1524 bytes which can't be sent through Ethernet medium and has to be fragmented.

Suppose an end host creates a packet sized 1476 byes or less which will easily accommodate the overhead and can be transmitted via Ethernet or it creates a packet sized greater than 1476 bytes which cannot accommodate the overhead within 1500 bytes and has to be fragmented.

This puts a question, what's the point of adjusting the MTU of the tunnel interface when at the end it is going to be fragmented anyway? The answer lies at what point the fragmentation will occur.

Consider the original IPv4 packet sent by PC-1. Assume the size of this packet is 1500 bytes and DF bit in IPv4 header is off.

Let's see what happens when it passes through the tunnel interface at R1 and let's say the MTU of tunnel interface is not lowered and set to 1500 bytes. The tunnel interface will not fragment the packet and attach the overhead resulting the packet size of 1524 bytes.

Ultimately this has to be transmitted through a physical Ethernet interface but as it has already exceeded the MTU of 1500 bytes, now this packet has to be fragmented utilizing the MF bit and Fragment Offset field of the outer IPv4 header. This is an ugly effect for the tunnel end device. The resulting fragmented packet will look like this.

Why is this ugly for tunnel end device? While transient routers (R2) will forward both the packets based on the Outer IPv4 header with no issue, tunnel end device (R3) will have to hold the packet on its buffer as the Outer IPv4 header has destination IP address of R3 and the router will have to hold on until all the fragmented packets are received, reassembled and set to its final journey. One of the fragmented packet could also be lost in transient and router will have to hold on to the initial fragmented packet for certain time only to discard it after timeout. This results in unnecessary burden for R3.

Now, let's see what the effect would be if we set tunnel interface MTU to 1476 bytes. Note the MTU of physical interface is left unaffected.

Here is the same original IPv4 packet sent by PC-1. Assume the size of this packet is 1500 bytes and DF bit in IPv4 header is off.

R1 receives it, but now the MTU of tunnel interface is set to 1476 bytes. Therefore, R1 decides to fragment this packet at this point before slamming the GRE/IPv4 header. The result looks like this:

Now both the packets can be safely transmitted via Ethernet interface. Note the differences, the packet is fragmented utilizing Inner IPv4 header's MF bit and Fragment Offset field, effectively allowing to attach GRE/IPv4 header on both the fragments. The advantage of this is now tunnel end device (R3) won't have to waste its precious buffer and CPU cycles. It can now immediately rip off the GRE/IPv4 header and forward the inner packet to end host.

The takeaway is that while fragmentation will definitely occur in the network, it's best if the fragmentation occurs at upper layer than lower lower layer. Fragmentation is a necessary evil. It will occur at point when the MTU of ingress interface is not equal to MTU of egress interface. As a general rule, Network designers should try to avoid it as much as possible. Some middleboxes like Firewall may not allow fragmented packets as they cannot gain full visibility within a fragmented packet. IP Fragmentation can also be used in DoS attacks. Therefore, a firewall may choose to simply drop fragmented packets for more security.