Flow Offload + PBR + NAT: Incorrect Layer-2 Delivery

Overview

This document describes an obscure networking issue observed on OpenWrt 24.10.x where enabling flow offloading alongside policy-based routing (PBR) and NAT can result in packets being delivered to the wrong Layer-2 (MAC) destination on the LAN.

Observed Behavior

✅ Layer-3 routing is correct (destination IP is accurate)
❌ Layer-2 delivery is incorrect (destination MAC is wrong)
❌ Occurs only when flow offload is enabled

In affected configurations, return traffic intended for a LAN client may instead be forwarded to another LAN device (often a VPN gateway used as a PBR next-hop).

Symptoms

Typical real-world indicators:

Intermittent DNS or UDP failures
Client experiences request timeouts despite valid routing

Packet captures show:

Correct destination IP
Incorrect destination MAC

Disabling flow offload immediately resolves the issue.

Example Capture Evidence

Expected destination MAC: 02:11:22:33:44:55 (client)
Observed destination MAC: 02:aa:bb:cc:dd:ee (VPN gateway)

Affected Configuration Pattern

The issue appears when all three conditions are present:

Flow Offloading enabled
Policy-Based Routing via fwmark
NAT (SNAT and/or DNAT) applied

Simplified Topology

Client (10.0.0.2)
        │
        ▼
OpenWrt Router
        │
        ├── WAN (203.0.113.0/24)
        │
        └── VPN Gateway (10.0.0.3)  ← PBR next-hop

Example PBR Rule

ip rule add fwmark 0xff lookup 100

Traffic marked with 0xff is routed through table 100.

Technical Hypothesis

Flow offload introduces a fast path that bypasses portions of the traditional Linux packet processing pipeline.

Normal Forwarding Path

conntrack
→ routing lookup
→ neighbour (ARP) resolution
→ L2 transmission

Offloaded Forwarding Path

flowtable cache
→ fast forwarding

When policy routing and NAT transformations are involved, cached forwarding metadata may become invalid, producing Layer-2 misdelivery despite correct Layer-3 routing.

Mitigation Strategy

Rather than modifying routing behavior, the mitigation constrains which flows are eligible for offloading.

Exclude policy-routed and NAT-modified flows from flow offload.

Implemented nftables Guard Rules

meta mark != 0 return
ct status { dnat, snat } return
flow add @ft

Effect

Marked packets are not offloaded
NAT-modified flows are not offloaded
Only stable flows enter fast path

Why This Works

Flow offload assumes:

stable output interface
stable neighbour resolution
consistent routing decisions

Policy-routed traffic violates those assumptions.

Excluding these flows restores correct neighbour resolution while preserving acceleration elsewhere.

Reproducible Test Lab

A Linux network namespace lab simulates:

ns_client — LAN host
ns_router — OpenWrt-like router
ns_vpngw — VPN gateway
ns_wan — DNS server

Smoke Test Scenarios

Flow offload disabled
Flow offload enabled with mitigation

The test automatically fails if MAC mismatch is detected.

CI Safety

Packet capture is bounded to prevent CI hangs:

timeout 8 tcpdump ...

Limitations

Mitigation only — not a kernel fix
Hardware offload behavior may vary
Root cause still under investigation

Future Work

Hardware offload validation
Neighbour cache analysis
firewall4 integration improvements
Upstream patch proposal

Summary

Reproduction: ✅ Implemented
Detection: ✅ Automated
Mitigation: ✅ Implemented
Kernel Fix: 🔬 Investigation
Upstream Patch: ⏳ Pending

Conclusion

When combining:

Flow Offload
Policy-Based Routing
NAT

the safest current approach is selective flow offload exclusion for marked and NAT-modified traffic.