Ran into a very weird situation where I had a system that was receiving UDP traffic.  I could see the traffic arriving at the network interface using TCPDump or Wireshark, however, the traffic was not being received by any layer-7 application.  Instructing netcat to listen on the port never saw any connection.  Same thing with any other application tested, the traffic just never seemed to make it up the network stack to the application.

The issue was that the system had no route back to the source system.  Even though the UDP traffic didn’t require a response, the system still fails to recognize the traffic when it is unable to route any response (or ICMP unreachable/disposition messages) back to it.

To fix, add a route back to the source system.

Alternatively, loosen the sysctl rp_filter (“reverse path filter”). That is /proc/sys/net/ipv4/conf/*/rp_filter (or sysctl -w “net.ipv4.conf.all.rp_filter=2”). Set it to “2” to allow as long as a route exists on any interface, or “0” to disable.