So it turns out the error messages were basically correct. I put a packet sniffer on the network and the packets are definitely not going out over that network card.
If I sent packets to 169.254.131.162 which happens to be a device on that network then they are routed over en0. If I send packets to 169.254.255.255 (i.e the broadcast address) I don’t see them on en0.