Context:
I am writing a program, which uses pcap to capture packets in the monitor mode on the openwrt router with ar9331 chip. I tested the program on a desktop with pcap 1.1 (which existed in my openwrt version) and found an issue: pcap_can_set_rfmon returned true, pcap_set_rfmon returned success, but attempt to activate capture resulted in “monitor mode isn't supported” error.
Google search showed a bug report of similar issue with wireshark. One of the comments says that with some wi-fi devices the issue is caused by an old version of pcap, which uses old version of another lib. I updated pcap version to 1.5.3 and the issue was resolved.
Problem: The issue appears again when I port our program to Openwrt. But now update of libpcap package to version 1.5.3 from newer openwrt branch doesn't help.
Sadly, the libpcap monitor-mode code on Linux works best when libpcap is linked with libnl, and it's often not linked with libnl for various reasons (including problems with a program using libpcap and libnl, and linked with a different version of libnl than the one with which libpcap is linked).
This needs to be redone in libpcap. It may end up being done with a "helper process" that libpcap runs to do various things; that would also improve cleanup if the program using libpcap exits abnormally and would allow packet capture operations requiring special privileges to be confined to the helper process rather than requiring the program using libpcap to run with those privileges. This is on my rather long to-do list.
The best workaround is probably to use airmon-ng to turn monitor mode on, as described in the Wireshark Wiki page on Wi-Fi capturing.