I need to modify the way the wireless access point handle its authentication method. Is it possible to change the behaviour of wireless access point a little bit? If so, by using what library? On what operating system?
This depends on various things, the first being whether your router can utilize custom firmware. Some routers like the Linksys WRT54G (pre V4 or the GL version) use a Linux based firmware that allow you to do alot more with the routers hardware. There are some custom distributions for these devices like DD-WRT, however OpenWRT might be the best place to start if you're comfortable with Linux.
I would start by looking if your router is listed as being supported on the above sites
Unfortuanetly there isn't one specific library that I know of that will help you here, you might have to reimplement part of the security logic in these small distros and recompile them to achieve what you want