I know that libvlc can transcode many video formats. But how to use it to encode raw video frames? I just can't find related APIs.
You want to use the memory access modules, which were introduced specifically for this use-case in the forthcoming v3.0 of libvlc. There is no public documentation for those as of yet.