I am trying to develop a custom cryptographically secure protocol for authenticating through an RFID smart card, all I can find over the internet are generic info on how write static Tags into the card. For my project I need for the reader to send a Challenge nonce to the card (fixed length random bytes), then the card should send back the challenge encrypted with the hard-coded key (for example with AES CBC) or an hash HMAC.
The problem is: how can I program the card to do such behavior? I can write the code to do so, but how to "flash" it to the card. Also the code for the reader is not a problem (I want to use an Arduino with a reader module).
And then has a passive chip enough power to do this kind of calculations or is better to use an active one, considering that the distance would be few centimeters, such as behind a door?
A passive tag can't do this. Those just hold serial numbers that they spit out back.
You need some smarter card. You have basically two choices:
1) Choose a general-purpose card that fit your needs
The authentication scheme you describe is very widely used, and implemented by virtually all general-purpose smart cards, whether with contact or contactless (RF) interfaces (or both). If you go this way, you don't need to program the card with your own code, you just need to get the specs and initialize the card with the key value you want by sending the appropriate commands. And actually, with these cards, there are often generic tools available to initialize them. Easy.
Mifare (by NXP), for example is very widely used, easily available, and cards are cheap (drawback: it is certainly not the most secure cards ever made). You could use Mifare Classic - with proprietary NXP security protocol - specs here (be careful, the reader must be compatible) or Mifare UltraLight C, with open 3DES crypto - specs here. There are other variants, as well (e.g. Mifare Plus, which uses AES). Just don't go for the simple Mifare Ultralight, which doesn't provide authentication means. You'll see that for all these type of cards, there is a builtin authentication scheme that can functionally fulfill your needs (it is actually a mutual authentication, so it does more that what you require, but it's fine).
FeliCa (made by Sony), ACOS3 (made by ACS - specs here) are other possibilities, although I'm less familiar with it.
2) Get programmable cards
There are some, but it is not very easy to find for contactless and they are very expensive. You can lookup:
BasicCard (ZeitControl), which exist with dual interface. Those cards are programmable in Basic, and there are some kits available (but I'm not familiar with it either).
JavaCard (any manufacturer), which is a standard, but those are hard to find for small volumes. A few online shops seem to have some, though. These cards are programmable in Java (actually a subset of Java). There are tools available for the development phase (from Oracle, or even as Eclipse Plugins, for example). Then, for loading your custom applet in the smart card, you need to get familiar with GlobalPlatform, which is another (publicy available) standard, describing the way applications are managed in a smart card. This would be too broad to describe in detail here, though. You can find resources on the internet, but this road is definitely the most difficult.