I need to implement WCF Service using HTTP protocol with encryption. Either message or transport security will do. Am I required to use x509 certificate to achieve security requirements for encryption? Or is there another way?
You'll want to use Transport security, not Message security. Transport security works just like regular https encryption - in fact, the URL to your host will be https. It's a little complicated to set up and get the certs and program configured correctly, but Transport has a big advantage over Message security.
With Message security, you must manually install the public key file of your x509 certificate on each of the client machines, and the public and private keys on the host machine, in order to get message security working. This means a lot of extra work - i.e. all the client cert file installations. Plus, since Message security is SOAP-based, you'll have to code everything along the SOAP lines, making REST programming difficult or impossible.
With Transport, you don't need to install copies of the cert public key on each client machine; just as with a normal https web server, the host transmits the public key to the client it first makes your connection.
Here's a link to a decent how-to article on https over wcf: http://www.codeproject.com/Articles/36705/Seven-simple-steps-to-enable-HTTPS-on-WCF-WsHttp-b