In appsync, there are 4 authorization mode, API Key, IAM, Cognito, OPEN ID. How can I make a customised one, like a lambda? Or how can I remove authorization at all which means it is open to public and I will do authorization in business layer?
AppSync doesn't currently support custom Lambda authorizer, you can +1 the issue. https://github.com/aws/aws-appsync-community/issues/2
Regarding your second question, you can't remove authentication altogether, the closest you can get to it is use API_KEY (no authentication essentially as any client or person with your API_KEY can access your API.