I want to modify kernel read/write functions (present in fs/read_write.c) in order to implement a simple substitution cipher, but I want to do this only for certain files and not all and hence inside kernel code I want to check if this is a file marked by user for encryption then only do this task.
One way I discovered was that user can set sticky bit using :
chmod +t filename
and then use __check_sticky() to detect but is there any other more elegant way to go around it ?
You could use extended file attributes. They allow you to set and get arbitrary metadata associated with a file.
Check man 5 attr and this question for how to set and get extended attributes from the kernel.