Ok. Wanted to share this - took me 10 hours to figure out.
I had properly installed mod-xsendfile, following the good instructions here.
I also configured xsendfile correctly in my /etc/httpd/conf/httpd.conf file, adding the settings: XSendFile on XSendFilePath /var/files_need_valid_session_to_view/
And I knew my code for generating the X-SENDFILE header was correct: it was working on a different server. However, I was getting 404 errors, no matter what I did. My OS was CentOS 6.4 final.
Label your directory with the correct SELinux label. See How to label an apache directory that is the same case for your folder '/var/files_need_valid_session_to_view/'.
Disabling SELinux is an unnecessary security risk.