Google can't crawl generated CSS and JS paths originating from Custom outputcache

I have set up Custom Outputcache for my website. Everything works as intended, I can see the cache folder with the binary files. When I visit the site I get the cached pages and it renders as it should.

The problem is that when I try to use Google webmaster tools to render the page, Google can't access the generated css path generated in the BundleConfig ~/bundles/styles/maincss/, the same goes to the javascript path.`. When I visit those two paths I get to see the minified JS and CSS files and the browser does render the page correctly.

This poses an issue, because now when I test the page using mobile-test tool, I get that the pages are not mobile friendly. For some reason Google can't access those paths, although when I run the web crawl in webmaster tools, it does render good for the user, but not for Google-bot.

This worked when I don't use the Custom output caching, only the default Outputcache.

Any idea how to tackle this issue.

Custom outputcache code:

Web.config:

</connectionStrings> 
 <appSettings>
   <add key="CacheLocation" value="~/Cache"/>
 </appSettings>
 <system.web>
   <caching>
     <outputCache defaultProvider="FileCache">
       <providers>
        <add name="FileCache" type="ProjectX.FileCacheProvider"/>
    </providers>
  </outputCache>
</caching>

Caching Class:

using System;
using System.Configuration;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
using System.Web;
using System.Web.Caching;

namespace ProjectX
{
    [Serializable]
    public class CacheItem
    {
        public object Item { get; set; }
        public DateTime Expiry { get; set; }
    }

    public class FileCacheProvider : OutputCacheProvider
    {
        private string CacheLocation
        {
            get
            {
                string strCacheLocation = ConfigurationManager.AppSettings["CacheLocation"];
                strCacheLocation = HttpContext.Current.Server.MapPath(strCacheLocation);
                return strCacheLocation + @"\";
            }
        }


        private string GetFullPathForKey(string key)
        {
            string temp = key.Replace('/', '$');
            return CacheLocation + temp;
        }

        public override object Add(string key, object entry, DateTime utcExpiry)
        {
            object obj = this.Get(key);
            if (obj != null)
            {
                return obj;
            }
            else
            {
                this.Set(key, entry, utcExpiry);
                return entry;
            }
        }

        public override void Remove(string key)
        {
            string filePath = GetFullPathForKey(key);
            if (File.Exists(filePath))
            {
                File.Delete(filePath);
            }
        }

        public override object Get(string key)
        {
            string filePath = GetFullPathForKey(key);
            if (!File.Exists(filePath))
            {
                return null;
            }
            CacheItem item = null;
            FileStream fileStream = File.OpenRead(filePath);
            BinaryFormatter formatter = new BinaryFormatter();
            item = (CacheItem)formatter.Deserialize(fileStream);
            fileStream.Close();
            if (item == null || item.Expiry <= DateTime.UtcNow)
            {
                Remove(key);
                return null;
            }
            return item.Item;
        }



        public override void Set(string key, object entry, DateTime utcExpiry)
        {
            string filePath = GetFullPathForKey(key);
            CacheItem item = new CacheItem { Expiry = utcExpiry, Item = entry };
            FileStream fileStream = File.OpenWrite(filePath);
            BinaryFormatter formatter = new BinaryFormatter();
            formatter.Serialize(fileStream, item);
            fileStream.Close();
        }
    }
}

Solution

If you wish to continue down this path (I assume you have used https://weblogs.asp.net/gunnarpeipman/asp-net-4-0-writing-custom-output-cache-providers as your starting point?), you will need to change GetFullPathForKey so that it generates valid filenames. How to remove illegal characters from path and filenames? will likely help you do that. You will also need to change your code so that it doesn't fall over if two threads try and write to the same file at the same time. Plus you really should introduce the use of a MemoryCache to avoid hitting the file system every time a Get call to your class occurs. This will be a lot of work.

I would suggest considering these NuGet packages as an alternative. They do this stuff out of the box, and are well tested:

Or just whack a reverse proxy in front of the web server which assists with caching - e.g. http://mikehadlow.blogspot.com.au/2013/05/the-benefits-of-reverse-proxy.html .