AppFabric Caching Service Crashes

I have a WCF 4.0 service hosted in IIS 7.5 on Windows 2008 R2 Enterprise 6 bit servers. The WCF service is configured to use ASP Compatibility].

On the Production and QA environments, we have a load balanced web farm. In development we only have a single server. We use AppFabric Caching as our distributed caching technology.

The AppFabric Caching Windows service crashes every few weeks (regardless of in the web farm or single server topography). This happens in conjunction with the IIS AppPool stopping; and also the WCF Web application. Other than a Windows event log entry indicating the Application Pool has stopped, there is no other information that I can identify that gives me any clues as to the reason for the crash. There is nothing in the Log files for the WCF application or AppPool. Also, there is nothing shown in my logging in the WCF application – probably because request fails in IIS before the WCF code runs.

The problem seems to occur overnight when the system is not in use - with the first request the next day failing. The cache is set to expire 1 hour before work begins in the new day. I do not have visibility of the servers in order to establish if any overnight server patches / updates might be related to the problem (or if the cache just stops because the AppPool recycles).

I am not 100% sure; but have a hunch that the Cache service stops first; and then the next WCF request causes the AppPool to crash.

I am using the Unity framework (v5.0) for Dependency injection. Below is the code for configuring the unity container:

public void Configure(IUnityContainer container)
{
    container.RegisterType<ICacheExpireCalculator, CacheExpireCalculatorCMF>("CMF_Cache_ExpiryCalculator", new InjectionMember[] { new InjectionConstructor(new object[] { ConfigurationManager.AppSettings["expireCacheDailyTimeUTC_HHmm"] }) });
if (bool.Parse(ConfigurationManager.AppSettings["useAppFabricCache"]))
{
    container.RegisterType<ICache, CacheAppFabricCMF>(new ContainerControlledLifetimeManager(), new InjectionMember[0]);
}
else
{
    container.RegisterType<ICache, CacheEntlibCMF>(new ContainerControlledLifetimeManager(), new InjectionMember[0]);
}
container.RegisterType<IJBADataAccess, JBADataAccess>("JbaDataDataAccess", new InjectionMember[] { new InjectionConstructor(new object[] { "jbaConnection" }) });
container.RegisterType<ISqlDataAccess, SqlDataAccess>("SQLDataDataAccess", new InjectionMember[0]);
container.RegisterType<IErrorProvider, CustomErrorProvider>(new ContainerControlledLifetimeManager(), new InjectionMember[0]);
container.RegisterType<IBusinessCMF, BusinessCMF>(new InjectionMember[0]);
container.RegisterType<IDataLookupLoader, DataLookupLoader>(new InjectionMember[0]);
container.RegisterType<IQueryHelper, QueryHelper>(new InjectionMember[0]);

}

When I started developing the solution, I used the Microsoft Enterprise Library Caching Service as the interim caching mechanism. In the WCF configuration file, I still have the Enterprise Library Caching configurations section - so that I can keep using a caching strategy on my Windows XP development machine (no AppFabric Caching). I use an appSettings flag to signal which of (a) AppFabric Caching or the (b) Enterprise library Caching service will be used. Obviously in the production and QA environments AppFabric Caching is flagged to be used.

I have just removed the Enterprise library setting from the web.config file; but will need to wait a few weeks before I understand if this has removed some runtime conflict - if both caching system assemblies are loaded.

Any thoughts would be appreciated:

Is it likely to cause issues if I leave the Enterprise library caching information in the web.config file (bearing in mind how I configure only one caching system to run via the Unity container)?
Is there any issue in the way I am configuring the Unity DI container [Cache and Error provider - singletons]?
Is there some logging that I can view (or turn) on for AppFabric Cache / or can configure in the Cache config files?
Do I need to explicitly dispose of any object instatiated via Unity?

Solution

Generally speaking, monitor the IO system (disk queue length) of the servers when the crash tends to occur. Specifically, check your backup agent/system.

I've seen AppFabric crash when it times out trying to access its configuration. It is VERY sensitive to being able to access its configuration. We use AF in a cluster, and we see crashes if connectivity to SQL Server is interrupted even briefly.

I mention the backup system because it might be that your backup system is creating a VSS snapshot, which "freezes" the machine a bit. This could be causing enough of an IO delay that AppFabric gets unhappy about not being able to see its config. This might also explain your AppPool issues that occur at the same time.