User Activity Profile and Statistics

I am just exploring on what is the best practice/framework for implementing feature for collecting and displaying user activity statistics that is user specific and site relating to login user activities in ASP.NET. For example, I will want to know for a particular login user on my site, which site he/she has visited in the last day, week or etc. What is the frequency? top 5/10? and for the overall site, what are the top 5 popular pages or search terms (based only on login users not crawler or anonymous visits)

I have used web stats tool like getclicky, webstats and google analytics. They are all great but the tracking is based on generic visits but what I want is to tie it in with individual users/roles or organisation structure defined in my system and be able to report on them to the stakeholders.

This gets more interesting too if we have hierarchical structure say, user->department/group->company and try reporting on what are the top 5 sites for this user? what are the top 5 sites visited by users in this department/business units/group? what are the page frequency?

QUESTION: So what is the best way to implement this in ASP.NET? Is there a httpModule handler, framework or product that does this?

Solution

I will tell you what I have done.

If you know what you're looking for in the statistics, eg as you say, what is the frequency then make this statistic calculation when you insert the new data.

For example let's say that you search how many pages user A have visited. Keep a line on statistics table that connect this user A, with the informations that you need to have, and you calculate them the moment you enter them, eg in every page just increase the counter of how many have see. That way you can have online statistics avoid background work, and thousands of line entries. I believe that you need to know what statistics you're looking for at the first place, and if there are some more in the future, then you add them later.

You add the call to statistics calculation and insertion on the end of page, either with a thread or with a call to an extra handler so the user do not have delay to see the page.

eg, on the end of the page:

<img src="/myStats.ashx?infos=CodeStringWithSomeInfos">

on myStats.ashx, save your statistics data, and return an emtpy gif.

Also when you show the data, even if you calculate most of your information on time, there are always extra things that you can not calculate, when you show this statistics and make this calculations, keep them on an extra table cache to speed up.