I am developing a medium site for my company, with large amounts of data (research publications, hundreds of employees, etc.) and security constraints which made me think of using django-guardian to handle object permissions. But now I'm realising that this may be slow within the admin.
We have already implemented a redis cache which seems to work fairly ok, but still loading large list views (hundreds of elements) take loong time.
We are using, so far, the following set up:
django 1.5.5
django-cms 2.4.3
django-redis-cache 0.10.2
django-guardian 1.1.1
hiredis 0.1.2
redis 2.9.1
python 2.7.5
postgresql
centos
As an example, this is the Person module, which list view takes ages to load (not as a superuser, in that case it's quite fast: this is the reason why I think the problem lies on the django-guardian multiple relations):
class Person(models.Model):
TYPE_CHOICES = (
('S', _('Student')),
('E', _('Researcher')),
)
class Meta:
permissions = (
('view_person', _('View person')),
)
index_together = (
('last_name', 'first_name'),
)
# Relations with other entities
topics = models.ManyToManyField('topics.Topic', blank=True, related_name='people')
competences = models.ManyToManyField('staff.Competence', blank=True, related_name='people', db_index=True)
# Person properties
cmsuser = models.OneToOneField(User, blank=True, related_name='person', null=True, db_index=True)
sebra_username = models.CharField(max_length=20, blank=True, db_index=True)
first_name = models.CharField(_('first name'), max_length=30, blank=True, db_index=True)
last_name = models.CharField(_('last name'), max_length=30, blank=True, db_index=True)
email = models.EmailField(_('email address'), blank=True, db_index=True)
username = models.CharField(_('username'), max_length=30, unique=True, db_index=True)
avatar = models.ImageField(upload_to='img/avatar/', blank=True)
web = models.URLField(_("web site"), blank=True)
cristin_profile = models.URLField(_('link to cristin profile'), blank=True)
twitter = models.CharField(_("twitter username"), max_length=20, blank=True)
telephone = models.CharField(
blank=True,
max_length=validators.MAX_LENGTH_PHONE,
validators=[validators.validate_phone_format]
)
telephone_country_code = models.ForeignKey(Country, null=True, blank=True, related_name='phone_person')
mobile = models.CharField(
blank=True,
max_length=validators.MAX_LENGTH_PHONE,
validators=[validators.validate_phone_format]
)
mobile_country_code = models.ForeignKey(Country, null=True, blank=True, related_name='mobile_person')
address = models.TextField(max_length=255, blank=True)
cv = models.FileField(_('Curriculum Vitae'), upload_to='attachments/cv/', blank=True)
vcard = models.FileField(_('Vcard'), upload_to='attachments/vcard/', blank=True)
person_type = models.CharField(choices=TYPE_CHOICES, max_length=1, blank=True)
extract = RichTextField(_('person extract'), blank=True, default='')
slug = AutoSlugField(populate_from=('first_name', 'last_name'))
def __unicode__(self):
if len(self.first_name) + len(self.last_name):
return '%s %s' % (self.first_name, self.last_name)
return self.username
def clean(self):
super(Person, self).clean()
if self.sebra_username.strip():
# here goes validation and checks on the related objects
def get_absolute_url(self):
if PersonDepartmentMembership.objects.filter(active__exact=True, person__exact=self):
return reverse('staff:profile_slug', kwargs={'slug': self.slug})
return ''
I understand that the bottleneck might be as well within my Admin class. This is the one we're using:
class PersonAdmin(ModelAdmin):
fields = (
'sebra_username', 'first_name', 'last_name', 'avatar', 'email', 'person_type', 'extract',
'topics', 'competences', 'web', 'cristin_profile', 'twitter', 'telephone_country_code',
'telephone', 'address', 'mobile_country_code', 'mobile', 'cv', 'vcard'
)
search_fields = ('sebra_username', 'first_name', 'last_name', 'email', 'departments__name')
list_filter = ('departments__name', 'research_groups__group_name', 'projects__project_name')
inlines = (SomeInline,)
class Media:
js = (
settings.STATIC_URL + 'js/jquery-1.9.0.min.js',
settings.STATIC_URL + 'js/jquery-ui-1.9.2.custom.min.js',
'modeltranslation/js/tabbed_translation_fields.js',
)
css = {
'screen': ('modeltranslation/css/tabbed_translation_fields.css',),
}
def formfield_for_manytomany(self, db_field, request=None, **kwargs):
if db_field.name == 'topics':
kwargs['queryset'] = get_objects_for_user(user=request.user, perms=('topics.view_topic',))
elif db_field.name == 'competences':
kwargs['queryset'] = get_objects_for_user(user=request.user, perms=('staff.view_competence',))
return super(PersonAdmin, self).formfield_for_manytomany(db_field, request, **kwargs)
def queryset(self, request):
if request.user.is_superuser:
return super(PersonAdmin, self).queryset(request)
return get_objects_for_user(user=request.user, perms=('staff.change_person',)).order_by('last_name')
def has_add_permission(self, request):
return request.user.has_perm('staff.add_person')
def has_delete_permission(self, request, obj=None):
return request.user.has_perm('staff.delete_person', obj)
def has_change_permission(self, request, obj=None):
return request.user.has_perm('staff.change_person', obj)
Could you give me any advice or suggest me any possible solution which we could integrate within the admin interface? :-)
Thanks in advance!
EDIT:
Using django-debug-toolbar, I can see that there are very few queries to django-guardian, and quite fast (all below 6ms). On the other hand, I have more than 7500 queries for a listview of 263, which slow the view up to 46 seconds to generate. Almost all of them are within my defined model, to load (I think) useless data: I suppose that just the name and object id are needed.
How can I limit the amount of data loaded within the queryset() method? Thanks.
If you're executing 7500 queries, your problem is probably that you're not loading related objects that you need - here's what I'd look at:
list_display setting that access related objects?Either eliminate calls to those things, or look at select_related.
Also - try changing the pagination of your admin views - if you reduce the number of records shown - how many fewer queries are there? That'll give you a clue as to how many of these problems you have.