Search code examples
djangodjango-import-export

Django Import Export - UNIQUE constraint failed

I am following this documentation on how to use django-import-export:

https://django-import-export.readthedocs.io/en/latest/getting_started.html#declaring-fields

I have an excel sheet that looks like below

enter image description here

I want to store the data in this model:

class ExcelData(models.Model):
    var1 = models.CharField(max_length=200)
    var2 = models.CharField(max_length=200,unique=True)
    var3 = models.CharField(max_length=200)
    var4 = models.CharField(max_length=200)

This is how far I got:

@admin.register(ExcelData)
class ViewAdmin(ImportExportModelAdmin):
    exclude = ('id',)

class ExcelDataResource(resources.ModelResource):
    var1 = Field(attribute='var1', column_name='Name')
    var2 = Field(attribute='var2', column_name='SAP_ID')
    var3 = Field(attribute='var3', column_name='Abbreviation')
    var4 = Field(attribute='var4', column_name='Max. Capa')

    class Meta:
        model = ExcelData
        import_id_fields = ('var2',)
        exclude = ('id',)

This is what I get:

enter image description here

Here is the CSV File:

http://www.sharecsv.com/s/9d1112392cd7f10378de7fc0811dd0c9/REAL_CSV_SIMPLE.csv

When I try to import multiple rows like this:

enter image description here

I get this error:

Line number: 2 - UNIQUE constraint failed: myapp_exceldata.var2
b, e, h, k
Traceback (most recent call last):
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 396, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: myapp_exceldata.var2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Users\...\env\lib\site-packages\import_export\resources.py", line 527, in import_row
self.save_instance(instance, using_transactions, dry_run)
File "D:\Users\...\env\lib\site-packages\import_export\resources.py", line 320, in save_instance
instance.save()
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 746, in save
force_update=force_update, update_fields=update_fields)
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 784, in save_base
force_update, using, update_fields,
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 887, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 926, in _do_insert
using=using, raw=raw,
File "D:\Users\...\env\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "D:\Users\...\env\lib\site-packages\django\db\models\query.py", line 1204, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "D:\Users\...\env\lib\site-packages\django\db\models\sql\compiler.py", line 1384, in execute_sql
cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 100, in execute
return super().execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 396, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: myapp_exceldata.var2

Line number: 3 - UNIQUE constraint failed: myapp_exceldata.var2
c, f, i, l
Traceback (most recent call last):
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 396, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: myapp_exceldata.var2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Users\...\env\lib\site-packages\import_export\resources.py", line 527, in import_row
self.save_instance(instance, using_transactions, dry_run)
File "D:\Users\...\env\lib\site-packages\import_export\resources.py", line 320, in save_instance
instance.save()
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 746, in save
force_update=force_update, update_fields=update_fields)
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 784, in save_base
force_update, using, update_fields,
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 887, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "D:\Users\...\env\lib\site-packages\django\db\models\base.py", line 926, in _do_insert
using=using, raw=raw,
File "D:\Users\...\env\lib\site-packages\django\db\models\manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "D:\Users\...\env\lib\site-packages\django\db\models\query.py", line 1204, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "D:\Users\...\env\lib\site-packages\django\db\models\sql\compiler.py", line 1384, in execute_sql
cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 100, in execute
return super().execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "D:\Users\...\env\lib\site-packages\django\db\backends\utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "D:\Users\...\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 396, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: myapp_exceldata.var2

Here is the CSV file:

http://www.sharecsv.com/s/3f09eb14f7916933604481fd999d03db/REAL_CSV_FULL.csv

Thank you for any suggestions

Solution

  • The way import works is the following:

    • You need a unique identifier so that when importing, the import-export module "knows" whether it needs to create a new row or modify an existing row.
    • If that unique identifier is not the model's id field (which in your case is the auto-incremented row number), then it must be some other unique field. None of the fields var1 ... var4 on your model are declared as unique, so as you've defined your model currently, it's impossible to import.
    • You can use multiple "id fields" if a combination of fields is unique together, but that's also not the case on your model.
    • The import module with actually do a get_or_create() call on your database, using the unique_id_fields as the get parameters. This should never return more than 1 row, if it were to return multiple rows, the import would crash.

    To give you an idea, and assuming SAP_ID is a unique identifier (I doubt Max. Capacity is, the name doesn't indicate it would be), this should be your model:

    class ExcelData(models.Model):
    var1 = models.CharField(max_length=200)
    var2 = models.CharField(max_length=200, unique=True)
    var3 = models.CharField(max_length=200)
    var4 = models.CharField(max_length=200)

    Note your model also has the implicit field id, which import-export knows about and you have to decide what to do with it, since it's not in your import data. This is why you're receiving the error you mentioned, because it's a field in your model that you haven't assigned to anything. Best is to exclude it, since it's not relevant here (Django will auto-increment a new id if the row doesn't exist yet, like when you create the model with ExcelData.objects.create()):

    class ExcelDataResource(resources.ModelResource):
    var1 = Field(attribute='var1', column_name='Name')
    var2 = Field(attribute='var2', column_name='SAP_ID')
    var3 = Field(attribute='var3', column_name='Abbreviation')
    var4 = Field(attribute='var4', column_name='Max. Capa')

    class Meta:
        model = ExcelData
        import_id_fields = ('var2',)
        exclude = ('id',)

    With this code and the following csv file, it works, I've tested it:

    Name,SAP_ID,Abbreviation,Max. Capa
a,d,g,j
b,e,h,k
c,f,i,l

    When trying to import a second time or if there's already some data in the database where the value of var2 is d, e or f, you'll get UNIQUE constraint failed errors for exceldata.var2.

    Note: If Max. Capa really is your unique id field, then you shouldn't assign it to var4 but to id. That would also work, although as I mentioned above I doubt that's your desired behaviour.

    UPDATE/Note 2: There's a bug in django-import-export that will cause UNIQUE constraints exceptions when you try to update existing data: This happens if your unique_id_field has a different name than your column_name in the data you import (e.g. var2 for SAP_ID) and you try to re-import data which contains an already existing row with that same id (e.g. because you want to change the other values). It should update the row, but currently it throws an exception. This doesn't explain the exception on var4 (that's because you set var4 to be unique as well and if you import a different row with a duplicate var4 (Max. Capa) value, then you'll also get an exception).