I intend to do the following in CKAN: When a new package is added with an Excel in it, I convert the individual worksheets to .csv and add them to the package.
I do this with a plugin, using the notify function. The extension code is excecuted and the .csv files are added correctly, but still I receive an error that I cannot solve:
ckan | 2019-05-08 11:44:09,640 ERROR [ckan.model.modification] 'NoneType' object has no attribute 'transaction'
ckan | Traceback (most recent call last):
ckan | File "/usr/lib/ckan/venv/src/ckan/ckan/model/modification.py", line 77, in notify
ckan | observer.notify(entity, operation)
ckan | File "/usr/lib/ckan/venv/lib/python2.7/site-packages/ckanext_pxconverter-0.0.1-py2.7.egg/ckanext/pxconverter/plugin.py", line 53, in notify
ckan | convert_px_data(entity, resources)
ckan | File "/usr/lib/ckan/venv/lib/python2.7/site-packages/ckanext_pxconverter-0.0.1-py2.7.egg/ckanext/pxconverter/converter.py", line 134, in convert_px_data
ckan | upload_file(registry, os.path.join(tmpdir, "px_run_info.csv"), api_key, package_id=package.id, name="px_run_info.csv")
ckan | File "/usr/lib/ckan/venv/lib/python2.7/site-packages/ckanext_pxconverter-0.0.1-py2.7.egg/ckanext/pxconverter/converter.py", line 56, in upload_file
ckan | name=data['name'])
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/ckanapi/common.py", line 50, in action
ckan | files=files)
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/ckanapi/localckan.py", line 71, in call_action
ckan | return self._get_action(action)(context, data_dict)
ckan | File "/usr/lib/ckan/venv/src/ckan/ckan/logic/__init__.py", line 464, in wrapped
ckan | result = _action(context, data_dict, **kw)
ckan | File "/usr/lib/ckan/venv/src/ckan/ckan/logic/action/create.py", line 329, in resource_create
ckan | model.repo.commit()
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/vdm/sqlalchemy/tools.py", line 107, in commit
ckan | self.session.commit()
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 157, in do
ckan | return getattr(self.registry(), name)(*args, **kwargs)
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 906, in commit
ckan | self.transaction.commit()
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in commit
ckan | self._prepare_impl()
ckan | File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 432, in _prepare_impl
ckan | stx = self.session.transaction
ckan | AttributeError: 'NoneType' object has no attribute 'transaction'
I use the following code in a 'convert_xlsx' function to upload the CSV file:
from ckanapi import LocalCKAN
registry = LocalCKAN()
registry.action.resource_create(
package_id=data['package_id'],
url='dummy-value',
upload=open(csv_file_path, 'rb'),
name=data['name'])
To code in the plugin.py looks like this:
def notify(self, entity, operation=None):
if isinstance(entity, model.Package):
if operation != 'deleted':
resources = entity.resources_all
log.debug(resources)
convert_xlsx(entity, resources)
Any suggestions?
When a new package is added, CKAN creates the Package object. It starts a transaction (vdm.sqlalchemy.new_revision()), but before it commits it, it calls observer.notify() for any listeners, as we see in your stack trace. During this notify your code calls resource_create(), which again starts a transaction (bad) and then commits it. Somewhere in here the transaction/session/vdm get messed up and you get the error you see:
File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 432, in _prepare_impl
stx = self.session.transaction
AttributeError: 'NoneType' object has no attribute 'transaction'
The solution is in the notify() to enqueue a background job, which does the resource_create. So you do the changes to the model in a separate session/transaction. https://docs.ckan.org/en/2.8/maintaining/background-tasks.html