Python2.7 / Centos7.5 / Apache2.4.6 + mod_wsgi
I'm using flask/jinja to generate some interactive html forms and saving data to the session file between changes (flask_session / type=filesystem). Its randomly breaking with an Unpickling error during the _prune function of werkzeug's contrib/cache.py and I'm at a loss for why. Deleting the session files would fix the issue until it cropped up again. I've since been able to reproduce the error, by forcing the session file to grow in size (making the form longer), so I suspect the size is related, but it doesn't entirely explain why they were happening in the first place. Is there some size limitations of server-side session files / pickles that I'm running into? It's not perfectly predictable, I just know that when I start overloading things it will eventually happen. AFAIK nothing is being sent over to client-side except the jinja-rendered html.
I load a decent-sized OrderedDict as a session item:
session['saved'] = OrderedDict(items_list)
The depth of the dict is only 3 levels, with a total of ~ 300 keys when it runs the risk of crashing.
I've tried changing the session items threshold, changing the permanence of the session file, updating werkzeug, and I have a workaround by forcing deletion of the session file on the UnpicklingError (which doesn't seem to affect the client-side experience at all). But that's a manual edit of the werkzeug lib file and is definitely not a long-term solution.
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1816, in full_dispatch_request
return self.finalize_request(rv)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1833, in finalize_request
response = self.process_response(response)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 2114, in process_response
self.session_interface.save_session(self, ctx.session, response)
File "/usr/lib64/python2.7/site-packages/flask_session/sessions.py", line 355, in save_session
total_seconds(app.permanent_session_lifetime))
File "/usr/lib/python2.7/site-packages/werkzeug/contrib/cache.py", line 815, in set
self._prune()
File "/usr/lib/python2.7/site-packages/werkzeug/contrib/cache.py", line 764, in _prune
expires = pickle.load(f)
UnpicklingError: invalid load key, '*'.
from flask import Flask, session, request, render_template
from flask_session import Session
from collections import OrderedDict
app = Flask(__name__)
SESSION_TYPE = app.config['SESSION_TYPE'] = 'filesystem'
SESSION_FILE_DIR = app.config['SESSION_FILE_DIR'] = os.path.join(local_dir,'flask_session')
SESSION_FILE_THRESHOLD = app.config['SESSION_FILE_THRESHOLD'] = 100
Session.init_app(app)
@app.route('/', defaults={'path': ''}, methods=['POST','GET'])
@app.route('/<path:path>', methods=['POST','GET'])
def configuration(path):
#Do Some Stuff
session['saved'] = OrderedDict(items_list)
return render_template('configuration.html', session=session)
% for element, details in session.saved.items()
<tr id="{{element}}" name="{{details['type']}}">
<td>
<select id="{{element}}" name="{{element}}">
% for item, desc in details["items"]
<option value="{{item}}">{{desc}}</option>
% endfor
</select>
</td>
</tr>
def _prune(self):
if self._threshold == 0 or not self._file_count > self._threshold:
return
entries = self._list_dir()
now = time()
for idx, fname in enumerate(entries):
try:
remove = False
with open(fname, "rb") as f:
expires = pickle.load(f)
remove = (expires != 0 and expires <= now) or idx % 3 == 0
if remove:
os.remove(fname)
except (IOError, OSError):
pass
# Add Exception to delete file with pickle errors
except pickle.UnpicklingError:
os.remove(fname)
self._update_count(value=len(self._list_dir()))
I don't see it being relevant, considering it should all be server side, but running the same scripts in a Windows environment with the flask development server doesn't present the pickling error. Instead I get a socket error in self.flush() - An established connection was aborted by the software in your host machine. There's no crash either, it just carries on. I suspect that's just error handling, but thought it was worth mentioning.
Turns out, there was another file in the flask_session directory that werkzeug/contrib/cache was trying to load and check if it needed pruning:
.gitignore
*facepalm*
The answer here, is that No, it's not a size limit, and no there is no corruption. the library was trying to prune files based purely on directory contents.