Flask app no longer working from cygwin when set to developer mode
This problem occurred immediately after attempting to run my python flask app in the background using:
$python app.py &
This immediately failed. Afterwards, any future attempts to run the app, which I have done before with no problem, ending up providing this error:
$ python app.py
Running on http://127.0.0.1:8050/
Debugger PIN: 962-843-370
* Serving Flask app "app" (lazy loading)
* Environment: development
* Debug mode: on
2 [main] python3.6m 37104 child_info_fork::abort: unable to remap _lbfgsb.cpython-36m-x86_64-cygwin.dll to same address as parent (0x48E0000) - try running rebaseall
Traceback (most recent call last):
File "app.py", line 644, in <module>
app.run_server(debug=util.DEBUG)
File "/cygdrive/c/Users/mkupfer/Desktop/my_documents/01_Visualizations/eurostat/venv/lib/python3.6/site-packages/dash/dash.py", line 1293, in run_server
**flask_run_options)
File "/cygdrive/c/Users/mkupfer/Desktop/my_documents/01_Visualizations/eurostat/venv/lib/python3.6/site-packages/flask/app.py", line 943, in run
run_simple(host, port, self, **options)
File "/cygdrive/c/Users/mkupfer/Desktop/my_documents/01_Visualizations/eurostat/venv/lib/python3.6/site-packages/werkzeug/serving.py", line 812, in run_simple
reloader_type)
File "/cygdrive/c/Users/mkupfer/Desktop/my_documents/01_Visualizations/eurostat/venv/lib/python3.6/site-packages/werkzeug/_reloader.py", line 275, in run_with_reloader
sys.exit(reloader.restart_with_reloader())
File "/cygdrive/c/Users/mkupfer/Desktop/my_documents/01_Visualizations/eurostat/venv/lib/python3.6/site-packages/werkzeug/_reloader.py", line 132, in restart_with_reloader
close_fds=False)
File "/usr/lib/python3.6/subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
restore_signals, start_new_session, preexec_fn)
BlockingIOError: [Errno 11] Resource temporarily unavailable
The error seems to stem from running in developer mode since when I run with app.run_server(debug=False) (btw, util.DEBUG is set to True in my local environment), the app works fine, but then I don’t get hot reloading which is important for me.
I’ve tried rebasing cygwin as per this post https://superuser.com/a/194537/276726, but this didn’t fix anything.
I've also tried creating a special rebase file following the steps in this post, but that didn't help either.
The app works in development mode from my windows command line, so that is my temporary fix for now, but I would love to get my Cygwin setup running properly again.
Thanks for the help!
You're running into a quite common problem in the Cygwin world. There are many URLs mentioning (dealing with) it, but I'm going to list the ones that I came across:
- [SO]: Cygwin error: “-bash: fork: retry: Resource temporarily unavailable”
- [SO]: Cygwin issue - unable to remap; same address as parent
- [SuperUser]: Cygwin fatal error unable to remap.. What does it mean?
- [WordPress]: Cygwin and Rails – unable to remap to same address as parent; died waiting for dll loading, errno 11
- [SO]: Cygwin error: “child_info_fork::abort: Loaded to different address:”
The "behind the scenes magic" is very well explained in [Cygwin]: Problems with process creation (emphases are mine):
The semantics of
forkrequire that a forked child process have exactly the same address space layout as its parent. However, Windows provides no native support for cloning address space between processes and several features actively undermine a reliableforkimplementation. Three issues are especially prevalent:
- DLL base address collisions. Unlike *nix shared libraries, which use "position-independent code", Windows shared libraries assume a fixed base address. Whenever the hard-wired address ranges of two DLLs collide (which occurs quite often), the Windows loader must "rebase" one of them to a different address. However, it may not resolve collisions consistently, and may rebase a different dll and/or move it to a different address every time. Cygwin can usually compensate for this effect when it involves libraries opened dynamically, but collisions among statically-linked dlls (dependencies known at compile time) are resolved before
cygwin1.dllinitializes and cannot be fixed afterward. This problem can only be solved by removing the base address conflicts which cause the problem, usually using therebasealltool.- Address space layout randomization (ASLR). Starting with Vista, Windows implements ASLR, which means that thread stacks, heap, memory-mapped files, and statically-linked dlls are placed at different (random) locations in each process. This behaviour interferes with a proper
fork, and if an unmovable object (process heap or system dll) ends up at the wrong location, Cygwin can do nothing to compensate (though it will retry a few times automatically).
A troubleshooter is attempted in [Cygwin.FAQ]: 4.45. How do I fix fork() failures? (emphases still mine). With the risk of spamming the answer, I'm going to paste it here:
Unfortunately, Windows does not use the fork/exec model of process creation found in UNIX-like OSes, so it is difficult for Cygwin to implement a reliable and correct
fork(), which can lead to error messages such as:
- unable to remap somedll to same address as parent
- couldn't allocate heap
- died waiting for dll loading
- child -1 - died waiting for longjmp before initialization
- STATUS_ACCESS_VIOLATION
- resource temporarily unavailable
Potential solutions for the above errors:
Restart whatever process is trying (and failing) to use
fork(). Sometimes Windows sets up a process environment that is even more hostile tofork()than usual.Ensure that you have eliminated (not just disabled) all software on the BLODA.
Switch from 32-bit Cygwin to 64-bit Cygwin, if your OS and CPU support that. With the bigger address space fork() is less likely to fail.
Try setting the environment variable CYGWIN to "detect_bloda", which enables some extra debugging, which may indicate what other software is causing the problem.
See this mail for more information.
Force a full rebase: Run rebase-trigger fullrebase, exit all Cygwin programs and run Cygwin setup.
By default, Cygwin's setup program automatically performs an incremental rebase of newly installed files. Forcing a full rebase causes the rebase map to be cleared before doing the rebase.
See /usr/share/doc/rebase/README and /usr/share/doc/Cygwin/_autorebase.README for more details.
Please note that installing new packages or updating existing ones undoes the effects of rebase and often causes
fork()failures to reappear.See the process creation section of the User's Guide for the technical reasons it is so difficult to make
fork()work reliably.
In order to reproduce the problem, I used:
- Cygwin 32:
- The chance of running into the issue is much higher
- It's not my main Cygwin env
- Python 3.6.4 + VEnv
- Located at /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0
I tried reproducing you exact behavior (with _lbfgsb*.dll), but pip -v install scipy failed to build it.
Since [GitHub.SciPy]: Installing SciPy on Windows describes a quite complex process, and I'd have no guarantee that at the end I'd be able to reproduce the problem, I tried with numpy's .dlls (numpy was successfully installed as a scipy dependency), but I wasn't able to (as a side effect, import numpy loads a bunch of .dlls), but calling fork (via subprocess.Popen) didn't fail.
Then I decided to take matter into my own hands, and create a small program that loads some .dlls, then forks itself (again, via subprocess.Popen), to make the problem as reproducible as possible.
dll.c:
#include <stdio.h>
#if defined(_WIN32)
# define DLL_EXPORT __declspec(dllexport)
#else
# define DLL_EXPORT
#endif
DLL_EXPORT int test() {
printf("[%s] (%d) - [%s]\n", __FILE__, __LINE__, __FUNCTION__);
}
code.py:
#!/usr/bin/env python3
import sys
import os
import subprocess
import time
import select
import random
import ctypes
DLLS = [os.path.join(os.path.dirname(__file__), "dll{:d}.dll".format(item)) for item in range(2)]
def main():
random.seed(os.getpid())
random.shuffle(DLLS)
if len(sys.argv) == 1:
print("Python {:s} on {:s}\n".format(sys.version.replace("\n", ""), sys.platform))
print("Process 0x{:08X}".format(os.getpid()))
for dll in DLLS:
ctypes.cdll.LoadLibrary(dll)
idx = 0
while sys.stdin not in select.select([sys.stdin], [], [], 1)[0]:
p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)])
#p.communicate()
time.sleep(1)
idx += 1
else:
sleep_time = 3
print("Process 0x{:08X} (inner) will end in {:d} seconds".format(os.getpid(), sleep_time))
time.sleep(sleep_time)
if __name__ == "__main__":
main()
Notes:
- The scenario is a bit different, instead of a Python Extension Module, I have a regular (dummy) .dll, which I attempt loading it via [Python.Docs]: ctypes - A foreign function library for Python. This should make no difference, as no matter how Python sees it (as an module, or as an external .dll) it will still have to load it into the process (the same way)
- The scenario is:
- I load 2 such .dlls (actually it's the same .dll copied under a different name, so they both have the same Preferred Base Address), so the 1st .dll will probably be loaded at that address, while the next .dll will be loaded at a different one
- Then I fork the process, and in the child, based on a random factor, I switch the .dll loading order
- When the 2nd .dll will be loaded at the preferred base, there'll be an inconsistency with the parent process, yielding the error
- At the beginning everything was in my cwd, then (to be closer to your problem), I created a Python package with the files. Note that I didn't do it the proper way (installation via a setup.py file), but manually copying everything
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q054370263]> ~/sopr.sh *** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages *** [prompt]> [prompt]> uname -a CYGWIN_NT-10.0-WOW cfati-5510-0 2.11.2(0.329/5/3) 2018-11-08 14:30 i686 Cygwin [prompt]> ls code.py dll.c scipy.txt [prompt]> # List the currently installed packages in the !!! VEENV !!! Python [prompt]> ls -l ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages total 33 drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 __pycache__ -rw-r--r-- 1 cfati None 126 Jan 30 01:40 easy_install.py drwxr-xr-x+ 1 cfati None 0 Feb 2 21:41 numpy drwxr-xr-x+ 1 cfati None 0 Feb 2 21:41 numpy-1.16.1.dist-info drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pip drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pip-19.0.1.dist-info drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pkg_resources drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 setuptools drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 setuptools-40.7.1.dist-info drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 wheel drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 wheel-0.32.3.dist-info [prompt]> [prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263 /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python: No module named q054370263 [prompt]> # Create the package in site-packages dir [prompt]> mkdir ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263 [prompt]> cp code.py ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py [prompt]> gcc -fPIC -shared -o ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll dll.c [prompt]> cp ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll1.dll [prompt]> ls code.py dll.c scipy.txt [prompt]> ls -l ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263 total 260 -rwxr-x--- 1 cfati None 1012 Feb 3 12:39 __main__.py -rwxr-xr-x 1 cfati None 129844 Feb 3 12:22 dll0.dll -rwxr-xr-x 1 cfati None 129844 Feb 3 12:22 dll1.dll [prompt]> [prompt]> # Attempt to reproduce the problem by simply running the package [prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263 Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin Process 0x00001B38 18 [main] python3 21616 child_info_fork::abort: address space needed by 'dll0.dll' (0xD90000) is already occupied Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 37, in <module> main() File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 25, in main p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)]) File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) BlockingIOError: [Errno 11] Resource temporarily unavailable [prompt]> [prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263 Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin Process 0x000055E8 Process 0x00005764 (inner) will end in 3 seconds 1 [main] python3 21224 child_info_fork::abort: address space needed by 'dll1.dll' (0x6D0000) is already occupied Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 37, in <module> main() File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 25, in main p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)]) File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) BlockingIOError: [Errno 11] Resource temporarily unavailable
So, the error is quite reproducible. I'd also like to add here the .dll details (Dependency Walker):
Why doesn't a regular rebase (rebaseall) solve the problem?
Cygwin packages have a post install script that calls rebase on their .dlls
rebase searches in standard library paths (/lib, /usr/lib, ...). That can be adjusted, according to /usr/share/doc/Cygwin/_autorebase.README:
Packages can make the potential locations of such dynamic objects known by dropping a file (named after the package) in /var/lib/rebase/dynpath.d/. If any dynamic objects are installed by users, these locations should be advertised in /var/lib/rebase/user.d/ (the file name should be identical to the user name if there are multiple users on this system)
Python requires such an adjustment, due to packages that may contain .dlls
pip like packages do not have a post install script (that would rebase their .dlls)
The VEnv is in the user's home path which is not in the standard library paths (so, even a rebaseall will ignore them)
Note that all the rebased .dlls are stored in a DB: /etc/rebase.db(.${ARCH}).
[prompt]> ls /var/lib/rebase/dynpath.d/ perl python2 python3 [prompt]> cat /var/lib/rebase/dynpath.d/python3 /usr/lib/python3.6/site-packages [prompt]> ls /var/lib/rebase/user.d/ [prompt]> [prompt]> grep -r "dll0.dll" /etc/rebase.db.i386 [prompt]>
In order the .dlls to be picked up by rebase tools, they need to be advertised. That can be done in 2 ways:
Specify them in one of the custom locations, so at the next full rebase they will no longer be ignored (simply add the VEnv dir, and others if any):
[prompt]> echo /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0 >/var/lib/rebase/user.d/${USER} [prompt]> cat /var/lib/rebase/user.d/cfati /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0Manually rebase the .dlls
Both of them worked for me, but I'm going to exemplify on the 2nd variant only (as it's simpler). The process consists of 2 steps:
Create a list of all .dlls that need to be rebased
[prompt]> find /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0 -name "*.dll" -o -name "*.so">/tmp/to_rebase.txt [prompt]> ls -l /tmp/to_rebase.txt -rw-r--r-- 1 cfati None 1773 Feb 3 13:05 /tmp/to_rebase.txt [prompt]> cat /tmp/to_rebase.txt /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_multiarray_module_test.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_multiarray_tests.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_operand_flag_tests.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_rational_tests.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_struct_ufunc_tests.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/core/_umath_tests.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/random/mtrand.cpython-36m-i386-cygwin.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll1.dllPerform the rebase
- All the Cygwin processes must be closed (including services: e.g. sshd)
- I launched the command from dash.exe (started directly from Win, from Cygwin bin dir), and not from Mintty (notice the prompt)
$ /bin/rebaseall -v -T /tmp/to_rebase.txt ... /usr/bin/cygargp-0.dll: new base = 6e7c0000, new size = 20000 /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll1.dll: new base = 6e7e0000, new size = 30000 /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll: new base = 6e810000, new size = 30000 /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/numpy/random/mtrand.cpython-36m-i386-cygwin.dll: new base = 6e840000, new size = 280000 ...
The updated Dependency Walker window (check its Preferred Base, and compare it to the one in the previous image):
And also the rebase DB "query" (now back from Mintty):
[prompt]> grep -r "dll0.dll" /etc/rebase.db.i386 Binary file /etc/rebase.db.i386 matches
What's more important, running the code:
[prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263 Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin Process 0x000052D0 Process 0x00004634 (inner) will end in 3 seconds Process 0x00004864 (inner) will end in 3 seconds Process 0x00005CFC (inner) will end in 3 seconds Process 0x00005A5C (inner) will end in 3 seconds Process 0x00005098 (inner) will end in 3 seconds Process 0x00005840 (inner) will end in 3 seconds Process 0x000058C4 (inner) will end in 3 seconds Process 0x000051DC (inner) will end in 3 seconds Process 0x00001A5C (inner) will end in 3 seconds Process 0x00003D2C (inner) will end in 3 seconds Process 0x00000DA0 (inner) will end in 3 seconds
As seen it is able to fork itself multiple times (it only stopped because I pressed Enter).
- Webscraping Roblox
- Remove the mandatory field label 'This field is required.' and fix the bug with 'clean_email'
- How to plot a Probability Density Function in Python?
- How large is a fresh install of Python?
- Appending new elements into an empty list
- Simple way to measure cell execution time in ipython notebook
- PyAudio working, but spits out error messages each time
- Reportlab show page number and page count IF there is more than one page in a document
- How to read SharePoint Online (Office365) Excel files into Python specifically pandas with Work or School Account?
- How to set a column which suffix name is based on a value in another column
- Debugging Python C++ extension from Visual Studio Code on Linux
- How can I get all users on Google admin_sdk?
- csv.Error: iterator should return strings, not bytes
- How to check if an object has an attribute?
- How to use selenium with proxy auth in headless mode?
- Is there a way to exit a pytest test and continue to the next one?
- Returning the lowest index for the first non whitespace character in a string in Python
- Formatting exceptions as Python does
- Prime factorization using list comprehension in Python
- Why does the power spectrum E(k) of my velocity field follow 𝑘 ^(−(n−1)) instead of 𝑘^(−n)?
- How to merge dataframes over multiple columns and split rows?
- How to create a Sympy IndexedBase using a custom subclass of Symbol?
- Removing dynamically an element from a list
- Returning boolean if set is empty
- Can variables be decorated?
- Fast(est) exponentiation of numpy 3D matrix
- Removing an element from a list based on a condition
- Printing elements of dictionary line by line
- Matplotlib does not display the hatch of a patch in a legend
- Python win32com - Class not registered error