I have the following Eclipse version on Windows 10:
Version: 2020-09 (4.17.0)
Build id: 20200910-1200
I am using PyDev along with it.
In my code I am using selenium to make a number of url calls (web scraping). When it happens that a particular url is not present or at least not present in the way most of the urls I am reading are, I get the following error:
Traceback (most recent call last):
File "C:\Users\foobar\eclipse-workspace\WeatherUndergroundUnderground\historical\BWI_Fetch.py", line 44, in <module>
main(city, month_date, start_year, end_year)
File "C:\Users\foobar\eclipse-workspace\WeatherUndergroundUnderground\historical\BWI_Fetch.py", line 22, in main
driver.get(city_url);
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror
Exception ignored in: <function Popen.__del__ at 0x0000019267429F70>
Traceback (most recent call last):
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 945, in __del__
self._internal_poll(_deadstate=_maxsize)
File "C:\Users\foobar\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1344, in _internal_poll
if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
OSError: [WinError 6] The handle is invalid
When I get this particular error, eclipse is still running and pushing the red stop button does not work to end the program. I can usually use the red stop button for just about any other python program I have written, but this code/error seems to hang things. How can I end the process from within the Eclipse application?
The error in the stack trace is not really related to PyDev, so, the stack trace error is only really fixable in Selenium/Python (the error says that it's trying to access a process which is already dead on the __del__).
Now, related to the reason why PyDev wasn't able to kill it, I think that you probably have some process which spawned a subprocess and is not reachable anymore because the parent process died and thus it's not possible to create a tree to kill that process from the initial process launched in PyDev.
The actual code which does this in PyDev is: https://github.com/fabioz/winp/blob/master/native/winp.cpp#L208
I think that it should be possible to use the windows api to create a JobObject and then AssignProcessToJobObject and on kill also kill the JobObject so that it kills all associated processes so that things are setup in a way that that this doesn't happen, but this isn't currently done.
As a note, usually I have an alias for: taskkill /im python.exe /f (which will kill all the python.exe processes running in the machine) and it's what I usually use in such cases, so, if something like that happens I just kill all the python.exe processes in the machine.
Although note that if you spawned some other process... say, chrome.exe -- in that process tree, that process must also be killed for the initial shell that launched python to be really disposed.