Apache workers get stuck in process on Windows Server running a single PHP webapplication
I have a web application running in a mostly up-to-date WAMP stack since 2017. It serves around 200-300 users, as an internal backoffice app. The current WAMP stack: Two VMs running Windows Server 2016, on App server: Apache 2.4.58, PHP 8.2.12, on DB server: MySQL 8.0.33
It run without any significant problem until about half a year ago.
The main symptom the users experience is a white screen in there browser after trying to load any page, and the tab stuck in a "loading state". It happens for random users, and not all the time. I cannot determine any pattern in the frequency it happends or with which user. After deleting the PHP session cookie from the browser, it returns to the normal operations. All users use Chrome (company policy).
On the server side, I can see the user's request "stuck" on the mod_status page. If they try to refresh the site before the cookie deletion, they can hold down multiple workers in the "stuck" state.
By "stuck" I mean, the worker's "M - mode of operation" is in "W - Sending reply" (At least in http/1.1 Protocol) and the "SS - Seconds since beginning of most recent request" is way higher than the configured Timeout. After changing the protocol to http/2, the workers stuck in "C - Closing connection" with the high "SS" value.
Multiple workers "stuck" in "W" state - using http/1.1 protocol
Single worker "stuck" in "C" state - using http/2 protocol
I tried to reconfigure the Apache as best as I can, here is the relevant parts:
# Core config
ThreadLimit 512
ThreadsPerChild 512
ThreadStackSize 8388608
MaxRequestsPerChild 0
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 500
TimeOut 60
ProxyTimeout 60
RequestReadTimeout handshake=0 header=10-40,MinRate=500 body=20,MinRate=500
# Http2 config
Protocols h2 http/1.1
H2Direct On
H2MinWorkers 64
H2MaxWorkers 512
H2MaxSessionStreams 512
H2StreamMaxMemSize 1048576
H2WindowSize 1048576
H2MaxWorkerIdleSeconds 10
H2StreamTimeout 60
After the change in the Apache config did not work, nor the change to the http2 protocol, and the problem seems to be related to the PHP session handling, I try to reconfigure that too. Here is the current PHP session config:
[Session]
session.save_handler = files
session.save_path = "c:/tmp"
session.use_strict_mode = 1
session.use_cookies = 1
session.cookie_secure = 0
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 14500
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly = 1
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 14500
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = sha512
session.hash_bits_per_character = 5
And I tried to rewrite the session handler in my application. Here are the relevant parts of the the session handling class:
<?php
// Framework Session handler
class Session {
// Generic PHP session token
private $_session_id;
// Number of seconds after the session id is needs to be regenerated
private $_session_keepalive = 300;
// Error logging handler
private $_error = null;
/**
* @Function: public __construct($config);
* @Task: Default constructor for Framework session handler
* @Params:
* array $config: associative configuration array for construction
* Default: array() - Empty array
* Note: It can contain any of the class' configurable variables
* without the first underscore as key
* @Returns:
* Session
**/
public function __construct($config = array()) {
// Setting config
foreach(get_object_vars($this) as $key => $var) {
if($key != '_session_id' && isset($config[mb_substr($key,1)]))
$this->$key = $config[mb_substr($key,1)];
}
// Make sure use_strict_mode is enabled.
// use_strict_mode is mandatory for security reasons.
ini_set('session.use_strict_mode', 1);
ini_set('session.cookie_secure', 1);
// Start the session
$this->start();
// Create error logging handler
$this->_error = new Error_Logging();
}
/**
* @Function: public __destruct();
* @Task: Destructor for Framework Session handler
* @Params: None
* @Returns: void
**/
public function __destruct() {
// Destroy variables
foreach(get_object_vars($this) as $key => $var)
unset($this->$key);
// Releases the session file from write lock
session_write_close();
}
/**
* @Function: private start()
* @Task: Starts the PHP session
* @Params: none
* @Returns: none
**/
private function start() {
session_start();
// Store the session id
$this->_session_id = session_id();
// Set CreatedOn if not set
if(!$this->exists('CreatedOn'))
$this->set('CreatedOn', date('Y-m-d H:i:s'));
// Do not allow the use of old session id
$time_limit = strtotime(' - ' . $this->_session_keepalive . ' seconds');
if(!empty($this->get('DeletedOn', '')) && strtotime($this->get('DeletedOn', '')) <= $time_limit) {
session_destroy();
session_start();
$this->set('CreatedOn', date('Y-m-d H:i:s'));
// Store the new session id
$this->_session_id = session_id();
}
// Regenerate the session when older than required
if(strtotime($this->get('CreatedOn', '')) <= $time_limit) {
$this->regenerate();
}
}
/**
* @Function: private regenerate()
* @Task: Regenerates the current PHP session
* @Params: none
* @Returns: none
**/
public function regenerate() {
// Call session_create_id() while session is active to
// make sure collision free.
if(session_status() != PHP_SESSION_ACTIVE) {
$this->start();
}
// Get all session data to restore
$old_session_data = $this->get_all();
// Create a new non-conflicting session id
$this->_session_id = session_create_id();
// Set deleted timestamp.
// Session data must not be deleted immediately for reasons.
$this->set('DeletedOn', date('Y-m-d H:i:s'));
// Finish session
session_write_close();
// Set new custom session ID
session_id($this->_session_id);
// Start with custom session ID
$this->start();
// Restore the session data except CreatedOn and DeletedOn
if(isset($old_session_data['CreatedOn']))
unset($old_session_data['CreatedOn']);
if(isset($old_session_data['DeletedOn']))
unset($old_session_data['DeletedOn']);
if(!empty($old_session_data))
$this->set_multi($old_session_data);
}
/**
* @Function: public set($key, $val);
* @Task: Set Session variable
* @Params:
* mixed key: Key of the session array variable
* mixed val: Value of the session variable
* @Returns:
* bool
**/
public function set($key, $val) {
// Return variable
$response = false;
try{
// check if session is started
if(empty(session_id()))
throw new Exception('Session Error [0001]: Session is not started.');
// check if key is not null
if(empty($key))
throw new Exception('Session Error [0002]: Session key cannot be empty.');
// set session key
$this->write($key, $val);
$response = true;
}
// Handle errors
catch(Exception $e) {
$this->_error->log($e->getMessage());
}
return $response;
}
/**
* @Function: public get($key);
* @Task: Get session variable
* @Params:
* mixed key: Key of the session array variable
* mixed default: Default value if result is empty
* @Returns:
* bool/mixed
**/
public function get($key, $default = '') {
// Return variable
$response = false;
try{
// check if session is started
if(empty(session_id()))
throw new Exception('Session Error [0001]: Session is not started.');
// check if key is not null
if(empty($key))
throw new Exception('Session Error [0002]: Session key cannot be empty.');
// get session key if exists, else false
$response = $this->read($key, $default);
}
// Handle errors
catch(Exception $e) {
$this->_error->log($e->getMessage());
}
return $response;
}
/**
* @Function: public exists($key);
* @Task: Check if session variable exists
* @Params:
* mixed key: Key of the session array variable
* @Returns:
* bool
**/
public function exists($key) {
// Return variable
$response = false;
try{
// check if session is started
if(empty(session_id()))
throw new Exception('Session Error [0001]: Session is not started.');
// check if key is not null
if(empty($key))
throw new Exception('Session Error [0002]: Session key cannot be empty.');
// get if exists
$response = isset($_SESSION[$key]);
}
// Handle errors
catch(Exception $e) {
$this->_error->log($e->getMessage());
}
return $response;
}
/**
* @Function: public set_multi($params);
* @Task: Set multiple session variables
* @Params:
* array params: Associative array of key/val pairs to be set as session variables
* @Returns:
* bool
**/
public function set_multi($params) {
// Return variable
$response = false;
try{
// check if session is started
if(empty(session_id()))
throw new Exception('Session Error [0001]: Session is not started.');
// check if key is not null
if(empty($params))
throw new Exception('Session Error [0003]: Params array cannot be empty.');
$res = array();
foreach($params as $key => $val) {
// check if key is not null
if(empty($key))
throw new Exception('Session Error [0002]: Session key cannot be empty.');
// set session key
$this->write($key, $val);
$res[] = true;
}
// Check if all set succeded
$response = count($params) == count($res);
}
// Handle errors
catch(Exception $e) {
$this->_error->log($e->getMessage());
}
return $response;
}
/**
* @Function: public get_all();
* @Task: Get all session variables
* @Params: None
* @Returns:
* array
**/
public function get_all() {
// Return variable
$response = false;
try{
// check if session is started
if(empty(session_id()))
throw new Exception('Session Error [0001]: Session is not started.');
$res = array();
$keys = array_keys($_SESSION);
foreach($keys as $key) {
// get session key
$res[$key] = $this->read($key);
}
// Check if all set succeded
$response = $res;
}
// Handle errors
catch(Exception $e) {
$this->_error->log($e->getMessage());
}
return $response;
}
/**
* @Function: private write($key, $val);
* @Task: write session variable
* @Params:
* mixed key: key of the session variable to be stored
* mixed val: value of the session variable to be stored
* @Returns:
* void
**/
private function write($key, $val) {
$_SESSION[$key] = $val;
}
/**
* @Function: private read($key, $default);
* @Task: get session variable
* @Params:
* mixed key: key of the session variable to be retrieved
* mixed default: default value, if session not found
* @Returns:
* mixed
**/
private function read($key, $default = '') {
if(!isset($_SESSION[$key]))
return $default;
else
return $_SESSION[$key];
}
}
I do not know what else can I do, or where I screwed up. Any help is greatly appreciated. If anyone need any more information, feel free to ask!
As of today, the problem seems to be solved. But I'm not certain, that the solution is the right one, so If anyone has any more tips, it would be welcome.
The clue was the reoccurring error, MySQL server has gone away, but at first I thought it was a separate problem, that is why I did not include in the problem description above.
At first, I wrote a reconnect function in my DB handler class, and called it every time, before executing any query on the MySQL server. Here is the reconnect function:
/**
* @Function: private reconnect();
* @Task: Reconnect to database if the connection has gone away
* @Params: None
* @Returns: void
**/
private function reconnect() {
try {
if($this->_db === null || !(@$this->_db->ping())) {
if($this->_reconnect_count > $this->_reconnect_try_max) {
throw new Exception('Database Error [0012]: MySqli connector could not reconnect.');
}
else {
// Count the reconnect trys
$this->_reconnect_count++;
// Dump old connection
unset($this->_db);
// Create MySQLi connector
$this->_db = new mysqli($this->_host, $this->_user, $this->_pass, '', $this->_port);
// Check if MySQLi connection is established
if($this->_db->connect_errno != 0) {
// Wait before re-try
sleep($this->_reconnect_wait);
$this->reconnect();
}
else {
// Reset the reconnect counter
$this->_reconnect_count = 0;
}
}
}
}
catch(Exception $e) {
// Log the error
$this->_error->log($e->getMessage());
// Terminate connection
header('location: ' . get_url() . '/500.html');
die(0);
}
}
This method checks if the db connection is still live (with the $mysqli->ping() function), and if the connection is gone, then try to reconnect every second, up to _reconnect_try_max times.
But at first, it did not help, because it turns out, the ping() method is the one, which throws the error, instead of retuning a false as is expected.
So after adding an error control operator (@) before the ping() - as is in the code above - and at any point where the $mysqli->ping() is called, the 'Mysql gone away' errors disappeared, and there was no "stuck" Apache worker since than (as of the writing of this, 7 days straight).
- What Does :!: Mean?
- PHP: Is it possible to create a SplFileObject object from file contents (string)?
- PHP Laravel Check if Array is empty
- PHP / Apache / Node / Puppeteer - Error: Failed to launch the browser process
- WooCommerce how to display "you save X%" on checkout page
- Update custom order item meta in WooCommerce
- Is Laravel-admin 1.8.19 not compatible with laravel 11.8.0? [Your requirements could not be resolved to an installable set of packages.]
- PHP XAMPP Config
- How to import Wikipedia XML dump into MongoDB?
- Possible bug with PHP PDO and with PostgreSQL
- Split string on dots unless the dot is inside double quotes
- How to Dynamically Populate Schema in WordPress Using XPath Variables?
- /pusher/auth endpoint returning 404 in Laravel
- php .env with Docker
- Add 1 to variable inside of an echo expression
- Show values from a MySQL database table inside a HTML table on a webpage
- Get all alphanumeric words with length of either 4 or 12 characters
- Replace the trailing digits of a URI with corresponding slug text
- Wrap <b> tags around substrings matching a regular expression
- Match urls with a specific signature
- UTF-8 all the way through
- Get words delimited by "#$"
- How do you delete one part in the plaintext I scraped from the site?
- PHP Scrape HTML Between <pre> tags
- How to extract content or scrape data sets from website source page
- Create joined string by looping through key value array
- how to re-migrate a laravel migration after deleting the table
- Validate user input string only contains letters, numbers, and underscores
- How escape characters in a variable to be used in a regex pattern?
- How to make preg_match() match more than once?