I am trying to dump data into database from CSV. This is time-consuming that's the reason I decided to go with laravel queue.
The main weird difference between server and localhost is here:
$insertJob = (new StoreUser($data))->delay(Carbon::now()->addSeconds(3));
dispatch($insertJob);
$msg = [
'status' => '1'
];
echo json_encode($msg);
exit();
Here, in the above case, in localhost as soon as the job is dispatched it sends status - 1. So, that it pop ups a message "You will be notified with the email once the job is finished". But, in the server it keeps on showing uploading bar i.e, it doesn't send status back.
And after some time it says internal server error.
PHP ERROR LOG:
27351#27351: *114475 FastCGI sent in stderr: “PHP message: PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480....)
I don't get it even in localhost I am allocating 128MB and in server its around 1GB.
SUPERVISOR LOG:
2019-01-08 15:48:01,121 CRIT Supervisor running as root (no user in config file)
2019-01-08 15:48:01,122 WARN No file matches via include “/etc/supervisor/conf.d/*.conf”
2019-01-08 15:48:01,134 INFO RPC interface ‘supervisor’ initialized
2019-01-08 15:48:01,134 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2019-01-08 15:48:01,134 INFO supervisord started with pid 2947
2019-01-08 16:17:45,836 INFO spawnerr: can’t find command ‘/home/forge/site_address/php’
2019-01-08 16:17:46,837 INFO spawnerr: can’t find command ‘/home/forge/site_address/php’
2019-01-08 16:17:48,841 INFO spawnerr: can’t find command ‘/home/forge/site_address/php’
2019-01-08 16:17:51,845 INFO spawnerr: can’t find command ‘/home/forge/site_address/php’
2019-01-08 16:17:51,846 INFO gave up: project_queue entered FATAL state, too many start retries too quickly
2019-01-10 16:14:18,566 INFO spawned: ‘project_queue’ with pid 25817
2019-01-10 16:14:19,591 INFO success: project_queue entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
In case if you are curious about what is exactly there inside StoreUser
$db_header_obj = new Class_user();
$db_header = $db_header_obj->getTableColumns();
$base_user_header = new Base_user();
$base_user_columns = $base_user_header->getTableColumns();
$csv_file_path = storage_path('app/files/class_user/').$filename;
if (!ini_get("auto_detect_line_endings")) {
ini_set("auto_detect_line_endings", TRUE);
}
$csv = Reader::createFromPath($csv_file_path, 'r');
$csv->setOutputBOM(Reader::BOM_UTF8);
$csv->addStreamFilter('convert.iconv.ISO-8859-15/UTF-8');
$csv->setHeaderOffset(0);
$csv_header = $csv->getHeader();
$error_arr = array();
$row_number = array();
$error_row_numbers = array();
$loop = true;
while($loop){
$rec_arr = array();
$records = array();
$records_arr = array();
$stmt = (new Statement())
->offset($offset)
->limit($limit)
;
$records = $stmt->process($csv);
foreach ($records as $record)
{
$rec_arr[] = array_values($record);
}
$records_arr = $service->trimArray($rec_arr);
if(count($records_arr)>0)
{
foreach($records_arr as $ck => $cv){
$existing = NULL;
$class_user_arr = array();
//format datatype and check either the column should be inserted or not
foreach ($map_data as $mk => $mv) {
if(isset($mv)){
$data_type = $service->getDatabaseColumnType($table,$mv);
if($data_type == 'date' || $data_type == 'datetime' || $data_type == 'timestamp'){
$datetime = (array)$cv[$mk];
$dt = array_shift($datetime);
$dt = date('Y-m-d h:i:s', strtotime($dt));
$class_user_arr[$mv] = $dt;
}else{
$class_user_arr[$mv] = $cv[$mk];
}
}
}
//setting some boolean variable
$updated_base = false;
$updated_class = false;
$error_encountered = false;
$base_user_table_id = NULL;
$base_user_arr = array();
foreach($class_user_arr as $cvk => $cvv){
if(in_array($cvk,$base_user_columns))
{
$base_user_arr[$cvk] = $cvv;
}
}
DB::beginTransaction();
//trying to insert into base or first table
try{
$base_user_row = Base_user::updateOrCreate(
['base_id' => $base_user_arr['base_id']],
$base_user_arr
);
if ($base_user_row->wasRecentlyCreated === true) {
$base_user_row->created_by = $this->data['user_id'];
}else{
$base_user_row->updated_by = $this->data['user_id'];
}
$base_user_row->save();
$base_user_table_id = $base_user_row->id;
$updated_base = true;
} catch (\Exception $e) {
$error_encountered = true;
$error_arr[] = $e->getMessage();
$error_row_numbers[] = $row_no;
}
//Using row id from first table inserting data into table number 2
if($error_encountered == false){
try{
$class_user_row = class_user::updateOrCreate(
['base_user_id' => $base_user_table_id],
$class_user_arr
);
if ($class_user_row->wasRecentlyCreated === true) {
$class_user_row->created_by = $this->data['user_id'];
}else{
$class_user_row->updated_by = $this->data['user_id'];
}
$class_user_row->save();
} catch (\Exception $e) {
$error_encountered = true;
$error_arr[] = $e->getMessage();
$error_row_numbers[] = $row_no;
DB::rollback();
}
}
DB::commit();
$row_no = $row_no + 1;
}
$offset = $offset + $limit;
}else{
$activity = new Activity();
$activity->url = $this->data['url'];
$activity->action = 'store';
$activity->description = $table;
$activity->user_id = $this->data['user_id'];
$activity->created_at = date('Y-m-d H:i:s');
$activity->save();
$arr_data = [
'filename' => $filename,
'user_name' => $this->data['user_name'],
'error' => $error_arr,
'error_row_numbers' => $error_row_numbers
];
//informing to the user about the completion of job
Mail::to($this->data['user_email'])->send(new CSVImportJobComplete($arr_data));
$loop = false;
}
}
if (!ini_get("auto_detect_line_endings")) {
ini_set("auto_detect_line_endings", FALSE);
}
I am out of clue, what might be the reason. Let me know if you need any further information.
UPDATE:
production.ERROR: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) I am pretty much sure updating allocating memory is not the solution. As I am dealing with it successfully on localserver with 128MB and not in server even with 2GB
After a lot of searching, I found it was because of one of the simple difference in .env file.
I need to update
QUEUE_DRIVER=sync
to
QUEUE_DRIVER=database
in the server.