I am having an almost identical problem to this one, posted in '09. It doesn't appear to be successfully solved, and I have been poring over related questions all day & implementing suggestions I have found.
I am attempting to pull dynamic (changes daily) data from an XML source (not a file) and insert into a mySQL database. When I attempt to pull the data without fwrite, the data returns without tags/keys in the output. Thus, I cannot properly assign the data to specific database fields.
I decided to write the information to a file and go from there, but the file aborts/"completes" at 334kb every single time. Unfortunately, I am a cURL novice and don't have the chops to see exactly what the issue is. Also, I am using vqMod for OpenCart to do this, otherwise I'd be using straight up PHP.
What am I missing?
Also, is there a way to pull the xml as an array instead of as a string (thereby potentially bypassing the fwrite step)? Or, should I be writing to some other file type?
Any suggestions or a redirect to a solved question I've missed are appreciated.
This is the relevant code, commented portions are fixes I've attempted:
$curl = curl_init();
$fp = fopen('dir/file.xml' , "w" );
//w+ will not download any information from the url - file is created but empty.
//AS IS, downloads first 334KB of file then lands on a blank page
//and a 500 error when any of these other options are implemented.
curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
curl_setopt($curl, CURLOPT_FILE, $fp);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
// curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// curl_setopt($curl, CURLOPT_TIMEOUT, 300);
// curl_setopt($curl, CURLOPT_NOPROGRESS, false);
// curl_setopt($curl, CURLOPT_RANGE, '0-1000');
// $data = array();
$data = curl_exec($curl);
fwrite($fp, $data);
curl_close($curl);
fclose($fp);
Update: Attempted to use simplexml_load_string instead of fwrite() to pull one product's information, but am still having limited success. Example of XML I am using:
<?xml version="1.0"?>
<response>
<root>
<part>
<![CDATA[PARTNUM]]>
</part>
<errorcode>0</errorcode>
<errormsg></errormsg>
<special>N</special>
<description>
<![CDATA[]]>
</description>
<price>75</price>
<weight>1.02</weight>
<webpurch>Y</webpurch>
<altnum>
<![CDATA[ALT-NUM]]>
</altnum>
<active>Y</active>
<redo>
<![CDATA[]]>
</redo>
<codes>
<code>
<fieldname>
<![CDATA[Mfr Part No]]>
</fieldname>
<fieldvalue>
<![CDATA[PARTNUM]]>
</fieldvalue>
</code>
<code>
<fieldname>
<![CDATA[Special Code 1]]>
</fieldname>
<fieldvalue>
<![CDATA[XYZ123]]>
</fieldvalue>
</code>
</codes>
<customtag>N</customtag>
<onhand>0</onhand>
<notes>
<![CDATA[PRODUCT-SPECIFIC NOTE]]>
</notes>
<mfr>
<mfr_name>
<![CDATA[MFR]]>
</mfr_name>
</mfr>
<altpartnums>
<altnum>
<![CDATA[PARTNUM.12]]>
</altnum>
</altpartnums>
<gtrue>N</gtrue>
<group>
<![CDATA[GROUP NAME]]>
</group>
<categories>
<cat>294</cat>
<cat>475</cat>
</categories>
</root>
</response>
This is an example of the $data return for multiple products when I do not use fwrite():
0
N
75
1.02
Y
Y
N
0
N
294
475
0
N
288
12
Y
Y
Y
18
Y
222
456
3786
I got it to work sans fwrite() using this code:
if(extension_loaded('curl')){
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, 30);
curl_setopt($curl, CURLOPT_NOPROGRESS, FALSE);
curl_setopt($curl, CURLOPT_MAXCONNECTS, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0);
$newparts = curl_exec($curl);
$xml = simplexml_load_string($newparts);
foreach($xml->modpart as $item){
if(($item->active == 'Y') || ($item->active == 'YES')){
$status = '1';
} else {
$status = '0';
}
if(!empty($item->mfr)){
$rs = $this->db->query("SELECT manufacturer_id FROM ".DB_PREFIX."manufacturer WHERE name = '".$item->mfr->mfr_name."'");
$mfr_id = $rs->row['manufacturer_id'];
}
if(!empty($item->codes)){
if($item->codes->code->fieldname == 'Mfr Part No'){
$mpn = $item->codes->code->fieldvalue;
}
}
$this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product (model, sku, mpn, quantity, date_available, sort_order, manufacturer_id, price, weight, status, date_added)
VALUES (
'".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."',
'".mysql_real_escape_string($item->sku)."',
'".$mpn."',
'".mysql_real_escape_string($item->mpn)."',
NOW(), 1000,
'".$mfr_id."',
'".mysql_real_escape_string($item->price)."',
'".mysql_real_escape_string($item->weight)."',
'".$status."',
NOW()
)");
$pr = $this->db->query("SELECT product_id FROM ".DB_PREFIX."product WHERE sku = '".$item->part."'");
$product_id = $pr->row['product_id'];
if(isset($product_id)){
$this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_description (product_id, language_id, name, meta_keyword, meta_description, description, tag)
VALUES(
'" . (int)$product_id . "',
'1',
'" . $item->part. "',
'" . $item->part. "',
'".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."',
'".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."',
'".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."'
)");
$this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_store SET product_id = '" . (int)$product_id . "', store_id = '0'");
$this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "url_alias SET query = 'product_id=" . (int)$product_id . "', keyword = '" .$item->part . "'");
foreach($item->cats as $cats){
$category = $cats->cat;
foreach($category as $category_id){
$this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_category SET product_id = '" . (int)$product_id . "', category_id = '" . (int)$category_id . "'");
}
}
$this->cache->delete('product');
}
}
curl_close($curl);
}
I still have had no luck with the timeout issue, but at least I can get my products into the database by some means. I will post the timeout solution if/when I find it.