1

I am working with an API to update our product information. The API allows me to do a bulk item upload of 20 items per call. In order to update all 1000+ items I have to loop through this script a number of times. This script basically compiles an XML of 20 items and uses curl to send it, then grabs the next 20 items creates XML and repeat. When I limit the outter loop to, say 15, it works fine and returns a success message for each API call. It fails at random times I've had it send up to 40 API calls before failing and I've also had it fail after 10 calls.

I can't find anything in the error logs as to why I am getting a 500 back. I've reached out to the folks managing the API to see if they have any idea.

My best guess is it has something to do with a timeout.

$query = "SELECT DISTINCT 
`id`,
`link`, 
`price`, 
`ship_group`,
`fixed_ship`, 
`do_not_sell`, 
`shipping_weight`, 
`shipping_length`, 
`shipping_width`, 
`shipping_height`, 
`title`, 
`brand`, 
`inventory`, 
`available`,
`type`,
`min_order_qty`,  
`image`, 
`call`, 
`allow_in_cart`, 
`allow_backorder` FROM $table WHERE `id` > 10001 AND `price` > 0 AND `base_match` = 'm' ORDER BY `id`";

$resultID = mysql_query($query, $linkID) or die(mysql_error());

//total number of rows divided by 20 rounded up to nearest whole number
//call createItems that amount of times 
$cycles =  ceil(mysql_num_rows($resultID) / 20);


$i = 0;
while( $i < 20 ){
//tried sleeping between calls but this didnt help
    sleep(2);
$i++;

$x = 0;
  $xml_output = "\t<items>\n";
    while( $x < 20){
        $row = mysql_fetch_assoc($resultID);
        $x++;

    $itemArray[$x]['id'] = $row['id'];
        if ($itemArray[$x]['id'] == "") {
            break;  
         };
    $itemArray[$x]['link'] = $row['link'];
    $itemArray[$x]['price'] = $row['price'];
    $itemArray[$x]['ship_group'] = $row['ship_group'];
    $itemArray[$x]['fixed_ship'] = $row['fixed_ship'];
    $itemArray[$x]['do_not_sell'] = $row['do_not_sell'];
    $itemArray[$x]['shipping_weight'] = $row['shipping_weight'];
    $itemArray[$x]['shipping_length'] = $row['shipping_length'];
    $itemArray[$x]['shipping_width'] = $row['shipping_width'];
    $itemArray[$x]['shipping_height'] = $row['shipping_height'];
    $itemArray[$x]['title'] = $row['title'];
    $itemArray[$x]['brand'] = $row['brand'];
    $itemArray[$x]['inventory'] = $row['inventory'];
    $itemArray[$x]['type'] = $row['type'];
    $itemArray[$x]['min_order_qty'] = $row['min_order_qty'];
    $itemArray[$x]['available'] = $row['available'];
    $itemArray[$x]['image'] = $row['image'];
    $itemArray[$x]['call'] = $row['call'];
    $itemArray[$x]['allow_in_cart'] = $row['allow_in_cart'];
    $itemArray[$x]['allow_backorder'] = $row['allow_backorder'];

    $desc = htmlspecialchars_decode($itemArray[$x]['title']);
    $desc = str_replace('"',"'", $desc);
    $desc = str_replace('&',"and", $desc);
    $desc = str_replace('%',"%25", $desc);

    $track_inventory = (strtolower($itemArray[$x]['inventory']) == "i") ? true : false;
    $allow_backorder = ($itemArray[$x]['allow_backorder'] == 0)? true : false;

    $cost = $itemArray[$x]['price'];

    if ($itemArray[$x]['do_not_sell'] == 1) {
        $inactive = true;
        $cost = 0.00;
    } else if (!$allow_backorder && $itemArray[$x]['available'] <= 0) {
        $cost = 0.00;
        $inactive = true;
    } else if ($itemArray[$x]['call'] == 1 ) {
        $cost = 0.00;
            $inactive = true;
        } else if  ($itemArray[$x]['allow_in_cart'] == 0) {
            $inactive = true;
            // call is set to no
            // allow in cart set to no
            // shows price but dont allow in cart
        } else if ($track_inventory && $itemArray[$x]['available'] <= 0) {
            $inactive = false;
            //allow in cart but
            //display shipping delay message
            //do nothing in UC  
        } else {
            $inactive = false;
        };

    $xml_output .= "\t\t<item>\n";
    $xml_output .= "\t\t\t<merchant_item_id>".$itemArray[$x]['id']."</merchant_item_id>\n";
    $xml_output .= "\t\t\t<description>".$desc."</description>\n";
    $xml_output .= "\t\t\t<view_url>".htmlspecialchars($itemArray[$x]['link'])."</view_url>\n";
    $xml_output .= "\t\t\t<cost>".number_format($cost, 2, '.', '')."</cost>\n";
    $xml_output .= "\t\t\t<uom_weight>LB</uom_weight>\n";
    $xml_output .= "\t\t\t<weight>".number_format($itemArray[$x]['shipping_weight'], 2, '.', '')."</weight>\n";
    $xml_output .= "\t\t\t<inactive>".var_export($inactive, true)."</inactive>\n";  
    $xml_output .= "\t\t\t<minimum_quantity>".$itemArray[$x]['min_order_qty']."</minimum_quantity>\n";  
    $xml_output .= "\t\t\t<inventory_quantity>".$itemArray[$x]['available']."</inventory_quantity>\n";  
    $xml_output .= "\t\t\t<track_inventory>false</track_inventory>\n";      
    $xml_output .= "\t\t\t<manufacturer_name>".htmlspecialchars($itemArray[$x]['brand'])."</manufacturer_name>\n";
    $xml_output .= "\t\t\t<manufacturer_sku></manufacturer_sku>\n";
    $xml_output .= "\t\t\t<uom_distance>IN</uom_distance>\n";

    // if its a solar panel or if it weighs over 70lbs remove dimensions
    //add per dustins request - "should fix panel shipping errors"
    if (strtolower($itemArray[$x]['type']) == "solar panel" || $itemArray[$x]['shipping_weight'] >= 70) {
        $xml_output .= "\t\t\t<length>0</length>\n";
        $xml_output .= "\t\t\t<width>0</width>\n";
        $xml_output .= "\t\t\t<height>0</height>\n";    
    } else {
        $xml_output .= "\t\t\t<length>".number_format($itemArray[$x]['shipping_length'], 2, '.', '')."</length>\n";
        $xml_output .= "\t\t\t<width>".number_format($itemArray[$x]['shipping_width'], 2, '.', '')."</width>\n";
        $xml_output .= "\t\t\t<height>".number_format($itemArray[$x]['shipping_height'], 2, '.', '')."</height>\n";
    }
    $xml_output .= "\t\t\t<froogle>\n";
    $xml_output .= "\t\t\t\t<image_url>".$itemArray[$x]['image']."</image_url>\n";
    $xml_output .= "\t\t\t</froogle>\n";
    $xml_output .= "\t\t\t<shipping>\n";
    if ($itemArray[$x]['ship_group'] == strtolower('fixed') && $itemArray[$x]['fixed_ship'] == 0) {
        $xml_output .= "\t\t\t\t<free_shipping>true</free_shipping>\n";
    } elseif ($itemArray[$x]['ship_group'] == strtolower('freight')) {
        $xml_output .= "\t\t\t\t<methods>\n";
        $xml_output .= "\t\t\t\t\t<method>\n";
        $xml_output .= "\t\t\t\t\t\t<name>Con-way: LTL</name>\n";
        $xml_output .= "\t\t\t\t\t\t<validity>valid only for</validity>\n";
        $xml_output .= "\t\t\t\t\t</method>\n";
        $xml_output .= "\t\t\t\t</methods>\n";
    } else {
        $xml_output .= "\t\t\t\t<free_shipping>false</free_shipping>\n";
    }
    $xml_output .= "\t\t\t</shipping>\n";               
    $xml_output .= "\t\t</item>\n";
}

$xml_output .= "\t</items>\n";

//open XML and write 20 items into it.
$fh = fopen('./bulk-item-pusher.xml','w') or die($php_errormsg);
fwrite($fh, $xml_output) or die($php_errormsg);
fclose($fh);

$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, "theapiurl");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "merchantId=".$merchId."&login=".$login."&password=".$password."&function=".$function."&Items=".$xml);
$content=curl_exec($ch);
if(curl_errno($ch)) {
    $err = curl_error($ch);
    $ef = fopen('./last-item.xml','w') or die($php_errormsg);
    fwrite($ef, $err) or die($php_errormsg);
    break;
}
curl_close($ch);

echo htmlspecialchars($content ." " . $itemArray[$x]['id'])."<br />";
}
echo "finished";


?>
Improve this question
4
  • 1
    Try use it in command line? Commented Oct 24, 2014 at 19:38
  • Usually a 500 error occurs in this kind of API because bad data was sent in the request. Commented Oct 24, 2014 at 19:38
  • To clarify - is it the PHP script exiting with a 500 error, or the curl request receiving a 500 back? Commented Oct 24, 2014 at 19:40
  • I'm not sure how to check where the 500 is coming from. I've tried this $content=curl_exec($ch); if(curl_errno($ch)) { $err = curl_error($ch); $ef = fopen('./last-item.xml','w') or die($php_errormsg); fwrite($ef, $err) or die($php_errormsg); break; }. But nothing is in the file after the 500 Commented Oct 24, 2014 at 19:58

2 Answers 2

Reset to default
1

Looks like you have a bug (maybe just a typo in your question?)

curl_setopt($ch, CURLOPT_POSTFIELDS, "merchantId=".$merchId."&login=".$login."&password=".$password."&function=".$function."&Items=".$xml);

$xml should be $xml_output there no? Also, unless you're using it somewhere else later on, why write to ./bulk-item-pusher.xml?

Add to your error handling a bit to check for error responses from the remote API

// Handle cURL error
if(curl_errno($ch)) {
    $err = curl_error($ch);
    $ef = fopen('./last-item.xml','w') or die($php_errormsg);
    fwrite($ef, $err) or die($php_errormsg);
    break;
}
// Handle error response from API
elseif(curl_getinfo($ch, CURLINFO_HTTP_CODE) === '500') {
    // Take note of which 20 records were sent to cause this error.
    // They need to be analyzed more deeply to determine which one caused
    // the remote service to crap out
}

Since you say the timeout doesn't appear to be the problem, I'll second @Jay Blanchard's comment above; it's probably the data you're sending on some of the requests.

At that point I would take each of the 20 items that was sent and send them individually to determine if one of the items you're sending is causing the remote side to crash.

From there determine what the problem is with that particular data item. I'd also follow up with some communication with the folks maintaining that API once you figure it out. 500's usually mean "our server couldn't cope with the request". It would be nice if they could revise it to handle such a case and send you back a 200 response with a component of the payload indicating any items that couldn't be processed, indexed by some type of correlation identifier.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

In the OP I said I think the error is coming a timeout. Also it wasnt a typo. I was writing each set of XML to a file then used // $xml = file_get_contents($xmlUrl); I've since changed it to not write to another file and am sending $xml_output directly in the http POST edit: Thanks btw I am really appreciating another set of eyes on this.
Seems like a comment in your code is contrary to the timeout theory however //tried sleeping between calls but this didnt help. Yeah, I would def just use memory for XML you generate to pass into cURL unless it's big chunk of data. My pleasure to look things over, the file_get_contents call isn't in your sample here, but I figured it prob wasn't a typo, just checking. What do you think about trying to send the XML payloads that failed individually like I suggested, or adding the 500 check; surely those were helpful suggestions? There's also the CURLOPT_CONNECTTIMEOUT option in cURL.
The really odd thing is that sometimes it will loop through 30 times and work perfectly and sometimes it will throw a 500 err after 5 loops. Where I've encountered an error I have sent the 20 surrounding items singly and received back the success code 0 for each of them (meaning api accepted them and updated the records). I included the 500 check in my error checking but I am still just receiving a standard 500 internal server error screen. Those were helpful suggestions. After further examining I have to believe it is something going wrong on my server and not a curl error.
I am looking into this section of the code as my only guess at this point allow_backorder FROM $table WHERE id > 10001 AND price > 0 AND base_match = 'm' ORDER BY id"; $resultID = mysql_query($query, $linkID) or die(mysql_error()); $i = 0; while( $i < 50 ){ $i++; $x = 0; $xml_output = "\t<items>\n"; while( $x < 20){ $row = mysql_fetch_assoc($resultID); $x++; $itemArray[$x]['id'] = $row['id']; if ($itemArray[$x]['id'] == "") { break; }; $itemArray[$x]['link'] = $row['link'];
I'm not sure how to get more information on where the error is coming from. I have this at the top of the script ini_set('display_errors',1); ini_set('display_startup_errors',1); error_reporting(-1); Error_log still comes back empty. This script is running on a subdomain also
|
1

The script completed when I ran it with the CLI. I am just curious as to why this is? My guess is it has to do something with memory usage? Is that correct? If I set up a cron to do this, does cron basically use the CLI also?

Improve this answer

1 Comment

Most likely some sort of discrepancy between your CLI configuration and Apache configuration for PHP. A CRON can use the CLI or webserver when invoking php depending on how you do it, but if you have it say php sync-to-api.php or w/e it will run PHP on the CLI.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.