Difference between command line cURL and PHP cURL

Question

I have a cURL command like this:

curl 'https://www.example.com' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' \
  -H 'accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' \
  -H 'authority: www.example.com'

Executing this in a command line like in Terminal app on my Mac, results to the expected output.

(In case you test it yourself: If this output contains the word Sicherheitsüberprüfung it's geo blocked and you have to use a German IP to test it.)

I transferred the exact command to PHP cURL like this:

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'https://www.example.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

$headers = array();
$headers[] = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36';
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3';
$headers[] = 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7';
$headers[] = 'Authority: www.example.com';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
curl_close($ch);
echo $result;
?>

When I run this code I'm getting a message that my request was recognized as automated request/robot: It says Sicherheitsüberprüfung, means security check.

Of course, I'm using the same IP for both, command line and PHP cURL request.

Why that? Isn't command line cURL the same as PHP cURL?

Or is there anything wrong with my PHP script?

UPDATE

I fortuitously found out the following: I'm using Coda as code editor on my Mac. This has a build-in PHP rendering engine. Using this with my PHP script, the result is as expected. It's the same result I'm getting in the command line.

UPDATE 2

I made what Jannes Botis suggested in his answer. I then ran the PHP script in my Coda code editor app (what output the expected) and with MAMP as localhost (what is always recognized as automated request).

I figured out that the the code executed with MAMP was using HTTP/2 while the code executed in Coda is using HTTP/1.1. To solve this, I added the following to the script:

curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

Now, both output exact the same string:

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Authority: www.example.com

But, it's still the same: The one is working, the other is recognized as automated request.

PHP script looks fine. Try to capture tcp packets via tcpdump/Wireshark and examine the differences. — Tns
– Tns, Commented Nov 4, 2019 at 12:59
Have you tried setting the headers using CURLOPT constants, e.g. $headers[CURLOPT_USERAGENT] = 'Mozilla/5.0...', etc, and then use curl_setopt_array() instead ? — code-kobold
– code-kobold, Commented Nov 4, 2019 at 15:13
@Tns I don't really understand what you mean and how to do that with my PHP script and over command line. Do you have a link or something for me? — David
– David, Commented Nov 5, 2019 at 11:50
Possible duplicate of php curl: how can i emulate a get request exactly like a web browser? — Nico Haase
– Nico Haase, Commented Nov 5, 2019 at 12:49
@SalmanA No, I'm doing it on my Mac. But I tested it on a "real" server. It's the same there. — David
– David, Commented Nov 12, 2019 at 14:27

David · Accepted Answer · 2019-11-26 08:38:31Z

4

+100

Try to debug the request in both cases:

a) Terminal: use curl verbose mode: curl -v and check the http request sent, especially check the header list

b) php curl: print the http request using CURLINFO_HEADER_OUT:

curl_setopt($ch, CURLINFO_HEADER_OUT, true);

curl_exec($ch);

$info = curl_getinfo($ch);
print_r($info['request_header']);

Testing the different headers, what made it work was adding "Pragma: no-cache" header to the request:

$headers[] = 'Pragma: no-cache';

On the other hand, in terminal curl, I had to uppercase the request headers, e.g. User-Agent etc.

Try to create a tcp connection with fsockopen:

$fp = fsockopen("ssl://"."www.example.com", 443, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)<br />\n";
} else {
    $out = "GET / HTTP/1.1\r\n";
    $out .= "Host: www.example.com\r\n";
    $headers = array();
    $headers[] = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36';
    $headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3';
    $headers[] = 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7';
    $headers[] = 'Authority: www.example.com';
    $out .= $headers;
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 1024);
    }
    fclose($fp);

and test if this works. Maybe the issue is either that php curl adds some info to the http request or the problem is on the tcp connection level, some info added there.

References

edited Nov 26, 2019 at 8:38

David

3,4766 gold badges30 silver badges71 bronze badges

answered Nov 6, 2019 at 22:02

Jannes Botis

11.2k3 gold badges24 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

David Over a year ago

I added $headers[] = 'Pragma: no-cache'; but it's still the same. Can you please post the output somewhere, e.g. on codeshare.io?

Jannes Botis Over a year ago

Hm, I am not sure. Try this: stackoverflow.com/questions/27088070/… Also you could try to add "Connection: Keep-Alive" header also

David Over a year ago

Connection: Keep-Alive didn't help. BTW: I updated my question with some new details.

Jannes Botis Over a year ago

@David can you try the tcp request at the 2nd section of the answer?

Mahek Manvar · Accepted Answer · 2019-11-13 12:40:31Z

Command line curl :

It is a tool to transfer data to or from a server, using any of the supported protocols (HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE). curl is powered by Libcurl. This tool is preferred for automation, since it is designed to work without user interaction. curl can transfer multiple file at once. For more details for Command line curl

Syntax:

curl [options] [URL...]

Example:

curl http://site.{one, two, three}.com

PHP cURL

$ch = curl_init('http://example.com/wp-login.php');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 3);

if($this->getRequestType() == 'POST')
{
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, 
        array(
            'user[name]'    => 'Generic+Username',
            'user[email]'   => '[email protected]'
        );
    );
}

$response   = curl_exec($ch);

Styx · Accepted Answer · 2019-11-13 11:33:26Z

0

The issue is with ciphers selected by PHP's cURL by default.

Running curl command with -Ivs options allows us to see what ciphers it uses:

* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

Setting them in PHP allows it to bypass this mysterious check:

curl_setopt($ch,
  CURLOPT_SSL_CIPHER_LIST,
  'ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH'
);

Also, it seems that Host header and using HTTPv2 should be added:

$headers[] = 'Host: www.11880.com';
// ...
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);

edited Nov 13, 2019 at 11:33

answered Nov 8, 2019 at 15:30

Styx

10.1k8 gold badges48 silver badges56 bronze badges

8 Comments

miken32 Over a year ago

"The issue is with ciphers selected by PHP's cURL by default." That sounds like a very definitive statement, with nothing in the answer to support it. If you determined by testing, you should say so. If it's an educated guess (nothing wrong with that of course) you should also say so.

David Over a year ago

@Styx I executed my curl command in command line with -Ivs and got the following: * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH I added curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, 'ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH'); to my PHP script but it's still not working.

Styx Over a year ago

@miken32 Forgot to point that, sorry. It was determined by testing using the host provided by OP and VPN in Germany.

Styx Over a year ago

@David Once I added this to my script it started working. Server started respond with 200 rather than 403 and no 'Sicherheitsüberprüfung' anymore.

David Over a year ago

@Styx Did you use the exact code I posted in my question above and only added curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, 'ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH');? Can you please post the output somewhere, e.g. on codeshare.io?

|

Collectives™ on Stack Overflow

Difference between command line cURL and PHP cURL

3 Answers 3

4 Comments

Comments

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related