Saturday 14 December 2013

CURL Example

CURL
CURL is a library and command-line tool for transferring data using various protocols like HTTP, FTP, SCP, PUT, POST, POP3 & SMTP etc. It also used for scrapping the data from websites. We can scrap get the text as well as images with use of CURL.


It have two products libcurl and cURL.

cURL: Its command line tool for transferring information.
Libcurl: It is a free client-side URL transfer library.


CURL COMMAND-LINE EXAMPLE

Curl Example with "GET" Method (By Default GET)
curl http://www.web-technology-experts-notes.in/p/sitemap.html

Curl Example with "POST" Method
curl --request POST http://www.web-technology-experts-notes.in/p/sitemap.html

Curl Example with "PUT" Method
curl --request PUT http://www.web-technology-experts-notes.in/p/sitemap.html

Curl Example with "DELETE" Method
curl --request DELETE http://www.web-technology-experts-notes.in/p/sitemap.html

Curl Example with "POST" Method and pass data (Data: username, password & logintype)
curl --request POST http://www.web-technology-experts-notes.in/p/sitemap.html
--data 'username=user&password=****&logintype=normal' 

Curl Example with "POST" Method and pass data(data in text file)
curl --request POST http://www.web-technology-experts-notes.in/p/sitemap.html --data @datafile.txt

Curl Example with "POST" Method, pass data and pass header
curl --request POST http://www.web-technology-experts-notes.in/p/sitemap.html --data @datafile.txt --header 'sessionid:9874563211235884' 

Curl Example with "POST" Method, pass data and pass header (Get Header in response) --include
curl --request POST http://www.web-technology-experts-notes.in/p/sitemap.html --data @datafile.txt --header 'sessionid:9874563211235884' --include

CURL LIBCURL EXAMPLE

Mostly to get data from an REST API, we use following method in php
echo file_get_contents('http://www.web-technology-experts-notes.in/p/sitemap.html');
But if, "allow_url_fopen" is disable in php.ini file due to security reason then above will not work. You need to use below code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.google.co.in");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$output = curl_exec($ch);
curl_close($ch);
echo $output;die;


CURL Example with Post Method with parameter
$url = "http://localhost/test/index2";
$post_data = array(
            "foo" => "bar",
            "foo1" => "testdata"
);
try {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
    $output = curl_exec($ch);
    curl_close($ch);
} catch (Exception $e) {
    echo $e->getMessage();die;
}

File Upload with CURL in PHP
 $file_name_with_full_path = realpath('uploadFile.zip');        
        $url = "http://localhost/test/index2";
        $post_data = array(
            "foo" => "bar",
            "upload" => "@".$file_name_with_full_path
        );
        try {
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            curl_setopt($ch, CURLOPT_POST, 1);
            curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
            $output = curl_exec($ch);
            curl_close($ch);
        } catch (Exception $e) {
            echo $e->getMessage();
            die;
        }
        
Transfer the File through FTP
curl_setopt_array($ch, array(
    CURLOPT_URL => 'ftp://ftp.example.com/test.txt',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_USERPWD => 'username:password'
));
 
$output = curl_exec($ch);

Set Header and Json data in Curl
$url='http://www.web-technology-experts-notes.in/';
$jsonData='{"name":"arun","email":"arun@gmail.com"}';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $jsonData);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
  'X-API-Token: ' . X_API_TOKEN,
  'Content-Type: application/json',
  "HTTP_X_FORWARDED_FOR: xxx.xxx.x.xx"
));

curl_setopt($ch, CURLOPT_INTERFACE, "xxx.xxx.x.xx");
echo curl_exec($ch);
curl_close($ch);
Following are some Options which can be set be set in "curl_setopt" function


CURLOPT_REFERER - Set the Http Refer

CURLOPT_USERAGENT - Set the UserAgent

CURLOPT_HEADER - Include the header in response or not

CURLOPT_RETURNTRANSFER - Print the output or not

CURLOPT_URL - Set the URL (Uniform Resource Locator)

CURLOPT_POST - set 0 or 1 for post method

CURLOPT_POSTFIELDS - set the data which you want to post

CURLOPT_FOLLOWLOCATION - if set true, cURL will follow redirects

CURLOPT_CONNECTTIMEOUT -Total seconds to spend attempting to connect

CURLOPT_TIMEOUT - Total Seconds to allow cURL to execute






Zend Curl Example

$config = array(
    'adapter' => 'Zend_Http_Client_Adapter_Curl',
    'ssltransport' => 'tls',
    'strict' => false,
    'persistent' => true,
);
$url = 'http://www.example.com';
$client = new Zend_Http_Client($url, $config);
$postArray = array('name'=>'Georage','age'=>33);
$client->setParameterPost($postArray);
/** set Headers **/
$client->setHeaders(array(
    'Host: www.example.com',
    'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0',
    'Accept: text/javascript, application/javascript, */*',
    'Accept-Language: en-gb,en;q=0.5',
    'Accept-Encoding: gzip, deflate',
    'Content-Type: application/x-www-form-urlencoded; charset=UTF-8',
    'X-Requested-With: XMLHttpRequest',
    'Referer: http://example.com'
));

/** set cookie **/
$client->setCookie('__utma', '174057141.1507422797.1392357517.1392698281.1392702703.3');
$client->setCookie('__utmz', '174057141.1392357517.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)');
$client->setCookie('__atuvc', '58');
$client->setCookie('PHPSESSID', '0a12ket5d8d7otlo6uh66p658a5');
$response = $client->request('POST');

$data= $response->getBody();
echo $data; 

Zend CURL Ignore SSL_VERIFYPEER
$url='www.example.com';
$config = array(
    'adapter' => 'Zend_Http_Client_Adapter_Curl',
    'curloptions' => array(
        CURLOPT_FOLLOWLOCATION => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE
    ),
);
 $client = new Zend_Http_Client($url, $config);
 $response = $client->request('GET');

Set CURL'S Timeout in Milli-seconds
 $ch = curl_init();
          //set the curl for crawl
          curl_setopt($ch, CURLOPT_URL, "http://www.web-technology-experts-notes.in/2014/03/captcha-code-example.html");
          //include the header in the output
          curl_setopt($ch, CURLOPT_HEADER, 0);
          //The number of seconds to wait while trying to connect. Use 0 to wait indefinitely.
          curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
          //The maximum number of milliseconds to allow cURL functions to execute.
          curl_setopt($ch, CURLOPT_TIMEOUT_MS, 4000); //timeout in seconds
          $output = curl_exec($ch);
          curl_close($ch); 

Set Zend CURL'S Timeout in Milli-seconds
$config = array(
            'adapter' => 'Zend_Http_Client_Adapter_Curl',
            'curloptions' => array(
                CURLOPT_FOLLOWLOCATION => FALSE,
                CURLOPT_HEADER => false,
                //The number of seconds to wait while trying to connect. Use 0 to wait indefinitely.
                CURLOPT_TIMEOUT_MS => 3000,
                //The maximum number of milliseconds to allow cURL functions to execute.
                CURLOPT_CONNECTTIMEOUT_MS => 3000
            )
        );
        $url = "http://www.web-technology-experts-notes.in/2014/03/captcha-code-example.html";
        $client = new Zend_Http_Client($url, $config);
        $response = $client->request('GET');
        $output = $response->getBody();



Question: How to Set the username/password in CURL?
curl -X POST -u "USERNAME":"PASSWORD" --header "Content-Type: audio/flac" --header "Transfer-Encoding: chunked"  "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize" 



Question: How to pass binary data(file) in CURL ?
curl -X POST -u "USERNAME":"PASSWORD" --header "Content-Type: audio/flac" --header "Transfer-Encoding: chunked" --data-binary @audio.flac "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize" 



Question: How to store linux command output in file?
curl -X POST -u "USERNAME":"PASSWORD" --header "Content-Type: audio/flac" --header "Transfer-Encoding: chunked" --data-binary @audio.flac "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize" > output1.txt