PHP: how to get content of many websites in the same time


The solution is CURL, this is powerful tool to do it.

Here’s the plain example, you can know how to get content of many websites in the same time:

<?php
 
$contents = array();

$u = 'http://demo.tutorialspots.com/curl/get1.php';
$u2 = 'http://demo.tutorialspots.com/curl/get2.php';
$u3 = 'http://demo.tutorialspots.com/curl/get3.php';
$u4 = 'http://demo.tutorialspots.com/curl/get4.php';

// array of URLs
$arURLs = array($u, $u2, $u3, $u4); 

 // init the curl Multi
$multi_handles = curl_multi_init();

// create an array for the individual curl handles
$arCurlHandles = array(); 

//add the handles for each url
foreach ($arURLs as $id => $url)
{ 
    $ch = curl_init(); // init curl, and then setup your options
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // returns the result - very important
    curl_setopt($ch, CURLOPT_HEADER, 0); // no headers in the output
    $arCurlHandles[$url] = $ch;
    curl_multi_add_handle($multi_handles, $ch);
}

$active = null;
//execute the handles
do {
  curl_multi_exec($multi_handles, $active);
  curl_multi_select($multi_handles);
} while ($active > 0);

// iterate through the handles and get your content
foreach ($arCurlHandles as $url => $ch)
{
    // get the content
    $contents[] = curl_multi_getcontent($ch); 
     
    // remove the handle (assuming you are done with it);
    curl_multi_remove_handle($multi_handles, $ch); 
    
    curl_close($ch);
}

// close the curl multi handler
curl_multi_close($multi_handles); 

var_dump($contents);

curl multi

Result:

array(4) {
  [0]=>
  string(10) "1478627736"
  [1]=>
  string(10) "1478627736"
  [2]=>
  string(10) "1478627736"
  [3]=>
  string(10) "1478627736"
}

PHP code of get1.php,…

<?php

echo time();

Online demo:
http://demo.tutorialspots.com/curl/curl_multi.php

Leave a Reply