[UniqueIdeaMan]

Php Folks,

My following 3 codes work.
The final two are the same except the latter one suppresses the WARNINGS.
Now, the difference between the first one and the second one is that the first one uses file_get_html() function while the second one uses cURL (aswell as the third one).

Anyway, trying to combine the two codes for some reasons which I will get into once this problem is solved.
I get error:
Fatal error: Uncaught Error: Call to a member function find() on string in C:\xampp\htdocs\cURL\crawler.php:24 Stack trace: #0 {main} thrown in C:\xampp\htdocs\cURL\crawler.php on line 24


WORKING CODE 1

FINDING HTML ELEMENTS BASED ON THEIR TAG NAMES

Suppose you wanted to find each and every image on a webpage or say, each and every hyperlink. 
We will be using “find” function to extract this information from the object. Here’s how to do it using Simple HTML DOM Parser :
*/

include('simple_html_dom.php');

$html = file_get_html('http://google.com');

//to fetch all hyperlinks from a webpage
$links = array();
foreach($html->find('a') as $a) {
 $links[] = $a->href;
}
print_r($links);
echo "<br />";
 
//to fetch all images from a webpage
$images = array();
foreach($html->find('img') as $img) {
 $images[] = $img->src;
}
print_r($images);
echo "<br />";
 
//to find h1 headers from a webpage
$headlines = array();
foreach($html->find('h1') as $header) {
 $headlines[] = $header->plaintext;
}
print_r($headlines);
echo "<br />";

?> 

WORKING CODE 2

<?php

/*
2a. Scrape Urls & Anchors And Echo Them By NOT Suppressing Warnings.

Using PHP's DOM functions to
fetch hyperlinks and their anchor text
*/

$url = 'https://google.com';
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($curl);

// DO NOT Hide HTML warnings
libxml_use_internal_errors(true);
$dom = new DOMDocument;
    // echo Links and their anchor text
    echo '<pre>';
    echo "Link\tAnchor\n";
    foreach($dom->getElementsByTagName('a') as $link) {
        $href = $link->getAttribute('href');
        $anchor = $link->nodeValue;
        echo $href,"\t",$anchor,"\n";
    }
    echo '</pre>';

?>

WORKING CODE 3

<?php

/*
2b. Scrape Urls & Anchors And Echo Them By Suppressing Warnings.

Using PHP's DOM functions to
fetch hyperlinks and their anchor text
*/

$url = 'https://google.com';
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($curl);

// Hide HTML warnings
libxml_use_internal_errors(true);
$dom = new DOMDocument;
if($dom->loadHTML($data, LIBXML_NOWARNING)){
    // echo Links and their anchor text
    echo '<pre>';
    echo "Link\tAnchor\n";
    foreach($dom->getElementsByTagName('a') as $link) {
        $href = $link->getAttribute('href');
        $anchor = $link->nodeValue;
        echo $href,"\t",$anchor,"\n";
    }
    echo '</pre>';
}else{
    echo "Failed to load html.";

}
?>

Here is the code I am getting the error:

<?php 

/* FROM dom_crawler_NOTES.php file.

2.
FINDING HTML ELEMENTS BASED ON THEIR TAG NAMES

Suppose you wanted to find each and every link on a webpage. 
We will be using “find” function to extract this information from the object. Here’s how to do it using Simple HTML DOM Parser :
*/

include('simple_html_dom.php');

$url = 'https://www.yahoo.com';
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
$html = curl_exec($curl);

//to fetch all hyperlinks from a webpage
$links = array();
foreach($html->find('a') as $a) {
 $links[] = $a->href;
}
print_r($links);
echo "<br />";

?>

Fatal error: Uncaught Error: Call to a member function find() on string in C:\xampp\htdocs\cURL\crawler.php:24 Stack trace: #0 {main} thrown in C:\xampp\htdocs\cURL\crawler.php on line 24


Why error in this line ? ....

foreach($html->find('a') as $a) { 

Why script failing to acknowledge the "find" when it exists in the file:
include('simple_html_dom.php');

The simple_html_dom.php file can be downloaded from here:
https://sourceforge.net/projects/simplehtmldom/files/

Look at the first 3 scripts in this post. The first script uses this "simple_html_dom.php" and it had no problem using the "find".
Since on my latter script's code I referenced it to this "simple_html_dom.php" and it is residing in the same directory on my xamp then I should not be getting this silly error:
Fatal error: Uncaught Error: Call to a member function find() on string in C:\xampp\htdocs\cURL\crawler.php:24 Stack trace: #0 {main} thrown in C:\xampp\htdocs\cURL\crawler.php on line 24


Now should I ?

[MrFujin]

You have 3 working code, why not use one of those?

Second, just because you include a file, does not mean you use the class/functions from that.

What part of the OOP design is that hard to understand? If you want to use a function in a class, you need an class instance object first!

[UniqueIdeaMan]

Originally Posted by MrFujin

You have 3 working code, why not use one of those?

Second, just because you include a file, does not mean you use the class/functions from that.

What part of the OOP design is that hard to understand? If you want to use a function in a class, you need an class instance object first!

Thing is, I haven't even started on OOP or pdo. Still at procedural. Those first 3 codes you see, none of them are mine. I grabbed the first one from a tutorial and the last 2 were given to me by a nice gentleman from Stack Over Flow.
Nevertheless, I do understand their codes, even though they are OOP. And, just because I understand them does not mean I'd be able to write OOP.
Still, saying all this. Does not mean I should not make an attempt to cut & paste and see if I can make things work before I even set foot on the style.
I have asked at Stack Over Flow for the guy to convert it to procedural style. That way, I can continue building the script further in procedural style rather than have half the script in one style and the other half in another. Laughing stock. I'll say it before any of you do and won't give any of you guys the chance to say it.
I will get into pdo soon. Having to delay it.

[UniqueIdeaMan]

I just replaced:

//$html = file_get_html('http://nimishprabhu.com'); 

with:

$url = 'https://www.yahoo.com'; 
$curl = curl_init($url); 
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0); 
$html = curl_exec($curl); 

That is all!
That should not result in that error!

[UniqueIdeaMan]

I am told:
"file_get_html is a special function from simple_html_dom library. If you open source code for simple_html_dom you will see that file_get_html() does a lot of things that your curl replacement does not. That's why you get your error."

Anyway, folks, I really don't wanna be using this limited capacity file_get_html() and so let's replace it with cURL. I tried my best in giving a shot at cURL here. What-about you ? Care to show how to fix this thingY ?

[UniqueIdeaMan]

UPDATE:

I have been given this sample code just now ...

Possible solution with str_get_html:

$url = 'https://www.yahoo.com'; 
$curl = curl_init($url); 
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0); 
$response_string = curl_exec($curl); 

$html = str_get_html($response_string);

//to fetch all hyperlinks from a webpage 
$links = array(); 
foreach($html->find('a') as $a) { 
    $links[] = $a->href; 
} 
print_r($links); 
echo "<br />"; 

Gonna experiment with it.
Just sharing it here for other future newbies!

[UniqueIdeaMan]

I did a search on the php manual for str_get_html to be sure what the function does. But, I am shown no results.
And so, I ask: Just what does it do ?

[UniqueIdeaMan]

Php Buddies,

Look at these 2 updates. They both succeed in fetching the php manual page but fail to fetch the yahoo homepage. Why is that ?
The 2nd script is like the 1st one except a small change. Look at the commented-out parts in script 2 to see the difference. The added code comes after the commented-out code part.

SCRIPT 1

<?php 

//HALF WORKING

include('simple_html_dom.php'); 

$url = 'http://php.net/manual-lookup.php?pattern=str_get_html&scope=quickref'; // WORKS ON URL
//$url = 'https://yahoo.com'; // FAILS ON URL

$curl = curl_init($url); 
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0); 
$response_string = curl_exec($curl); 

$html = str_get_html($response_string);

//to fetch all hyperlinks from a webpage 
$links = array(); 
foreach($html->find('a') as $a) { 
    $links[] = $a->href; 
} 
print_r($links); 
echo "<br />"; 


?>

SCRIPT 2

<?php 

//HALF WORKING

include('simple_html_dom.php'); 

$url = 'http://php.net/manual-lookup.php?pattern=str_get_html&scope=quickref'; // WORKS ON URL
//$url = 'https://yahoo.com'; // FAILS ON URL
$curl = curl_init($url); 
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0); 
$response_string = curl_exec($curl); 

$html = str_get_html($response_string);

/*
//to fetch all hyperlinks from a webpage 
$links = array(); 
foreach($html->find('a') as $a) { 
    $links[] = $a->href; 
} 
print_r($links); 
echo "<br />"; 
*/

// Hide HTML warnings
libxml_use_internal_errors(true);
$dom = new DOMDocument;
if($dom->loadHTML($html, LIBXML_NOWARNING)){
    // echo Links and their anchor text
    echo '<pre>';
    echo "Link\tAnchor\n";
    foreach($dom->getElementsByTagName('a') as $link) {
        $href = $link->getAttribute('href');
        $anchor = $link->nodeValue;
        echo $href,"\t",$anchor,"\n";
    }
    echo '</pre>';
}else{
    echo "Failed to load html.";

}

?>

Don't forget my previous post!

Cheers!

[MrFujin]

Originally Posted by UniqueIdeaMan

I did a search on the php manual for str_get_html to be sure what the function does. But, I am shown no results.
And so, I ask: Just what does it do ?

If you want to know what it does, then look it up in the simple_html_dom.php file.

[UniqueIdeaMan]

Originally Posted by MrFujin

If you want to know what it does, then look it up in the simple_html_dom.php file.

Thanks man! I never thought of that. Just found this:

// get html dom from string
function str_get_html($str, $lowercase=true, $forceTagsClosed=true, $target_charset = DEFAULT_TARGET_CHARSET, $stripRN=true, $defaultBRText=DEFAULT_BR_TEXT, $defaultSpanText=DEFAULT_SPAN_TEXT)
{
    $dom = new simple_html_dom(null, $lowercase, $forceTagsClosed, $target_charset, $stripRN, $defaultBRText, $defaultSpanText);
    if (empty($str) || strlen($str) > MAX_FILE_SIZE)
    {
        $dom->clear();
        return false;
    }
    $dom->load($str, $lowercase, $stripRN);
    return $dom;
} 

But reading all that, I do not understand what it does and so might aswell ask the pros.
Guessing it gets the whole html into a single line. Yeah, I know. It sounds absurd.

[MrFujin]

If you want to have a the deeper understanding, then you have to learn about OOP - with the usage of classes and its functions and the relation between.

In basic, it create a class instance of simple_html_dom, load it with the html string and then return that instance.
That is what it does. Why it does that? Read my first line.

[Barand]

The bit that is relevant to your other problem is the middle portion.

If you pass it an empty string (ie your cUrl didn't return anything) then the function returns "false" instead of a valid simple_html_dom object. Hence the error message about using find on a boolean.

[UniqueIdeaMan]

Originally Posted by MrFujin

If you want to have a the deeper understanding, then you have to learn about OOP - with the usage of classes and its functions and the relation between.

In basic, it create a class instance of simple_html_dom, load it with the html string and then return that instance.
That is what it does. Why it does that? Read my first line.

A good morning to you my fine gentleman!
What are you doing early in the morning helping a guy like me ? Lol!
Yes, I will start on pdo (oop on php).
Anyway, did you noT say you don't know php ? Then how come you know all this about php ? You sure have some very high common sense! Lol!
Anyway, if you do know php then hang on with me for atleast 2 days and I reckon this crawler of mine would be complete!
Look-out for all my threads tonight and tomorrow.

Thank You!

[UniqueIdeaMan]

Problem was fixed.
Thanks to folks here.