my gift to the php multi-curl data scrapers

Hi ,
my name Joe (a.k.a @rxhector on twitter/steemit )

and this is the result of 15 years data-scraping using php w/multi-curl.

I have just recently solved one of the biggest problems I've had in scraping using php w/multi-curl,
the problem where you have to post to first page and use that data to get to yet another page of results (without having the never-ending unweildy cascading if/then/else url check bullshit)

it took 15 years because I started as a carpenter by trade (15 years) and just
kind of accidentally discovered php/mysql,
but we will save that story for another time.

I have a working example to check out so you wont be left flying blind like i was while i learned this shit.

https://github.com/rxhector/ultimate-multicurl

And I have tried to comment the code (still needs a ton of better / prettier format)

The first caveat from the code is this little xml beauty
load any web page into xml without it breaking the shit out of php simplexml_import_dom

if (!function_exists('load_simplexml_page')) {
    /*

        this will 'force' xml to load a web page (pure html)
        sometimes simplexml_import_dom breaks when trying to import html with bad mark up (i.e - old crappy coding / scripts)
        DOMDocument will auto-magically fix shitty html

        then we can simplexml-ize it !!!

        NOTE : 
            when using php file_get_contents or dom document or simplexml
            those functions use the php.ini user_agent

            most web sites will not return a web request to empty user_agent
            or user_agent "PHP"
            i ALWAYS set php.ini user_agent to a valid browser string

            php.ini
            user_agent="Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"

    */
    //load from url(default) or string
    function load_simplexml_page($page , $type='url'){
        /*
            static DOMDocument::loadHTMLFile() errors on malformed documents
            $dom = new DOMDocument('1.0', 'UTF-8'); //force doctype/utf - get rid of warnings
            prepend @$dom = tell php to ignore any warnings/errors
        */
        $dom = new DOMDocument('1.0', 'UTF-8');
        $type === 'url'
        ? @$dom->loadHTMLFile(trim($page))
        : @$dom->loadHTML($page);
        return simplexml_import_dom($dom);
    }//end function
}//end function check

here's a quick and dirty script that lets you test your php.ini user_agent from curl request

<?php   
    // save this on your  web server so you can hit it as web page ex. /www/test_user_agent.php
    // then run from cmd line ex. php /www/test_user_agent.php

    //require( somefile with load_simplexml_page )
    
    if(isset($_SERVER['REQUEST_METHOD'])){
        //this is a webpage hit 
        $return = print_r($_SERVER , true) ;
        echo $return;   //return something to browser/curl request
    }
    else
    {
        //no server request - this is running as cmd line script
        
        //returns default php user_agent "PHP" (if set)
        //$text = file_get_contents("http://localhost/test_user_agent.php");
        //echo $text;

        //returns default php user_agent "PHP" (if set)
        $xml = load_simplexml_page("http://localhost/test_user_agent.php");
        echo $xml->asXML();

    }  

i tried regex and substr and other normal php string processing functions
but i learned early on that this can become unweildy and complex
thats when i discovered simplexml - what an awesome tool for html !!!
it is super easy to use xpath expressions to get to any element on a web page to get data !!!

my first few scraping jobs i was a NOOB and was using good old file_get_contents and getting one slow page at a time,
but that has it limitations...
when you get a little more advanced you have to learn to make a query string and post data..(pagination is a bitch)

that's when i discovered curl - wow it took me about a week to learn how to set cookies so i could log in to sites and get backend data.

so my load_simplexml_page and curl tools came in handy - but man was it slow doing 'synchronous' page loading - one slow page at a time.
when you get a client that wants 10,000 pages at a time (instead of 100) - you better figure out how to do it AS FAST AS POSSIBLE !!!

then i stumbled into multi-curl - HOLY SHIT , talk about a learning curve..
i know there are a few 'wizards' out there that probably picked it up right away
but you gotta remember - I was a 15 year carpenter/guitar player/stoner - it took me a bit
to grab the 'asynchronous' concept - let's load 1000 pages all at once - and process them.

so now we can move on to the main multi-curl object itself
i wont post the code in its entirety here - you can find that posted at PUT GITHUB PROJECT LINK HERE

some of my favorite code tricks i've discovered along the way

    //notice the &(reference) here
    if ($x = &$this->result_callback($this->rs[$i]['options'])) {
    
    //$x is now a string and can be called as a function
        
        $x($this->rs[$i] , $i); //old school would be something like mixed call_user_func_array ( callable $callback , array $param_arr )
        //call_user_func_array($x , [$this->rs[$i] , $i]);
        

another pretty cool use for references

    public function start_callback(&$options , $set = false){ 
    //$options can now be changed from with the callback function (trust me - it comes in handy for passing variables in multi-curl)
    
        $options['result_callback'] = $set;     //now $options is changed in the main(calling) flow 
        

the next really cool caveat you get is a proxy scraper - you know - get a list of 300 proxies
then do a quick check on the target domain to make sure you have a good proxy ;)

so - the whole point for me was to get a better understanding of how multi-curl works.
if you want to know how something works - you gotta break it and try to rebuild it.

looking forward
this code really needs some more clean up and better comments/formatting.

i would really like to add some of the functionality from zebra-curl (i didnt need all the bells right away - so i just built this for quick get/post json requests)

I hope you guys like it

https://github.com/rxhector/ultimate-multicurl

this is running in production for mid-market cap company - we are scraping about 30k records @ 1000/hr (25pages/second) - not bad !!!

tipping is allowed
the old slow way - paypal rxhector2k5@yahoo.com
the super fast ~3 second way twitter @xrptipbot https://twitter.com/@rxhector
twitter trx bot @goseedit