Web Scraper API: Python, PHP & ASP.NET

Through GrabzIt's Web Scraper API, you can provide your application with scraped data as a web service, enabling you to integrate extracted information directly into your database or software logic. This allows you to automatically control when scrapes start and stop, as well as request results to be re-sent programmatically.

We provide native client libraries for Python, PHP, and ASP.NET to make integration effortless. However as our code is open source and available on GitHub there is no reason you can not make one for a programming language not listed here or you can ask us to create a library for you. If you do why not share it with the world?

How It Works: The Callback Handler

The integration of data into your application is achieved through a callback handler you specify on the Export tab when you create your scrape. This is a script or controller method on a publicly accessible URL on your server. When GrabzIt finishes scraping, it posts the data files to this URL sequentially.

While you can export data in various formats (CSV, Excel), we recommend using JSON or XML for the Callback Handler, as these structured formats are easily parsed by object-oriented languages.

Step 1: Start the Scrape

First, you need to tell GrabzIt to start the scrape. This is done by sending a request using the Scrape method from the client library. You must pass the status of the scrape (e.g., "Start") and the `id` of the scrape you wish to control.

• Python
• PHP
• ASP.NET

from GrabzIt import GrabzItScrapeClient

client = GrabzItScrapeClient.GrabzItScrapeClient("Sign in to view your Application Key", "Sign in to view your Application Secret")
//Get all of our scrapes
myScrapes = client.GetScrapes()
if (len(myScrapes) == 0)
{
    raise Exception('You have not created any scrapes yet! Create one here: https://grabz.it/scraper/scrape/')
}
//Start the first scrape
client.SetScrapeStatus(myScrapes[0].ID, "Start")
if (len(myScrapes[0].Results) > 0)
{
    //re-send first scrape result if it exists
    client.SendResult(myScrapes[0].ID, myScrapes[0].Results[0].ID);
}
            

include("GrabzItScrapeClient.class.php");

$client = new \GrabzIt\Scraper\GrabzItScrapeClient("Sign in to view your Application Key", "Sign in to view your Application Secret");
//Get all of our scrapes
$myScrapes = $client->GetScrapes();
if (empty($myScrapes))
{
    throw new Exception("You haven't created any scrapes yet! Create one here: https://grabz.it/scraper/scrape/");
}
//Start the first scrape
$client->SetScrapeStatus($myScrapes[0]->ID, "Start");
if (count($myScrapes[0]->Results) > 0)
{
    //re-send first scrape result if it exists
    $client->SendResult($myScrapes[0]->ID, $myScrapes[0]->Results[0]->ID);
}
            

GrabzItScrapeClient client = new GrabzItScrapeClient("Sign in to view your Application Key", "Sign in to view your Application Secret");
//Get all of our scrapes
GrabzItScrape[] myScrapes = client.GetScrapes();
if (myScrapes.Length == 0)
{
    throw new Exception("You haven't created any scrapes yet! Create one here: https://grabz.it/scraper/scrape/");
}
//Start the first scrape
client.SetScrapeStatus(myScrapes[0].ID, ScrapeStatus.Start);
if (myScrapes[0].Results.Length > 0)
{
    //re-send first scrape result if it exists
    client.SendResult(myScrapes[0].ID, myScrapes[0].Results[0].ID);
}
            

Step 2: Handle the Result

Once the scrape is complete, GrabzIt will send the data to your callback URL. You use the ProcessScrape method within your handler to automatically capture the file and send a confirmation back to GrabzIt.

Process Scraped Data

The easiest way to process scraped data is to access the data as a JSON or XML object, as this enables the data to be easily manipulated and queried. The JSON will be structured in the following general format with the dataset name as the object attribute, itself containing an array of objects with each column name as another attribute.

{
  "Items": [
    {
      "Column_One": "https://grabz.it/",
      "Column_Two": "Found"
    },
    {
      "Column_One": "http://dfadsdsa.com/",
      "Column_Two": "Missing"
    }]
}

First of all it must be remembered that the handler will be sent all scraped data, which may include data that can not be converted to JSON or XML objects. Therefore the type of data you are receiving must be checked before being processed.

• Python
• PHP
• ASP.NET

from GrabzIt import GrabzItScrapeClient

# The library automatically reads the posted file
client = GrabzItScrapeClient.GrabzItScrapeClient("Sign in to view your Application Key", "Sign in to view your Application Secret")
scrapeResult = ScrapeResult.ScrapeResult()

if scrapeResult.getExtension() == 'json':
    json = scrapeResult.toJSON()
    for json["Dataset_Name"] in obj:
        if obj["Column_Two"] == "Found":
            #do something
        else:
            #do something else
else:
    #probably a binary file etc save it
    scrapeResult.save("results/"+scrapeResult.getFilename())
            

include("GrabzItScrapeClient.class.php");

// The library automatically reads the posted file
$client = new \GrabzIt\Scraper\GrabzItScrapeClient("Sign in to view your Application Key", "Sign in to view your Application Secret");
$scrapeResult = new \GrabzIt\Scraper\ScrapeResult();

if ($scrapeResult->getExtension() == 'json')
{
    $json = $scrapeResult->toJSON();
    foreach ($json->Dataset_Name as $obj)
    {
        if ($obj->Column_Two == "Found")
        {
            //do something
        }
        else
        {
            //do something else
        }
    }
}
else
{
    //probably a binary file etc save it
    $scrapeResult->save("results/".$scrapeResult->getFilename());
}
            

using GrabzIt.Scraper;

// The library automatically reads the posted file
GrabzItScrapeClient client = new GrabzItScrapeClient("APPLICATION KEY", "APPLICATION SECRET");
ScrapeResult scrapeResult = new ScrapeResult(context.Request);

if (scrapeResult.Extension == "json")
{
    DataSet dataSet = scrapeResult.FromJSON<DataSet>();
    foreach (Item item in dataSet.Items)
    {
        if (item.Column_Two == "Found")
        {
            //do something
        }
        else
        {
            //do something else
        }
    }
}
else
{
    //probably a binary file etc save it
    scrapeResult.save(context.Server.MapPath("~/results/" + scrapeResult.Filename));
}
            

The above example shows how to loop through all the results of the dataset Dataset_Name and do specific actions depending on the value of the Column_Two attribute. Also if the file received by the handler is not a JSON file then it is just saved to results directory. While the ScrapeResult class does attempt to ensure that all posted files originate from GrabzIt's servers the extension of the files should also be checked before they are saved.

Class Reference

Below is a detailed reference of the methods and properties available in the Scrape Client for controlling and processing scrapes.

Debugging & Testing

If you are having issues with your callback handler, you can enable Debug Mode in the Scrape Options tab. This will output the response returned by your callback handler into the logs, allowing you to see any errors your script might be generating.