Download PHP Screen Scraping Class

Updates

20009-07-30 Added setHeader() function

Again, I am using the cScrape class I wrote, which you can download.

Step 1
First go to twitter.com and look at the source code of the login to get the form field names and the form post location. You will see that the form posts to https://twitter.com/session and the username and password fields are session[username_or_email] and session[password] respectively.

Step 2
Now you are ready to login. So using the fetch function in the Scrape class you create an associative array to contain the form values you want to post. The other thing you will need to do is uncomment the lines for CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR. Cookies will be required to stay logged in and scrape around. The paths to the cookie files need to be writable by your app. Also you will need to uncomment the line about CURLOPT_FOLLOWLOCATION.

Step 1.5
Oops that didn't work. All I got back was 403 Forbidden: The server understood the request, but is refusing to fulfill it. Ahhh I see another variable called authenticity_token I bet Twitter was looking for that. So let's back up and first hit twitter.com to get the authenticity_token variable, and then make the login post request with that variable included in our array of parameters.

So that's basically it. Now you are logged in and can scrape around and request other pages as you normally would. Sorry it wasn't a longer post. I really do enjoy this kind of stuff so if anyone has a request, hit me up.

Errors?
1) Make sure that you are properly parsing the token variable
2) Make sure that you uncommented the lines about CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR, those options need to be enabled and be sure the path set is writable by your application
3) Make sure that the path to the cookie file is writable and that it is getting data written to it
4) If you get a message about being redirected you need to uncomment the line about CURLOPT_FOLLOWLOCATION, that option needs to be enabled true

So here we go. This time I made a quick PHP class with some basic functions to grab the source fo the page as well as fetchBetween, fetchAfter, fetchAll, etc. You can get the latest version of the class at http://www.bradino.com/downloads/cScrape.txt - be sure to rename it to cScrape.php. If there is an interest I can continue to develop this class a tool for screens craping with PHP.

Anyway so here we go scraping all the companies from this page http://www.newyork411.com/Ad_Agencies_Production_Companies/category-cid-50553.htm as well as the details of each company, found by clicking on the company.

Step 1 - Initialize the class and fetch the page:

Step 2 - find your anchor and get the chunk of html that contains what you want

Step 3 - parse out the individual values and print out the first record for demo

Screen Scraping is a great skill that every PHP developer should have experience with. Basically it involves scraping the source code of a web page, getting it into a string, and then parsing out the parts that you want to use. A simple application of screen scraping could be to build a database of all the NFL teams complete with player details.

What the heck, let's do it... The first step is to get the page HTML into a PHP variable. This is super easy if the page is publicly accessible via a URL - no login or form post required to access... For more complex scraping you can use cURL to get the html source of the page but the rest of the process would be about the same. Anyway, let's scrape the site.

The easiest way to do pattern matching I have found is without newlines. Here is how I remove them from the raw html before I start parsing out the data I want to scrape.

So now you have the source code of the page as a string variable, you need to parse out the results. Tis is where each scraping application will differ. Depending on the page structure and what elements you want to retrieve, you will have to alter the regular expression matching. You can view the source and see that the roster data you want is in a nice table with class name "standard_table". I also notice that this class name is unique to the page. So the next step is to get the start and end string positions for this table, and then extract just the table from the content:

Now we have just the table containing the roster data, and we need to parse out the rows and cells. The easiest way to do this is with preg_match_all. If this code is not clear, you can print_r and die() in the loop to see what the rows and cells arrays contain.

So now we have parsed all the data for a given team from the official NFL site. To do all the teams, wrap this in a loop and as a final step, write all the data to a database table and voila, you have a database of all team rosters for the NFL.

This simple scraping example is just to illustrate the basic concept. Also keep in mind that if the source structure of the page you want to scrape changes, you will need to adjust your pattern matching. You should always scrape the page once and save the results in a file, then read that file into your code for development testing to minimize the hits to the live server. My personal opinion is that anything that is publicly accessible via the internet should be able to be scraped. What is the difference if you were to copy and paste it, basically that is what you are doing but doing it programmatically. You can definitely get into trouble if you misuse some data that you scraped, you could probably violate copyrights or whatever. Please scrape responsibly :)