Scraping the Craigslist API
If you need to scrape data from Craigslist, we’ll walk through the only legal way to do so that doesn’t violate their Terms of Service by using a recording of your web traffic. We’ll simply browse any set of results on Craigslist while recording what the Craigslist API sends to our browser and then scrape that recording instead of the website.
You’ll be able to scrape the basic information for any post in a search, including the full body of the post which often contains emails and phone numbers for contact.
1. Browse Craigslist Postings
Go to your favorite Craigslist Site and browse to your desired category. You can apply any filters or sort options you’d like, the only thing you need to do is select the “Preview” option for viewing the search results.
Once you have that set up, right click on the page and hit “Inspect” to open up developer tools, which will automatically begin recording your web traffic now as you browse Craigslist. Now refresh the web page to force the Craigslist API to re-send the details for the first result now that you’re recording.
Now click on the 2nd result to get it to load the “Preview” (this is getting the Craigslist API to send us the data for the 2nd post) and wait until you see it load on the right side of the page. Then keep clicking on each result and going through as many pages as you need.
While a bit tedious, this approach is completely safe and legal as we’re not using any automation or other risky practices that could get your IP address banned.
2. Export a HAR File
Once you’ve clicked on enough results, go to the “Network” tab in developer tools and click the down arrow labeled “Export HAR…” to download a HAR file containing all of the network traffic that was sent to your browser as you loaded up each Craigslist post.
Upload that HAR file to the HAR File Web Scraper and then look for the group similar to the above screenshot. Specifically, look at the fields and look for the group that has fields like “title” and “items” in it. Click “Parse Group” to combine the API responses together into a downloadable file.
3. Download Craigslist Listings
When the parse finishes, look for the <root> collection which will flatten and combine all of the individual API responses for each of the Craigslist posts into a single CSV file you can download or browse.
Some useful fields you’ll find in the CSV download are outlined below. Please be responsible with this data and only use it for internal research. Do not republish it elsewhere without permission!
- Post ID (Useful to de-duplicate results if needed)
- Title
- Full Body (Contains contact information)
- Category
- Post Timestamp (Unix Epoch)
- Post URL
- Location Area
- Location Neighborhood
- Latitude & Longitude Coordinates
- Custom Attributes (E.g. in-person or virtual)