Instagram Hashtag Scraper
We’ll walk through how to scrape Instagram hashtags legally (without violating the Instagram Terms of Service) using a little-known technique involving HAR files.
You’ll be able to scrape all of the public posts for any hashtag, allowing you to scrape captions, engagement metrics and the usernames of the post authors. You can also use this HAR scraping method for anything else on Instagram, just see the video below to learn more.
1. Browse Hashtag Posts
Open up Instagram in your web browser, then navigate to the hashtag page you want to scrape the data of the posts for. You can use a search engine to find this page, or simply find a post on Instagram with the hashtag you’re interested in, then click through that hashtag to access the Instagram Hashtag Posts page.
On your web browser (we’ll assume you’re using Google Chrome, but these instructions should be similar for any browser), right click anywhere on the page and hit Inspect to open up developer tools (this will allow you to record your web traffic as Instagram serves back the hashtag posts). You can then click on the “Network” tab to see your web traffic with the data in it.
You may now want to refresh the page (now that you’re recording your web traffic), so you get the initial set of posts loaded into your browser. This will also load the “Top Posts” of the page into your browser so you’ll be able to extract this data later on.
Now with developer tools open, simply scroll through the posts (just keep scrolling down the page) and your web browser will record the raw data that Instagram is sending back to your browser about the posts.
You should be able to scroll through at least a few thousand or so… just try to take a break or don’t scroll too quickly so Instagram doesn’t think you’re a bot (even though you’re not since you’re just using the website normally).
2. Export a HAR File
Once you’re done scrolling, look for a download arrow in your browser’s developer tools (Network Tab) that’s labeled “Export HAR…” Click this and save the HAR file somewhere you can access quickly, like your Desktop:
Now that we have the raw Instagram data inside this large file, the trick is to get the data out of this file. For this, we use the HAR File Web Scraper where you can upload the file to, and the software will parse everything out for you and group together all the different requests with the hashtag posts in it:
Look for the parse group labeled with graphql - this will contain the useful data Instagram sends to your browser to show the hashtag posts.
3. Download Instagram Data
You can click the small disk icon next to each of the requests to download the raw data individually, or scroll up and click the “Parse Group” button (above all the individual responses) to download a combined CSV file with the hashtag posts in them.
You can see that the HAR File Scraper combined all of the individual requests into a combined list that you can now download as a single CSV file. The file will contain details about each post, including links to the images, videos and basic information about the author, such as user ID.
Scrape Related Hashtags
If you don’t care too much about the actual posts using the hashtag, but just want to find related hashtags, say to fill your post’s maximum of 30 unique hashtags requirement, then you can use the same HAR File Web Scraping approach outlined here.
Simply go to the Instagram website, begin recording your web traffic, and instead of browsing posts for a hashtag, just type the hashtag into the search bar and you’ll see related results. Keep typing new hashtags if you need more, and when you’re done export the HAR file and follow the steps above to extract the search results (and hashtag suggestions).
Why Scrape Instagram Hashtags?
Instagram hashtags can be very helpful for finding social media content or tracking branded content for hashtags that you or your clients promote. While it’s great to mindlessly scroll through these Instagram posts for hashtags of interest, it can be difficult to make any actual decisions without the raw data behind the hashtag posts. For example, how many people are posting with the hashtag daily, weekly, or monthly? What’s the engagement for the posts - is it increasing or decreasing?
Legal Concerns
Regardless of how useful this raw data for Instagram hashtag posts can be for influencers, marketers & researchers, Instagram prohibits the automated scraping of their data, making collecting the raw hashtag post data tricky (and we wonder why some people call it a dying platform).
Although the Instagram graph API offers an “Instagram basic” type of hashtag API for Instagram business account, you’ll find it to be extremely limiting and only allows scraping a very narrow set of data each month.
While there are automated browser extensions & hashtag scrapers that promise to scrape Instagram data for you, these nearly all violate the Instagram Terms of Service, which prohibit automated access to Instagram or the scraping of its data through the website. Using these unofficial tools will often leave footprints behind that Instagram can detect and then ban and block your account.