Using Scrapy's Rule class this can be accomplished with Hence, all we have to do is look for URLs matching this pattern. The idea is that we can control the spider's behavior by specifying a few simple rules for which links to parse, and which to follow in order to find more links.Įvery product has a storefront URL /app// determined by its unique Steam ID. Scrapy has an existing CrawlSpider class for exactly this kind of job. This listing is more than 30,000 pages long, so our crawler needs to be able to navigate between them in addition to following any product links. Our job is made somewhat easier due to the existence of a complete product listing which can be found by heading to Steam's search page, and sorting the products by release date. We first write a crawler whose purpose is to discover game pages and extract useful metadata from them. We will improve this caching setup a bit later. HTTPCACHE_EXPIRATION_SECS = 0 # Never expire. Next, configure rate limiting so that your scrapers are well-behaved and don't get banned by generic DDoS protection by adding Install the required Python packages: pip install scrapy smart_getenvĪnd start a new Scrapy project in the current directory with scrapy startproject steam. I decided to go with Python 3.6 but you should be able to follow along with earlier versions of Python with only minor modifications along the way.
Start with setting up and initiating a virtualenv: mkdir steam-scraper
Setupīefore we jump into the nitty-gritty of scraper construction, here's a quick setup guide for the project. If you are only interested in using the completed scraper, then you can head directly to the companion GitHub repository. What follows is a step-by-step guide explaining how to build up the code that's in this repository, but you should be able able to jump directly into a section you're interested in.
My goal in this guide is to help scraping beginners bridge that gap. Here's what some of the fields we are interested in look like on the page.Įven for a well-designed and well-documented project like Scrapy (my favorite Python scraper) there exists a definite gap between the getting started guide and a larger project dealing with realistic pitfalls. 1 If you want to perform your own analysis of Steam reviews, you therefore have to extract them yourself.ĭoing so can be tricky if scraping is not your primary concern, however. While all kinds of Steam data are available either through official APIs or other bulk-downloadable data dumps, I could not find a way to download the full review dataset. The Steam game store is home to more than ten thousand games and just shy of four million user-submitted reviews. This is a guest post from the folks over at Intoli, one of the awesome companies providing Scrapy commercial support and longtime Scrapy fans.
0 kommentar(er)
