Gamestop is back online
Gamestop changed their website two weeks ago and the scraper has been broken for as long. I finally figured out how to fix everything so the scraper is back online.
Moving over to the new version was not straight forward. First, none of the filters you apply on the website are reflected in the URL. I had to dig through the network requests to find the params that set various filters and manually add them to the URL. This enables us to filter only for Gamestop Exclusives and put a large number of items on the page at once so we don't need to scrape multiple times. Second, we needed to get the HTML without using basic get requests. Their server is configured to only respond to legit client requests, so browser spoofing was required to achieve success. However, this means we can drop the Incapsula breaker library as it appears their site no longer uses Incapsula to prevent access. Finally, we needed to filter sold out times. An item being sold out is not obvious when looking at the search results page. Fortunately, the availability is specified deep in the HTML of the page so we're able to parse that out to determine if something is in stock and reverse engineer a URL to post to various sites.
Moving over to the new version was not straight forward. First, none of the filters you apply on the website are reflected in the URL. I had to dig through the network requests to find the params that set various filters and manually add them to the URL. This enables us to filter only for Gamestop Exclusives and put a large number of items on the page at once so we don't need to scrape multiple times. Second, we needed to get the HTML without using basic get requests. Their server is configured to only respond to legit client requests, so browser spoofing was required to achieve success. However, this means we can drop the Incapsula breaker library as it appears their site no longer uses Incapsula to prevent access. Finally, we needed to filter sold out times. An item being sold out is not obvious when looking at the search results page. Fortunately, the availability is specified deep in the HTML of the page so we're able to parse that out to determine if something is in stock and reverse engineer a URL to post to various sites.
Changed to work with new Funko Shop
Previously, all Funko shop exclusives were listed at funko-shop.com. However, Funko has removed that site and merged it into shop.funko.com. Their weekly exclusives will now be sold on shop.funko.com. I have updated the scraper to pull the new items from there. This also has the benefit of pulling any restocks of exclusives that might happen. We are still scraping the store exclusives page as well on shop.funko.com.
Thanks
ReplyDelete