@FunkoWalgreens is back online
The twitter bot @FunkoWalgreens has been offline for some time as it was unreliable. A change to the way the HTML is scraped allowed for all products to reliably be captured on each run. The bot will once again work as intended.
Issues revolved around the bot trying to scrape the page as a spoofed browser due to limitations o f a normal get request. However, adding appropriate headers to the get request allows for the entire page the be returned. Using browser spoofing was ineffective because the site would lazy-load, causing us to only pick up about half the items on each execution.
No comments:
Post a Comment