3 Ways To Scrape Infinite Scroll Sites with Playwright
Check out ScrapingBee for youself here: https://www.scrapingbee.com/?utm_source=youtube&utm_medium=link&utm_campaign=jwrooney
This is a great method for easily scraping infinite scroll websites. Three ways to scroll down the page in playwright, and using the network events we can easily expose JSON data being transferred from the websites backend.
I use this method a lot for exploring sites and seeing whats going on but it's also a great way to extract data.
# Scraper API https://www.scrapingbee.com/?fpr=jhnwr
# Patreon: https://www.patreon.com/johnwatsonrooney
# Donations: https://www.paypal.com/donate/?hosted_button_id=7HNSFPRR9N63Y
# Hosting: Digital Ocean: https://m.do.co/c/c7c90f161ff6
# Gear I use: https://www.amazon.co.uk/shop/johnwatsonrooney