I've been neglecting my blog lately, posting everything to Facebook instead.
Troubleshooting why a previously-working scraper stopped working today, I remembered WebDriver.get_screenshot_as_file("path.png") and found out the site had started returning the mobile page rather than the desktop page I was expecting. As I was already getting tired of Firefox using massive amounts of RAM in "headless" mode, I switched from using MOZ_HEADLESS=1 to using xvfb-run. I kept increasing the virtual screen size until it returned the desktop page. The Makefile recipe now runs: xvfb-run --server-args="-screen 0 2048x1024x24" $(PYTHON) $<. Problem solved, I think. Waiting to hear from my client.
last updated 2020-09-14 23:39:31. served from tektonic.jcomeau.com