caching proxies are one of many banes of the World Wide Web, especially when sites don't configure them properly. a scraper I wrote for a client stopped working reliably some weeks ago. testing showed the problem was intermittent, so I suspected a cache, added a cachebreaker parameter to the URL, and problem solved. the trick is to give cachebreaker a unique value each call (I'm using an up-to-the-second timestamp) and to use a parameter name that's meaningless to the server itself.

Back to blog or home page

last updated 2016-07-19 16:54:00. served from tektonic.jcomeau.com