Work sample: source triage before you write the extractor

Paste a URL. It fetches the page twice, once as a bare client and once with the headers a browser sends, then reads the body for the tells that decide the extraction strategy: which of the five shapes the source is, where it will refuse you, and whether the refusal is yours or theirs.

A working prototype by Edward Tay, built for the Wynd Labs Web Scraping Specialist application. Mine, not Wynd Labs code. Method and the production system it came from: wynd.edwardtay.com