Web Scraping

Web scraping is the process of automatically extracting data from websites, web pages, and online documents, typically using specialized software or algorithms. As the amount of online data continues to grow, web scraping has become a crucial tool for businesses, researchers, and developers to gather insights, monitor trends, and inform decision-making, making it a highly relevant topic in the tech community.

Web Scraping

Related Stories

Feed the Bots

AI Scrapers Request Commented Scripts

Messing with Scraper Bots

AI Crawlers, Fetchers Are Blowing Up Websites; Meta, Openai Are Worst Offenders

Aggressive Bots Ruined My Weekend

Blocking LLM Crawlers Without Javascript

You Don't Need Anubis

Feedmaker: URL + CSS Selectors = Rss Feed

Asking AI to Build Scrapers Should Be Easy Right?

Stop Crawling My HTML – Use the API

A Stateful Browser Agent Using Self-Healing Dom Maps

Poisoning Well

Webhound (yc S23) – Research Agent That Builds Datasets From the Web

You Can't Parse XML with Regex. Let's Do It Anyways

Simplex (yc S24) – Browser Automation Platform for Developers

Feed Me Up, Scotty – Custom Rss Feed Generation Using CSS Selectors

Tomcp.org – Turn Any URL Into an Mcp Server

Sacrificing Accessibility for Not Getting Web Scraped

Amazon Sends Legal Threats to Perplexity Over Agentic Browsing

The Accidental Click That Changed Everything: the Apify Origin Story