3 free resources on this topic. Everything here is free and hand-picked. You can also search within this topic.
A KDnuggets tutorial by Abid Ali Awan that walks through scraping thousands of pages with the Olostep Batch API in Python, downloading the results as Markdown and checking how many requests completed. Running the code requires an Olostep API key.
Developer documentation for Olostep's /v1/batches endpoint, which submits large lists of URLs for scraping in one job and returns page content such as Markdown or HTML, with request parameters, status polling and retrieval examples. Olostep is a paid web-data API; reading the documentation is free.
A walkthrough of using the Apify command line tool to let AI agents run web scraping and automation tasks, aimed at people building their own small automations.
Why I recommend it: This is for the tinkerers. If you have ever wanted a repeatable way to pull data for lead lists or market research, this is a concrete starting point rather than another think piece.
