Tech news from the best sources
Most football data scrapers on the market only extract high-level final scores (e.g. 2-1 ). But quantitative sports analysts, data scientists, and p…
A reader on an earlier post described a scraper that was pulling Amazon listings through a rotating residential proxy and getting prices back in thr…
A YouTube channel scraper in Python can collect channel names, subscriber counts, and URLs. That solves extraction. It does not solve creator qualif…
Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations) YouTube search in a browser loads HTML. YouTube search as a client s…
If you scrape Amazon through rotating residential proxies, your price column probably contains dollars, euros and zloty at the same time. Nothing in…
Browser automation is already a solved problem. At least, that's what I thought before I started working on Figranium. There are excellent projects…
Google Trends has no public API. What it has is the same internal JSON endpoints the trends.google.com single-page app calls — and those endpoints d…
A funding announcement tells you a company has money. A job posting tells you exactly where they're about to spend it — and it's published weeks bef…
Most of what B2B enrichment vendors sell is information companies publish about themselves, for free, on their own websites. What you're paying per-…
Most data jobs re-pull the whole source every run and waste time and money on data that didn't move. A change-detection engine wakes up, checks what…
Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my e…
I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part Indonesia's biggest marketplace has a data problem: everyone…
We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of release automation worked, and the day ended wi…
The best lead you will ever get is someone publicly asking for the thing you sell. Not a scraped list. Not a cold list bought from a broker. An actu…
I spent the last 4 days building a multi-site job scraper. Not a toy project — an actual production tool that runs on Apify and works reliably. This…
Every government funding program answers the same five questions: who can apply, how much money, what it's for, what the deadline is, and how to app…
Every developer who has worked with web scraping knows the pain: a website updates its UI, a CSS class name changes, and suddenly your entire data p…
Every few days someone in a scraping forum asks a version of the same question: "I'm collecting documentation text for an AI tool, but the pages ren…
I have a scraper that watches ten sites I consider adjacent to my work. It has been running since early July. The output goes into a vector store th…
In the last post I listed six ATS platforms whose job boards are open JSON APIs — Greenhouse, Lever, Ashby, Workable, Recruitee, SmartRecruiters. Th…
Most "job scraping" tutorials teach you to render a careers page in a headless browser and pick apart the HTML. For the majority of company career p…
My oldest actor is also my most profitable one - it's caught more paying clients than anything else I've shipped. Which made it worse when I found o…
Most tech companies run their careers page on an ATS: Greenhouse, Ashby, Workable, Workday, BambooHR and a handful of others. Nearly all of those ex…
What if you could collect YouTube search results, videos, channels, comments, replies, and transcripts through one Python interface? No browser auto…
If you have ever tried to pull data from Google Trends programmatically, you have met this: HTTP 429 Too Many Requests Not after a hundred requests.…
I scraped all 122 products on HMRC's MTD software directory. 71% cannot finish a tax return. By Solomon Amos, Founder of TapTax The UK is in the mid…
Every guide to scraping job boards starts the same way: "first, get the company's board token." Greenhouse, Ashby and Workable each expose a public…
A few months back I was running Sales Navigator searches for a client project — filtering down to "VP Sales, fintech, based in Italy or Spain" type…