Proxy Cost Optimization: Reduce Bandwidth Spending Without Sacrificing Performance and Speed
Introduction Proxy services have become essential infrastructure for businesses operating at scale. Whether you're scraping web data, conducting mar…
Latest News
⚑ Report a ProblemTech news from the best sources
All topics
AI
Gear
News
Tech
agents
ai
api
architecture
automation
beginners
career
database
devchallenge
devops
javascript
llm
machinelearning
mcp
opensource
performance
productivity
programming
python
react
security
showdev
testing
tutorial
typescript
webdev
Building an Enterprise Football Data Pipeline: Decoding Flashscore's Protocol for xG & Referee Analytics
Most football data scrapers on the market only extract high-level final scores (e.g. 2-1 ). But quantitative sports analysts, data scientists, and p…
Your Proxy Is in Germany. The Rest of Your Client Isn't.
A reader on an earlier post described a scraper that was pulling Amazon listings through a rotating residential proxy and getting prices back in thr…
YouTube Channel Scraper in Python: Filter Active Creators
A YouTube channel scraper in Python can collect channel names, subscriber counts, and URLs. That solves extraction. It does not solve creator qualif…
Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)
Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations) YouTube search in a browser loads HTML. YouTube search as a client s…
Your Amazon scraper is returning mixed currencies and won't tell you
If you scrape Amazon through rotating residential proxies, your price column probably contains dollars, euros and zloty at the same time. Nothing in…
Why I Built Another Browser Automation Platform
Browser automation is already a solved problem. At least, that's what I thought before I started working on Figranium. There are excellent projects…
Google Trends API: the 200 OK that means you got soft-blocked
Google Trends has no public API. What it has is the same internal JSON endpoints the trends.google.com single-page app calls — and those endpoints d…
Job Postings Are the Most Underused B2B Intent Signal (And You Don't Even Need to Scrape)
A funding announcement tells you a company has money. A job posting tells you exactly where they're about to spend it — and it's published weeks bef…
Company Enrichment Without the Per-Credit Tax: Website to Firmographics JSON With an LLM
Most of what B2B enrichment vendors sell is information companies publish about themselves, for free, on their own websites. What you're paying per-…
Building a Change & Alert Engine That Webhooks Any Diff on Any URL
Most data jobs re-pull the whole source every run and waste time and money on data that didn't move. A change-detection engine wakes up, checks what…
Three Lines to Draw Before You Scrape Instagram
Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my e…
I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part
I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part Indonesia's biggest marketplace has a data problem: everyone…
The day our scraper fleet shipped nothing — and the afternoon it shipped three
We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of release automation worked, and the day ended wi…
Find people asking for what you sell, using LinkedIn post search
The best lead you will ever get is someone publicly asking for the thing you sell. Not a scraped list. Not a cold list bought from a broker. An actu…
I Built a Production Job Scraper in 4 Days — Here's What I Learned
I spent the last 4 days building a multi-site job scraper. Not a toy project — an actual production tool that runs on Apify and works reliably. This…
The Hidden Reason Government Funding Data Is So Hard to Trust
Every government funding program answers the same five questions: who can apply, how much money, what it's for, what the deadline is, and how to app…
I Built a Self-Healing Web Scraper That Refuses to Break
Every developer who has worked with web scraping knows the pain: a website updates its UI, a CSS class name changes, and suddenly your entire data p…
Stop Reaching for a Headless Browser to Scrape Documentation Sites
Every few days someone in a scraping forum asks a version of the same question: "I'm collecting documentation text for an AI tool, but the pages ren…
A permissive robots.txt is not a licence
I have a scraper that watches ten sites I consider adjacent to my work. It has been running since early July. The output goes into a vector store th…
Workday job boards have a JSON API too. It's just better hidden.
In the last post I listed six ATS platforms whose job boards are open JSON APIs — Greenhouse, Lever, Ashby, Workable, Recruitee, SmartRecruiters. Th…
Six ATS platforms publish their job boards as open JSON. Here are the endpoints.
Most "job scraping" tutorials teach you to render a careers page in a headless browser and pick apart the HTML. For the majority of company career p…
My most profitable actor was silently failing most of its runs. The bug was one unprotected line.
My oldest actor is also my most profitable one - it's caught more paying clients than anything else I've shipped. Which made it worse when I found o…
How to find any company's job board from just its domain, without a list
Most tech companies run their careers page on an ATS: Greenhouse, Ashby, Workable, Workday, BambooHR and a handful of others. Nearly all of those ex…
ytscrape: One Python Interface for YouTube Data — No Browser, No API Key
What if you could collect YouTube search results, videos, channels, comments, replies, and transcripts through one Python interface? No browser auto…
Google Trends 429s your first request. The fix is 106 bytes.
If you have ever tried to pull data from Google Trends programmatically, you have met this: HTTP 429 Too Many Requests Not after a hundred requests.…
I scraped all 122 products on HMRC's MTD software directory. 71% cannot finish a tax return.
I scraped all 122 products on HMRC's MTD software directory. 71% cannot finish a tax return. By Solomon Amos, Founder of TapTax The UK is in the mid…
Search every Greenhouse, Ashby and Workable job board at once, without a company list
Every guide to scraping job boards starts the same way: "first, get the company's board token." Greenhouse, Ashby and Workable each expose a public…