Jobs Scraper
The Challenge
A large share of Indonesian job vacancies is published as an Instagram post rather than on a job board: an image, a caption, and a WhatsApp number. Collecting those manually does not scale, and the raw output is unusable — the same account mixes genuine vacancies with promotions, event announcements, and reposts. Scraping is also inherently unreliable: selectors move, accounts rate-limit, and a run that dies halfway needs to be diagnosable after the fact rather than silently retried.
The Solution
We built the pipeline on Bun with Puppeteer, monitoring accounts grouped by region on a schedule. Each scraped post is classified by Gemini to separate real vacancies from noise before anything is written to PostgreSQL, and the extracted fields are validated with Zod so malformed records never enter the dataset. A React 19 dashboard exposes scrape sessions as first-class objects with structured Pino logs streamed live over WebSocket — so when a run fails, the question 'what happened' has an answer in the interface instead of a server log nobody can reach.
AI classification of scraped post content
WebSocket session dashboard with per-run logs
Designed & engineered by Manggala Code
