Radar – Web Crawling & Data Processing Platform
A platform that collects and processes product and pricing data from around 40 Korean e-commerce marketplaces, built on an event-driven architecture with Kafka, AWS Lambda, and SQS.
Project Overview
Radar is an internal platform that collects and processes product, pricing, and promotion data from around 40 Korean e-commerce marketplaces. It schedules crawl jobs, dispatches them to parallel crawlers, parses raw pages into structured product data, and syncs the results into a central database for price tracking and comparison.
System Architecture
The platform consists of three main components that communicate through message queues:
- Orchestration Backend (NestJS): Manages platform configs, categories, and crawl schedules. Cron jobs create crawl requests, generate and release access identities (proxies, headers), monitor failed jobs, and send daily reports. Data is stored in MongoDB and MySQL (TypeORM).
- Serverless Crawlers (AWS Lambda): Three Lambda functions chained via SQS — product listing crawl (PLP), product detail crawl (PDP), and parsing. Each marketplace has its own Crawler/Parser pair created through a Factory, using Cheerio for static pages and Puppeteer for browser-rendered pages; raw data is stored in S3.
- Browser Crawler Service (NestJS): A long-running service for platforms with strong bot protection, managing a headless browser pool with cookie and persona rotation, plus LLM-assisted product data extraction.
Data flow: Schedule → SQS → Crawl (PLP/PDP) → S3 → Parse → Kafka → Job status pipeline → Database.
Tech Stack
Key Responsibilities & Implementations
- Multi-provider Proxy Rotation: Researched and improved how crawlers pick proxies: filtering available proxies by the provider and tier configured for each platform, assigning them round-robin to spread load across IPs, and temporarily blocking a proxy per platform once it gets flagged (with an unblock window). This made crawlers more stable and reduced IP blocking by target websites.
- Kafka Integration: Integrated Kafka for distributed log events and asynchronous processing between services, decoupling crawl-result intake from the main processing flow so data pipelines stay resilient as traffic grows.
- AI & Automation: Used AI tools to speed up feature implementation, generate unit tests (Jest), and automate checks, reviewing every output carefully before merging it into the codebase.
Related projects
Ecochic - Internal Business Management Platform
Internal enterprise platform supporting HRM, financial management, and digital signing workflows, built with a microservices-oriented architecture.
NestJSFastifyPostgreSQL+12DevLink – IT Recruitment Platform
An IT recruitment platform connecting candidates and employers, featuring AI-assisted resume matching, career guidance, and automated job aggregation from external sources.
Next.jsReactRedux Toolkit+11