Back to Home
Featured Case Study

Radar – Web Crawling & Data Processing Platform

A platform that collects and processes product and pricing data from around 40 Korean e-commerce marketplaces, built on an event-driven architecture with Kafka, AWS Lambda, and SQS.

Role
Backend Developer Intern
Clear timeframe
October 2025 - January 2026
Internal Enterprise Project (Confidential Source)

Project Overview

Radar is an internal platform that collects and processes product, pricing, and promotion data from around 40 Korean e-commerce marketplaces. It schedules crawl jobs, dispatches them to parallel crawlers, parses raw pages into structured product data, and syncs the results into a central database for price tracking and comparison.

System Architecture

The platform consists of three main components that communicate through message queues:

  • Orchestration Backend (NestJS): Manages platform configs, categories, and crawl schedules. Cron jobs create crawl requests, generate and release access identities (proxies, headers), monitor failed jobs, and send daily reports. Data is stored in MongoDB and MySQL (TypeORM).
  • Serverless Crawlers (AWS Lambda): Three Lambda functions chained via SQS — product listing crawl (PLP), product detail crawl (PDP), and parsing. Each marketplace has its own Crawler/Parser pair created through a Factory, using Cheerio for static pages and Puppeteer for browser-rendered pages; raw data is stored in S3.
  • Browser Crawler Service (NestJS): A long-running service for platforms with strong bot protection, managing a headless browser pool with cookie and persona rotation, plus LLM-assisted product data extraction.

Data flow: Schedule → SQS → Crawl (PLP/PDP) → S3 → Parse → Kafka → Job status pipeline → Database.

Tech Stack

Backend
NestJSTypeScriptNode.jsPuppeteerCheerio
Database
DynamoDBMongoDBMySQLTypeORM
Infrastructure
KafkaAWS LambdaAWS SQSAWS S3Docker
Testing & Tools
Jest

Key Responsibilities & Implementations

  • Multi-provider Proxy Rotation: Researched and improved how crawlers pick proxies: filtering available proxies by the provider and tier configured for each platform, assigning them round-robin to spread load across IPs, and temporarily blocking a proxy per platform once it gets flagged (with an unblock window). This made crawlers more stable and reduced IP blocking by target websites.
  • Kafka Integration: Integrated Kafka for distributed log events and asynchronous processing between services, decoupling crawl-result intake from the main processing flow so data pipelines stay resilient as traffic grows.
  • AI & Automation: Used AI tools to speed up feature implementation, generate unit tests (Jest), and automate checks, reviewing every output carefully before merging it into the codebase.