CrawlerX - Web & Media Extractor V3

CrawlerX V3 is an enterprise-grade web crawling and media extraction extension rebuilt from the ground up for Chrome’s modern Manifest V3 Service Worker architecture. ⚡ KEY FEATURES & CAPABILITIES: • Distributed Async Dispatch: Offloads complex DOM parsing, link extraction, and dataset crawling to backend worker nodes so your browser stays fast and smooth. • Redis Task Queue Integration: Monitor crawl job statuses in real-time with automatic retries, queue metrics, and batch URL processing. • Proxy Rotation & Header Management: Route crawling requests through rotating proxies with custom user-agents to bypass strict rate limits and IP blocks effortlessly. • Intelligent AI Selector Generator: Automatically detect and build optimal CSS/XPath selectors for target elements with a single click. • Media Extractor & Downloader: Instantly discover, filter, and extract high-resolution images, videos, audio files, and documents from active web tabs. • Database Synchronization: Export datasets directly to JSON/CSV or sync with PostgreSQL, MySQL, and Redis storage backends. 🔒 PRIVACY & COMPLIANCE: CrawlerX V3 adheres strictly to Chrome Web Store security guidelines and Manifest V3 standards. All operations are isolated, permission-scoped, and secured via encrypted HTTPS background communication. What Changed in Version 3.0: • Architecture: Upgraded from local single-browser tab scraping to an enterprise Manifest V3 asynchronous Service Worker architecture integrated with a cloud backend. • Execution Engine: Added a high-concurrency Redis Task Queue engine capable of delegating heavy multi-page web crawling to server workers without freezing local browser tabs. • Proxy Management: Integrated automated residential proxy pool rotation with automatic proxy header injection (declarativeNetRequest) to bypass IP rate limits and anti-bot blocks. • Data Extraction: Upgraded from basic container scraping to support structured JSON field schema extraction, multi-level depth crawling, and automated media file downloading. • Database & Optimization: Added backend database indexing and query optimization to handle sub-second queries across large crawling datasets (100,000+ scraped records). • Centralized Dashboard: Added a dedicated web management panel (ext.q4iltd.com) for real-time task queue tracking, proxy health metrics, and job results alongside popup controls. • User Interface: Redesigned dark-themed popup UI with instant Redis job status feedback, flexible execution modes, and one-click CSV and JSON data exporting.