Research Crawling Engineer

📋 Description

• • As a Research Crawling Engineer at MLabs, you will design and operate large-scale web data acquisition systems that power advanced AI model development by delivering massive-scale public web data to research organizations.
• • Day to day, you will construct and maintain distributed web crawlers across diverse domains, build high-throughput fault-tolerant data collection systems handling millions to billions of URLs per day, navigate anti-bot measures and JavaScript-heavy sites, and develop pipelines for cleaning, deduplicating, filtering, and normalizing multimodal data for machine learning training.
• • You will join a lean, technical team at MLabs that prioritizes speed, direct execution, and high output, supporting a client that operates a global-scale distributed crawler and sophisticated data ingestion pipelines for frontier AI research labs.
• • In this role, you will deepen your expertise in distributed systems, web-scale data engineering, and adversarial crawling environments while contributing directly to the creation of high-quality datasets used in frontier AI research and model training.