Senior Web Scraping Engineer - Python

📋 Description

• • Architect and own the next generation of Oxylabs’ 60 PB/month web-scraping engine. You will design fault-tolerant scrapers that handle 300 k requests/sec while keeping success rates above 99.9 % for Fortune-500 clients.
• • Reverse-engineer anti-bot defences in real time. From CAPTCHAs to JavaScript challenges, you will craft resilient strategies that unblock sites at scale and keep data pipelines flowing without interruption.
• • Build and maintain Python micro-services that parse, validate and enrich terabytes of raw HTML/JSON daily. Your code will run inside Docker/Kubernetes clusters and integrate with FastAPI, SQLAlchemy, Redis, Elasticsearch and Kafka to deliver sub-second latency.
• • Drive observability across the scraping stack. You will add metrics, traces and structured logs to Grafana, Prometheus and Kibana so that anomalies are caught before customers notice.
• • Optimise resource utilisation and cost. With 6 PB of Ceph storage and hundreds of nodes, every CPU cycle and egress byte counts; you will profile, benchmark and tune until the platform is both lightning-fast and cost-efficient.
• • Collaborate with a multicultural squad of gamers, cyclists and guitarists who share a passion for clean code. Daily stand-ups, design reviews and pair-programming sessions ensure knowledge spreads quickly and no blocker lasts long.
• • Mentor junior engineers and write internal playbooks. Your tips, tricks and hacks will become the gold standard that shortens onboarding from weeks to days.
• • Contribute to open-source and speak at meet-ups. Oxylabs sponsors conference trips and encourages you to showcase the cutting-edge techniques you develop.
• • Maintain legacy scrapers while green-fielding new ones. You will refactor monoliths into testable modules, introduce CI/CD pipelines and raise test coverage to enterprise grade.
• • Stay ahead of the web-scraping arms race. Whether it is TLS fingerprinting, headless browsers or AI-based bot detection, you will prototype countermeasures and roll them out globally within days.
• • Work fully remote from anywhere in Poland with flexible hours. Async communication via Slack and Notion keeps the team aligned across time zones while respecting your personal schedule.