Python Scraping Engineer (EU Applicants Only)

📋 Description

• • Join Adthena, an award-winning leader in paid search intelligence, and play a pivotal role in shaping the future of our industry. Powered by patented AI technology and a team of dedicated experts, Adthena provides unparalleled insights into the paid search market, enabling brands, marketers, and agencies to dominate their competitive landscapes. Our cutting-edge solutions, including Whole Market View, Smart Monitor, and Local View, empower clients to optimize spend, increase ROI, and maintain a decisive edge over competitors. Since our inception in 2012, we have consistently redefined search intelligence, a commitment underscored by our numerous accolades, including multiple wins at the 2025 US, European, and Global Search Awards for Best AI Search Software Solution, Best Search Software Tool, and Best Software Innovation. If you're ready to make a significant impact in an industry-leading, scaling tech organization that offers the stability of a well-established company without the associated bureaucracy, Adthena is the place for you.
• • As a Python Scraping Engineer, you will be instrumental in designing, building, and operating large-scale, resilient web scraping systems. This role is at the critical intersection of data engineering, reverse engineering, and systems reliability, ensuring the delivery of accurate, timely, and trustworthy data at scale. You will tackle complex scraping challenges where conventional methods fall short, including navigating dynamic rendering, sophisticated anti-bot defenses, frequent markup changes, and high-volume data collection. Adthena is at the forefront of adopting AI in coding practices, making this role an excellent opportunity for individuals who embrace agentic coding principles and are eager to enhance their skills in this domain.
• • Your core responsibilities will encompass:
• • Designing and maintaining robust, large-scale Python scraping systems engineered for reliable operation against highly defended targets like Google and other similar platforms.
• • Developing sophisticated scrapers for dynamic, JavaScript-heavy web pages, leveraging advanced browser automation techniques and hybrid approaches to ensure comprehensive data capture.
• • Continuously adapting and evolving scraping strategies to counter changes in page structure, request flows, and anti-automation mechanisms implemented by target websites.
• • Engineering resilient extraction pipelines that prioritize data accuracy, consistency, and observability, ensuring the integrity and usability of the collected information.
• • Implementing advanced proxy, fingerprinting, and request-routing strategies to maximize system stability, throughput, and efficiency.
• • Proactively monitoring scrape health, detecting failures at their earliest stages, and rapidly debugging production issues to minimize downtime and data loss.
• • Optimizing the performance, cost-effectiveness, and latency of our scraping infrastructure to ensure efficient resource utilization.
• • Collaborating closely with data engineers, data scientists, and product teams to guarantee that scraped data is not only accurate but also readily usable and trusted for downstream applications.
• • Documenting scraping logic, operational procedures, and runbooks to ensure long-term maintainability and knowledge transfer within the team.
• • Embracing and contributing to our forward-thinking approach to coding with AI tools, exploring and implementing agentic coding methodologies to enhance development efficiency and effectiveness.
• • Contributing to a culture of continuous improvement, sharing knowledge, and mentoring junior engineers in best practices for web scraping and data engineering.
• • Participating in the full lifecycle of scraping projects, from initial design and development through to deployment, monitoring, and ongoing maintenance.
• • Investigating and understanding the evolving tactics used by large platforms to detect and block automated scraping, developing innovative solutions to overcome these challenges.
• • Ensuring compliance with all relevant data privacy regulations and ethical considerations in data collection practices.
• • Contributing to the architectural decisions for our scraping infrastructure, ensuring scalability, reliability, and security.
• • Developing and implementing comprehensive testing strategies for scraping systems to ensure robustness and accuracy.
• • Staying abreast of the latest advancements in web scraping technologies, Python libraries, and anti-bot detection techniques.
• • Working within an agile development environment, participating in sprint planning, daily stand-ups, and retrospectives.
• • Taking ownership of critical systems that operate continuously and at scale, demonstrating a strong bias toward reliability, correctness, and observability.
• • Clearly explaining complex technical behaviors and challenges to both technical and non-technical stakeholders.
• • Demonstrating curiosity and a proactive approach to understanding how large platforms evolve their defenses and infrastructure, using this knowledge to inform our strategies.