Senior Data Engineer

📋 Description

• • Are you a seasoned data professional with a passion for building robust data infrastructure and pioneering the integration of Artificial Intelligence? Kong Inc. is seeking a Senior Data & AI Engineer to join our dynamic Revenue Analytics team. This pivotal role offers a unique opportunity to own both the foundational data infrastructure and the cutting-edge AI systems that drive revenue insights and intelligent experiences across our entire organization.
• • In this capacity, you will be instrumental in designing, constructing, and maintaining reliable, scalable data pipelines and sophisticated data models. Your expertise will be crucial in determining how we effectively connect these vital data assets with advanced AI tools. This includes making strategic decisions about which AI solutions to invest in, and ensuring their implementation is both secure and cost-efficient, aligning with Kong's long-term data and AI strategy.
• • You will operate at the exciting intersection of data engineering, AI/ML, and platform architecture, enjoying high visibility and the chance to make a tangible impact on our strategic direction. This role is perfect for someone who thrives on innovation and wants to shape the future of data and AI within a leading technology company.
• • **DATA INFRASTRUCTURE & PIPELINES:**
• • Design, build, and meticulously maintain ETL/ELT pipelines, leveraging Fivetran and Snowflake integrations to seamlessly ingest data from a diverse array of sources into our Snowflake data warehouse.
• • Develop and expertly manage robust data models within Snowflake, ensuring optimal structure for performance, unwavering reliability, and intuitive usability for analysts and business stakeholders.
• • Utilize Hightouch to operationalize data, efficiently syncing warehouse data to downstream CRM, marketing, and sales tools, thereby enhancing business process automation.
• • Proactively monitor pipeline health, swiftly troubleshoot data quality issues, and implement sophisticated alerting mechanisms to preemptively identify and address potential failures.
• • Create comprehensive documentation for data models, pipelines, and data lineage, fostering a culture of data literacy and enabling self-service analytics across the organization.
• • **AI INFRASTRUCTURE & INTEGRATIONS:**
• • Seamlessly integrate cutting-edge LLMs, such as Claude and others, directly with our Snowflake data warehouse, unlocking the potential for AI-powered querying, sophisticated summarization, and insightful generation directly on live revenue data.
• • Construct and maintain essential data sources, semantic layers, and advanced search services within Snowflake Cortex and other connected AI platforms.
• • Design and deploy intelligent AI agents capable of reasoning over both structured and unstructured revenue data to provide critical support for go-to-market workflows.
• • **AGENT ORCHESTRATION:**
• • Architect and expertly manage complex, multi-step agent workflows, orchestrating interactions across various tools, APIs, and data sources to automate intricate analytical and operational tasks.
• • Conduct thorough evaluations of potential orchestration frameworks, such as LangChain, LlamaIndex, or custom-built solutions, selecting the most effective options for our specific use cases.
• • **AI STRATEGY & EVALUATION:**
• • Execute rigorous evaluations of AI tools, models, and platforms to identify the optimal solution for each unique use case, comparing options like Snowflake Cortex, Claude, Gemini, and custom fine-tuned models.
• • Develop comprehensive evaluation frameworks that assess quality, latency, cost, and security, providing critical data to inform build vs. buy decisions and guide our overarching AI roadmap.
• • Remain at the forefront of the rapidly evolving AI landscape, proactively recommending innovative tools and approaches as the field matures.
• • **SECURITY & GOVERNANCE:**
• • Implement and manage robust row-level security (RLS) within Snowflake to ensure AI tools only surface data to users based on their authorized access levels.
• • Maintain and evolve role-based access controls (RBAC) in conjunction with new RLS policies, ensuring a layered security approach.
• • Actively contribute to data governance practices, encompassing access controls, PII handling, and meticulous schema management.
• • Collaborate closely with Data, Security, and Legal teams to establish and enforce stringent AI data governance standards and operational guardrails.
• • **COST MANAGEMENT & OPTIMIZATION:**
• • Diligently monitor and manage AI credit consumption across Snowflake Cortex, API usage, and other platforms to ensure all spending remains within allocated budgets.
• • Identify and implement strategic optimizations, including caching, prompt tuning, model selection, and query efficiency improvements, to reduce costs without compromising the quality of AI outputs.
• • Develop insightful reporting to provide stakeholders with clear visibility into AI spend and usage trends.
• • **CROSS-FUNCTIONAL PARTNERSHIP:**
• • Forge strong partnerships with Revenue Operations, Finance, and Sales teams to deeply understand their data needs and translate these requirements into scalable, effective engineering solutions.
• • Collaborate effectively across both technical and non-technical teams to deliver impactful data and AI solutions that directly influence and shape our revenue strategy.