We are seeking an experienced AI/LLM Data Engineer to build and maintain the data pipeline for our Generative AI platform. The ideal candidate will be well-versed in the latest Large Language Model (LLM) technologies and have a strong background in data engineering, with a focus on Retrieval-Augmented Generation (RAG) and knowledge-base techniques. This role sits in the AI COE within DX Tech & Digital. As a AI/LLM Data Engineer (you will report into the Director, AI Solutions & Development who oversees the AI COE.
You will work on highly visible strategic projects, collaborating with cross-functional teams
to define requirements and deliver high-quality AI solutions.
The ideal candidate will have a passion for Generative AI and LLMs, with a proven track record of delivering innovative AI applications.
Responsibilities
• Design, implement, and maintain an end-to-end multi-stage data pipeline for LLMs, including Supervised Fine Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) data processes
• Identify, evaluate, and integrate diverse data sources and domains to support the Generative AI platform
• Develop and optimize data processing workflows for chunking, indexing, ingestion, and vectorization for both text and non-text data
• Benchmark and implement various vector stores, embedding techniques, and retrieval methods
• Create a flexible pipeline supporting multiple embedding algorithms, vector stores, and search types (e.g., vector search, hybrid search)
• Implement and maintain auto-tagging systems and data preparation processes for LLMs
• Develop tools for text and image data crawling, cleaning, and refinement
• Collaborate with cross-functional teams to ensure data quality and relevance for AI/ML models
• Work with data lake house architectures to optimize data storage and processing
• Integrate and optimize workflows using Snowflake and various vector store technologies
• Master's degree in Computer Science, Data Science, or a related field
• 3-5 years of work experience in data engineering, preferably in AI/ML contexts
• Proficiency in Python, JSON, and related tools
• Strong understanding of LLM architectures, training processes, and data requirements
• Experience with RAG systems, knowledge base construction, and vector databases
• Familiarity with embedding techniques, similarity search algorithms, and information retrieval concepts
• Hands-on experience with data cleaning, tagging, and annotation processes (both manual and automated)
• Knowledge of data crawling techniques and associated ethical considerations
• Strong problem-solving skills and ability to work in a fast-paced, innovative environment
• Familiarity with Snowflake and its integration in AI/ML pipelines
• Experience with various vector store technologies and their applications in AI
• Understanding of data lakehouse concepts and architectures
• Excellent communication, collaboration, and problem-solving skills.
• Ability to translate business needs into technical solutions.
• Passion for innovation and a commitment to ethical AI development.
• Experience building LLMs pipeline using framework like LangChain, LlamaIndex, Semantic Kernel, OpenAI functions.
• Familiar with different LLM parameters like temperate, top-k, and repeat penalty, and different LLM outcome evaluation data science metrics and methodologies.
Preferred Skills
...Job Description Job Description Senior Financial Accountant Reporting to the Manager of Accounting and Financial Services,... ...financial statements of the Health System performance. This is a part-time position with our Finance Team. Position Qualifications...
...Perform repairs or installation of special purpose personnel elevators, dumbwaiters, material lifts and freight lifts. Learn to diagnose... ... A proactive team player with a strong work ethic. Basic mechanical knowledge or prior experience in construction, electrical, or...
...ManTech seeks a Python Developer to join our team in the Ft. Meade, MD area. In this role, you will develop, maintain, and enhance complex and diverse software systems (e.g., processing-intensive analytics, novel algorithm development, manipulation of extremely...
...About Us At Point Quest support isnt just a valueits the foundation of everything... ...supportive workplace can make. School Social Worker Key Responsibilities: Student... ...for interventions. Family & Community Liaison: Act as a bridge between home, school...
...work with a great management team in a clean and safe environment. Our team in University Park, IL is hiring immediately for Stand-Up Forklift Operators. Forklift Operators are the backbone of our client. All shifts are available. . Perks & Benefits: Paid...