The Data Engineer’s Guide to GenAI and AI Integration - The New Stack

Join our community of software engineering leaders and aspirational developers. Always
stay in-the-know by getting the most important news and exclusive content delivered
fresh to your inbox to learn more about at-scale software development.

RESUBSCRIPTION REQUIRED

It seems that you’ve previously unsubscribed from our newsletter
in the past. Click the button below to open the re-subscribe form
in a new tab. When you’re done, simply close that tab and continue
with this form to complete your subscription.

RE-SUBSCRIBE

The New Stack does not sell your information or share it with
unaffiliated third parties. By continuing, you agree to our
Terms of Use and
Privacy Policy.

Welcome and thank you for joining The New Stack community!

Please answer a few simple questions to help us deliver the news and resources you are interested in.

COUNTRY

REQUIRED

Great to meet you!

Tell us a bit about your job so we can cover the topics you find most relevant.

How many employees are in the organization you work with?

REQUIRED

Welcome!

We’re so glad you’re here. You can expect all the best TNS content to arrive
Monday through Friday to keep you on top of the news and at the top of your game.

What’s next?

Check your inbox for a confirmation email where you can adjust your preferences
and even join additional groups.

Follow TNS on your favorite social media networks.

Become a TNS follower on LinkedIn.

Check out the latest featured and trending stories while you wait for your
first TNS newsletter.

As a JavaScript developer, what non-React tools do you use most often?

✓

Angular

✓

Astro

✓

Svelte

✓

Vue.js

✓

Other

✓

I only use React

✓

I don’t use JavaScript

2025-01-24 11:00:57

The Data Engineer’s Guide to GenAI and AI Integration

contributed,sponsor-hasura,sponsored-topic,

While the early GenAI applications required a lot of coding, more straightforward tools are emerging.

Jan 24th, 2025 11:00am by

Saket Saurabh

When we started Nexla in 2016, our mission was to make data readily available to any data consumers, which also involved making it easier to integrate apps. We believed that ML and AI would completely change data engineering.

However, we did not expect generative AI (GenAI) to change everything this fast. Its adoption by enterprises is happening faster than Cloud Computing or Big Data, and these past technologies were labeled as do-or-die extinction events. GenAI seems much more urgent, like an incoming tsunami, and you have to ride or die now.

GenAI is difficult to use. High-tech powerhouses and startups are already demonstrating its power. However, as with most innovations, the tools they create and use require skillsets that most companies lack. Tooling needs to be simplified before the rest of the world can adopt GenAI.

Don’t panic.

First, you will continue to have a job. GenAI is not (yet) replacing people, and it’s not replacing data engineers for a long time. If you’re a data engineer willing to try out some new tools and know where to use GenAI first, you’ll be fine.

While the early GenAI applications required a lot of coding, more straightforward tools are emerging. At Nexla, we used GenAI to help automate the repetitive tasks that take up a lot of time in integration. Less technical engineers can integrate data by building reusable data products and workflows without coding. The connectors, schema, and how the data products and workflows are implemented all get generated. This includes building GenAI retrieval augmented generation (RAG) pipelines (more on RAG below.)

Finding your first uses of GenAI has also become more straightforward. Just figure out which employees or customers could use assistants, also called copilots, to let them do a task independently. One of the more common examples is a chatbot assistant that provides recommendations utilizing a knowledge base such as a support database.

You can think of this dual use of GenAI as AI integration — integration for AI by AI.

GenAI In the Real World: TripAdvisor and Bloomreach

Many examples now exist of companies using GenAI to create more personalized experiences or improve operations. We work with several companies on it. You’re probably already benefitting from GenAI and may not realize it.

The travel platform TripAdvisor uses GenAI to improve the experience of more than 400 million monthly active users and 1 billion reviews. They created an AI-powered trip planning application that creates detailed day-by-day itineraries with personalized recommendations based on user profiles and preferences to help users plan their trip, such as an overview of restaurants in the Back Bay area of Boston. TripAdvisor has also started to use GenAI internally to support customer service, sales, marketing, finance, accounting, HR, and analytics departments. This information is now more accessible to users, search engines, and external (partner) apps.

TripAdvisor integrated GenAI with its recommendation engine, using GenAI to help the engineering team write code. This is one of many examples of AI integration.

Another great example is Bloomreach, a cloud-based e-commerce experience platform that personalizes 25% of eCommerce experiences in the US and UK. They use AI to improve search and product matching. They also provide Bloomreach Clarity, a conversational shopping product that uses natural language, such as English, and images to help find the right products.

AI-powered integration has helped Bloomreach cut its integration times in half. This is important when onboarding eCommerce customers. But more importantly, it also improves data accuracy. There is no good AI without good data. Better AI outcomes lead to improved end-user satisfaction with each eCommerce experience.

The Cliff Notes — The Data Engineer’s Guide to GenAI and AI Integration

After seeing these and other GenAI projects, here’s my advice.

Invest in Core Data Engineering Skills

AI can’t replace data engineers. These projects still required data engineers who understood the fundamentals of data engineering — including data modeling, DataOps best practices, and, yes, SQL optimization. Make sure you continue to build up these core skills. Even if GenAI makes recommendations, you still need to be the expert that makes the final decision.

Learn GenAI RAG Now

Gen AI RAG is the most common pattern for implementing GenAI. Instead of re-training or fine-tuning a large language model (LLM), which can be very expensive and complicated, you load the relevant data, your knowledge base, into a vector database. Whenever a question (prompt) is asked, you first search in the vector database for the most relevant context. You then send the question and context to a generic LLM, like ChatGPT or Llama.

GenAI RAG is the way to go. You need to know how to use it; this tutorial can help.

Try Using GenAI-Powered Tools

The tutorial uses Nexla to build your GenAI RAG pipeline, and several parts of the pipeline development are actually “developed” for you using GenAI. Don’t be afraid to try new AI-powered integration tools. Be scared if you don’t because others will.

AI-powered integration has improved integration productivity by 2x or more in most of the projects we’ve seen. It automates the little details that slow integration down, like extracting data schema and identifying the best standard model, mapping transforms, or automating schema evolution in a data pipeline.

Also, GenAI isn’t perfect. As an expert, you must be able to spot issues (see Point 1).

Get Ready for Agentic AI

Agentic AI just means an agent is driving the “chatbot” interaction with an LLM to make decisions independently. Over time, more companies will begin to fine-tune models using techniques like Retrieval-Augmented fine-tuning (RAFT) or training and re-training their oodels.

This may sound hard; again, don’t panic. Some are saying 2025 will be the year of agentic AI. That means you have some time to learn.

It’s Deja Vu All Over Again

While some say AI is changing the world, it’s not changing the need for data engineers. Those core skills are still needed. But as with every new trend, you must keep up with the times. That means you need to learn and use RAG now and start learning more about fine-tuning and model training next.