(New Africa/Shutterstock)
Because the earliest days of huge information, information engineers have been the unsung heroes doing the soiled work of transferring, remodeling, and prepping information so extremely paid information scientists and machine studying engineers can do their factor and get the glory. Because the agentic AI period dawns on us, it opens up a bunch of latest information engineering alternatives–in addition to doubtlessly catostrphic pitfalls.
Frank Weigel, the previous Googel and Microsoft government who was just lately employed by Matillion to be its new chief product officer, brazenly puzzled to a reporter just lately whether or not the Agentic AI Air was on a glideslope for catastrophe.
“Mainly, we see there’s an enormous downside coming for information engineering groups,” Weigel mentioned in an interview in the course of the current Snowflake Summit. “I’m undecided all people is totally conscious of it.”
Right here’s the problem, as Weigel defined it:
The explosion of supply information is one side of the issue. Knowledge engineers who’re accustomed to working with structured information at the moment are being requested to handle, prep, and remodel unstructured information, which is tougher to work with, however which in the end is the gasoline for many AI (i.e. phrases and photos processed by neural networks).
Knowledge engineers are already overworked. Weigel cited a research that indicated 80% of knowledge engineering groups are already overloaded. However whenever you add AI and unstructured information to the combo, the workload situation turns into much more acute.
Agentic AI offers a possible resolution. It’s pure that overworked information engineering groups will flip to AI for assist. There’s a bevy of suppliers constructing copilots and swarms of AI brokers that, ostensibly, can construct, deploy, monitor, and repair information pipelines once they break. We’re already seeing agentic AI have actual impacts on information engineering groups, in addition to the downstream information analysts who in the end are those requesting the info within the first place.
Supply: Shutterstock
However in keeping with Weigel, if we implement agentic AI for information engineering the unsuitable means we’re doubtlessly setting ourselves a entice that can be powerful to get out of.
The issue that he’s foreseeing would stem from AI brokers that entry supply information on their very own. If an analyst can kick off an agentic AI workflow that in the end entails the AI agent writing SQL to acquire a chunk of knowledge from some upstream system, what occurs when one thing goes unsuitable with the info pipeline? AI brokers would possibly be capable to repair fundamental issues, however what about critical ones that demand human consideration?
“You’ll have autonomous AI brokers that run whole enterprise capabilities,” Weigel mentioned. “However equally, they begin to have an enormous want for information. And so if the info workforce already was overloaded earlier than, nicely, it’s now going to be like trying down the abyss and saying ‘How on earth can we do something? How am I going to have a human information engineer reply a query from an AI agent?’”
As soon as human information engineers are out of the loop, unhealthy issues can begin taking place, Weigel mentioned. They doubtlessly face a state of affairs the place the amount of knowledge requests–which initially had been served by human information engineers however now are being served by AI brokers–is past their functionality to maintain up.
The accuracy of knowledge will even undergo, he mentioned. If each AI agent writes its personal SQL and pulls information instantly out of its supply, the percentages of getting the unsuitable reply goes up significantly.
“We’re now again at the hours of darkness ages, the place we had been 10 years in the past [when we wondered] why we want information warehouses,” he mentioned. “I do know that if particular person A, B, and C ask a query, and beforehand they wrote their very own queries, they received completely different outcomes. Proper now, we ask the identical agent the identical query, and since they’re non-deterministic, they are going to truly create completely different queries each time you ask it. And in consequence, you now have the completely different enterprise capabilities all getting completely different solutions, insisting after all that it’s proper.
Matillion CPO Frank Weigel
“You might have misplaced all of the governance and management of why you established a central information workforce,” Weigel continued. “And for me, that’s the angle that I believe a number of information orgs haven’t actually thought of. After I get a demo of an AI agent, they by no means discuss that. They simply have the agent entry the info instantly. And certain, it may well. However the issue is, it shouldn’t actually.”
The reply to this dilemma, in keeping with Weigel, is twofold. First, it’s vital to maintain information warehouses, because it serves as a repository for information that has been vetted, checked, and standardized.
It’s additionally crucial to maintain people within the loop, in keeping with Weigel. And to maintain people within the loop, human information engineers should one way or the other be prevented from changing into utterly overwhelmed by the unstructured information requests and the brand new AI workflows. To perform that, he mentioned, they primarily should develop into superhuman information engineers, augmented with AI.
Matillion is constructing its agentic AI options round this technique. As an alternative of setting AI brokers free to put in writing their very own SQL towards supply information programs, Matillion is utilizing AI brokers as supporting solid members who’s aim is to help the human information engineer in getting the work accomplished.
This on-demand workforce of digital information engineers is dubbed Maia, which the corporate introduced earlier this month. The brokers, which run within the Matillion Knowledge Producdtivity Cloud (DPC), are capable of help information engineers with a variety of duties, together with creating information connectors, constructing information pipelines, documenting adjustments, testing pipelines, and analyzing failures.
“We have to supercharge the info engineering operate, and we have to allow them to match the AI capabilities,” he mentioned. “As an alternative of only a copilot idea, it has develop into a part, a collection of completely different information engineers which have completely different duties. They will do various things.”
Maia acts because the lead agent that controls numerous sub-agents. The corporate has three or 4 such information engineering sub-agents at present, Weigel mentioned, and it’ll have extra sooner or later. Maia, which is constructed utilizing a set of enormous language fashions (LLMs), together with Anthropic’s Claude–may even appropriate itself when it does one thing unsuitable.
Matillion is near delivery a preview of Maia
“It’s actually fascinating,” Weigel mentioned. “Once you see it work, it’ll break down the issue into the steps. Then it’ll begin doing it. It can have a look at the info and determine whether or not it’s going heading in the right direction. It would roll again. ‘That wasn’t fairly proper.’ And so it actually is sort of a information engineer in its job and pondering, together with trying on the information. It can ask the human for sure at sure factors if it needs enter.”
Regardless of the potential for agentic autonomy, that isn’t a part of the Matillion plan, as the corporate sees the human engineer as a crucial backstop that may’t be eradicated from the equation.
One other vital backstop that might assist Matillion prospects keep away from agentic AI pitfalls: No AI era of SQL.
Whereas LLMs like Claude have gotten actually, actually good at writing SQL, Matillion is not going to hand the reins over to AI for this crucial part. The ETL vendor has been robotically producing SQL as a part of its information pipeline resolution for Snowflake, Databricks, and different cloud information warehouses for years, and it’s not about to begin from scratch.
“The key in Matillion is we’ve abstracted that layer so we’re a lot nearer to the person intent,” Weigel mentioned. “So the person is constructing that information pipeline intent with predefined constructing blocks that in the end write SQL. But it surely’s Matillion that writes SQL, not the person.”
This method additionally avoids the issue of getting spaghetti SQL code that may’t be up to date and modified over time, which is a chance with AI-generated code.
“We’ve this abstraction of this intermediate illustration of those elements that in flip points SQL,” Weigel mentioned. “And so our agent doesn’t should generate no matter code you want. As an alternative, it’s about selecting the correct part and configuring the precise part after which sequencing them collectively.”
It’s simple to get mesmerized by the “shiny object” syndrome within the tech world. With all of the advances in generative AI, it’s tempting to letting these shiny new copilots free to attempt to replicate the job of the overworked, under-appreciated information engineer, at a fraction of her price.
But when changing information engineers with AI additionally means changing a lot of the governance and management the info engineer brings, that might spell catastrophe for corporations. “I believe information engineering groups aren’t perhaps totally conscious of the potential doom that’s there,” Weigel mentioned.
As an alternative, corporations needs to be trying to super-charge these overworked information engineers utilizing AI, which Weigel mentioned is the perfect hope for surviving the AI information deluge.
Associated Objects:
Are We Placing the Agentic Cart Earlier than the LLM Horse?
Matillion Bringing AI to Knowledge Pipelines
Matillion Seems to be to Unlock Knowledge for AI
