Unlocking Agent Creation - Agentic Architecture Lessons

Ben Kus

Summary

Building Intelligent Systems: An Agent-Based Approach

Introduction: The talk explores the journey from traditional large-scale systems using machine learning and generative AI to an agent-based approach, highlighting the challenges and benefits of this shift.

Initial Challenges:

• Large-Scale Systems: The initial use of machine learning and generative AI faced issues with data extraction accuracy and system scalability.
• Generative AI Issues: Problems like incorrect field extractions and inconsistent results highlighted the need for a different approach.

Agent-Based Approach:

• Transition to Agents: Implementing agents that handle specific tasks (e.g., OCR) led to better problem-solving and adaptability.
• Sub-Agents and Feedback: Each agent could specialize in tasks, with feedback loops allowing continuous improvement without system rebuilds.

Key Insights:

1. Agent Communication and Coordination: Agents work like a team, each handling their task while contributing to the overall goal, similar to human collaboration.
2. Scalability and Manageability: Breaking down problems into smaller tasks makes scaling more manageable and adaptable.
3. Quality Feedback Mechanisms: Agents learn from mistakes by trying different techniques, enhancing accuracy over time.

Benefits:

• Flexibility and Adaptation: The system can easily evolve with new document types, requiring minimal changes.
• AI-First Engineering Organization: This approach fosters a culture where engineers and customers collaborate, leading to efficient tool development.

Conclusion: Adopting an agent-based approach offers a modular, adaptable solution for handling complex documents. It emphasizes flexibility, ease of evolution, and collaborative problem-solving between humans and machines.

Auto-Highlights

Auto-Highlights: Highlight: large scale systems, Count: 1, Rank: 0.08 Highlight: AI first engineering, Count: 1, Rank: 0.07 Highlight: machine learning based systems, Count: 1, Rank: 0.07 Highlight: generative AI, Count: 2, Rank: 0.07 Highlight: AI generator, Count: 1, Rank: 0.07 Highlight: weird things, Count: 1, Rank: 0.06 Highlight: data extraction steps, Count: 1, Rank: 0.06 Highlight: other document, Count: 1, Rank: 0.06 Highlight: different models, Count: 1, Rank: 0.06 Highlight: first customer, Count: 1, Rank: 0.06 Highlight: data scientists, Count: 1, Rank: 0.06 Highlight: unstructured data, Count: 1, Rank: 0.06 Highlight: structured data, Count: 1, Rank: 0.06 Highlight: sub agents, Count: 1, Rank: 0.05 Highlight: client file customers, Count: 1, Rank: 0.05

Photos

Transcript

...requests from many companies and many of. There's actually a whole industry here that you've probably never heard of called idp. And it's really oriented around machine learning based systems where you would like train a model, get a bunch of data scientists to do this kind of extraction. But it really didn't work that well. Historically, many companies just, they would only automate things that are extremely high scale. It just wasn't very commonly utilized. And also we would break all the time, very brittle because if you change the format of anything, it would just kind of stop working. So when generative AI came out, this was like a gift for anybody who deals with unstructured data because you can actually just use the power of AI to be able to pull out structured data. So for us we started with this architecture really straightforward. Take a document, take the fields that you're looking for, do some pre processing and then some OCR and then be able to give it to the large ending model. You say give me these fields and it pops it up. You can be attracted to. This is amazing. When we did this, we immediately deployed it. 10 million pages, first customer, everything was working and we got to the point where we were saying like, this is can do any document now. This is amazing. And so it was truly built around the basics of AI on content. And so this was great. It was kind of like generative AI solved. We did it. High fives. But then we started to hit the problems when we start telling our customers, just give us any data and we'll be able to extract the things you want. Like they did. And so they were like, oh, I've never been able to automate this thing before. This 300 page document that was well beyond the context windows of the time. And we're like, okay, no problem, we'll pre process more. We built the concept of like an enterprise rag where we were able to get the data out and so okay, solve that. But then they were like, okay, turns out OCR doesn't work that well in certain cases when people cross things out or when you have languages. So we had to start to solve that. Then we had this challenge where some people were like, okay, I want not just 20 pieces of data from this document, but like 200 or 500 different pieces. And that just kind of like overwhelmed the attention of the model to be able to pull all those things out for you, especially on complex documents. And then people in this world are used to things like confidence. They're like, well, how do I know what's right, what's your confidence score. And of course generally AI doesn't have confidence scores like what ML models do. So we had to like start to do things like oh, we'll run an element as a judge and it'll tell you after it's done it thinks it was accurate or not. And I was like okay, sure. But like it told me it was wrong, so why are you telling me if it says it's wrong? So we ended up with all these challenges and this was like our moment of like the tropical disillusionment of AI generator because the thing that was working so well was so awesome, it was so elegant, just didn't work. And so for us, we like a natural engineer response to this is like, okay, we'll pre process more or we will solve each of those little problems. But then we were thinking about it more and we watched answering's deep learning class with Erison and then we realized that if we applied an agentic approach to this then maybe you can get a much better outcome. And some people at the time were like that's kind of crazy. This is not an agent agent, this is just a function. Get the data out of this document. It's not that hard. And so then we ended up re architecting from scratch with an agentic approach. So rather than just do the pre process, pull out the data post process, we did a steps multi agent architecture where we separated out the problems that we had into a series of sub agents whose job was to solve these kind of problems and solve them intelligently. So like when it came across some of these files and somebody said I have 500 fields. Our previous heuristic based approach is like oh, just chop it up into different field groups. Stopped working when you had like client file customers of your contract and then customer addresses. Those kind of go together. Otherwise weird things happen in a large model. And so those kinds of of things that we learn to group together and then being able to do things like when it went to go extract the data rather than us pre deciding what it should do it agentically figure out, I'm going to, I'm going to call this to get these parts of the data. Maybe he's going to look at the picture of the pages in addition to just the OCR and then we incorporated a quality feedback loop not just to give you confidence but then also to give feedback so that the AI could try different techniques. Looks like field number three is wrong. So all right, well let me try again. Maybe I'll use different models to vote and to do other techniques. And this approach really solved a lot of our problems, not just because it solved issues at that moment, but because it became easy for us to update. And this really sort of is the key of what we learned here, which is that when you're thinking of building these intelligent powered solutions, if you do an agent based approach, it's a much cleaner abstraction and you start to like from an engineering perspective, especially if you're dealing with like large scale systems, you start to separate out rather than it be like, okay, we need a large scale conversion, an OCR system that processes all these things. You start to think of it like I've got one document and I've got to get through these fields and I'm just going to think of it the way that you would do it as a person or as a team of people. And this really helped the abstraction for us to then go in and be able to improve it. And this is maybe the biggest benefit was it was very easy to evolve. We got to the point where we were saying, oh, for this kind of lease document and for this other kind of other document, like it's going to make sometimes a specialized agent who has some specialized routine to do these things and the ability for us to quickly evolve rather than say, oh I know, we'll build a new distributed large scale system and just say let's just add a new supervisor to the graph to double check the results when you're done. This let us quickly evolve. And then so that when a customer came to us and said it's not working very well on this new crazy type of document, given it, we could say ah, give us a little bit of time to build you a slightly updated agent to go do these things. And then the last piece here is I didn't quite like sort of fully realize this at the time was that by making your engineers think about AI and agentic workflows and think about the kind of lessons that you learn when you're building these things, they then start to think about customers. So many of our customers are actually building their own land graft powered or other system powered agents. And then so they'll call us as a tool. And so then they'll be. Our engineering teams will now be like, oh, I have some ideas on how it might be easier for us to make the tools that call us to do these kind of data extraction steps or anything else easier. And so this is one of the key lessons was as many people on this quest to build an AI first engineering organization, these kind of like actually building this way helps quite a bit. So if I went back in time to talk to myself before, or if you asked me for advice on anybody who's got an existing system and you said, I'm going to build some intelligent features, what advice would I give? My number one piece of advice, I think I'm the last speaker here, so maybe this is a piece of advice that can hopefully summarize. Part of this conference is if you think you have. If you start to go down a path of building something, build agentic, build it early. And with that, thanks, everyone.