Last night's Data & AI London meetup tackled two critical topics shaping data and AI right now: building reliable multi-agent systems and navigating the evolving open storage landscape. With AI agents moving from hype to production reality, understanding both the technical foundations and practical implementation challenges is essential.
We were joined by Tom Stuart, Founding Engineer at PortiaAI, who shared battle-tested insights from building production multi-agent systems, and Holly Smith, Staff Developer Advocate at Databricks, who took us on a deep dive to reveal what really matters for data teams.
Tom brought real-world experience from PortiaAI, where he's solving one of the biggest challenges for the next wave of AI: making agent systems reliable enough for production. With his background as a former fintech CTO and Monzo security engineer, Tom offered practical insights on bridging the gap between hype and production reality, and the importance of software engineering fundamentals to building reliable agent systems.
Tom identified three core pillars for AI agents for ensuring reliability: Spectrum of autonomy and understanding where you want to aim for, maintaining a human in the loop, and Security at the heart of the approach.
These systems need mechanisms to hand off to humans in a structured way, reducing hallucinations and increasing accuracy.
“You need to understand where on the spectrum your use case sits to pick the right tooling and approach. Pick the right approach for the problem you’re trying to solve.”
Deploying agents shouldn’t mean organisations open themselves up to risks we’re already familiar with.
Multi-agent systems need security and authentication built into their core architecture, not bolted on later. Anyone developing or owning these systems should take a hands on approach to security issues. Without this, adoption will stall, trust will fall and we won’t see the real impact of these systems.
Holly, an award-winning Data & AI expert with over a decade of experience, delivered an engaging session bringing a critical look at what makes good data storage, comparing Delta, Hudi, and Iceberg. She provided a valuable perspective on what really drives format decisions.
Everyone wants data that's fast to read and fast to write to, but this isn't the only consideration for data storage. On top of that, you need to consider its compatibility with other tools in your data ecosystem, be it current or future states. Finally consider data flexibility features like schema & type evolution alongside ACID transactions and time travel.
Although many people face similar problems solved by open lakehouse table formats, these problems are not identical, and neither are the three leading projects in this space. Choose the right projects that match the business problems you have.
The conversations between sessions – shared challenges, collaborative problem-solving, connections that spark new ideas – are just as valuable as the formal presentations. As we navigate rapid changes in AI and data infrastructure, learning together makes all the difference.
Connect with the Data & AI London community and join us at our next meetup. We're not just witnessing the future of AI and data—we're building it together.
