Search a title or topic

Over 20 million podcasts, powered by 

Player FM logo
Artwork

Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://staging.podcastplayer.com/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Episode 51: Why We Built an MCP Server and What Broke First

47:41
 
Share
 

Manage episode 491006621 series 3317544
Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://staging.podcastplayer.com/legal.

What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?

In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.

We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world

If you’re working on LLMs in production, this one’s for you!

LINKS

🎓 Learn more:

📺 Watch the video version on YouTube: YouTube link

  continue reading

51 episodes

Artwork
iconShare
 
Manage episode 491006621 series 3317544
Content provided by Hugo Bowne-Anderson. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Hugo Bowne-Anderson or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://staging.podcastplayer.com/legal.

What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?

In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.

We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world

If you’re working on LLMs in production, this one’s for you!

LINKS

🎓 Learn more:

📺 Watch the video version on YouTube: YouTube link

  continue reading

51 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play