AI Engineering Library

AI Engineering is developing quickly. We are often asked about the resources we can recommend to learn more. This is a non-exhaustive list of what we have found useful/interesting.

Prompting

• Learn Prompting, docs page
• The Prompt Report: A Systematic Survey of Prompting Techniques, paper on arxiv, summary in tweets
• How to prompt o1 (o1 isn’t a chat model – and that’s the point), blog post, by Ben Hylak

Agents

• Building Effective Agents, blog post, by Anthropic
• Hugging Face Agents Course, course
• How We Built Ellipsis (or: Lessons from 27 months building LLM coding agents), blog post, by Nick Bradford

LLMOps

• What We Learned from a Year of Building with LLMs, by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu and Shreya Shankar, Part 1, Part 2, Part 3
• Traceability and Observability in Multi-Step LLM Systems, webinar by Marc Klingen
• Data Flywheels for LLM Applications, blog post, by Shreya Shankar
• Latency optimization, cookbook, by OpenAI
• The OSS LLMOps Stack, page by LiteLLM and Langfuse

Evaluation

• Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge), blog post, by Eugene Yan
• Your AI Product Needs Evals, blog post, by Hamel Husain
• Creating a LLM-as-a-Judge That Drives Business Results, blog post, by Hamel Husain
• Evaluating Voice AI Agents, blog post and video, by Marc Klingen and Brooke Hopkins

LLM 101

Beginner-level, non-technical resources

• Intro to Large Language Models talk by Andrej Karpathy
• How I use LLMs talk by Andrej Karpathy

News

• AI News newsletter, daily roundup of top AI discussions from Discord, Reddit, and X/Twitter
• Last Week in AI podcast, weekly summary AI news and research
• Latent Space podcast, deep dives and interview episodes
• Stratechery newsletter/podcast, tech/business strategy deep dives and news, many episodes related to AI/Labs, e.g. DeepSeek FAQ, Deep Research and Knowledge Value

Libraries & Tools

While there are lots of libraries and tools out there, these are the ones we see most often among Langfuse users

• Vercel AI SDK, TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js, and more.
• LiteLLM, use 100+ LLMs as a drop in replacement for OpenAI GPT models, either library or proxy.
• Instructor, library for structured outputs with LLMs.
• Popular frameworks to get started quickly with RAG/Chains/Agents: LangChain, LlamaIndex, Haystack
• Search: Exa, Firecrawl
• Browsing: Browserbase, Browser-use
• Code sandboxes: E2B