I’ve been watching a pattern emerge across the best AI tools, and it’s the opposite of what everyone expected. The winners aren’t the ones generating perfect final outputs. They’re the ones generating instructions that humans can inspect, modify, and execute.

The old paradigm treats AI like an oracle. You prompt, it generates, you take it or leave it. The new paradigm treats AI like a recipe writer. You prompt, it generates transparent instructions, you control the execution, you get exactly what you need.

This shift makes AI actually useful, not any less powerful.

Why Black Boxes Keep Failing

Direct generation has fundamental problems that no amount of model improvement will fix. When something’s wrong, you can’t trace the logic. When requirements change, you start from scratch. Hallucinations lurk in final outputs until production explodes. You lose all optionality.

A few years ago, I was advising teams building financial analysis tools. Their first instinct was having LLMs generate complete spreadsheets. Seemed logical, right? Except when formulas broke, nobody could trace why. When assumptions shifted, they’d regenerate everything and pray. The breakthrough came when we flipped the approach: have the LLM write Python that generates spreadsheets. Suddenly you had inspectable logic, testable assumptions, and modifiable parameters.

That lesson has become a pattern.

Why Instructions Beat Outputs

I’ve identified four concrete benefits that explain why this approach dominates:

Inspectable Logic means seeing the thinking, not just the answer. Errors become visible before they cause damage. Hallucinations surface as syntax errors in code rather than silent failures in production.

Parametric Control lets you change ingredients without rewriting recipes. Look at napkin.ai - you adjust one color and keep the entire design system. Claude Code lets you tweak a function while preserving the architecture.

Format Flexibility gives you one recipe for many meals. The same graphic becomes PNG for web, PDF for print, PPT for presentations. Same code deploys to different environments.

Timing Control means executing when ready. Review before running. Test in sandbox first. Re-run with confidence.

I’ve written about before, combining the creativity of probabilistic generation with the reliability of deterministic execution. We get both paradigms working together.

Code Leads and Everyone Else Follows

Programming had a 50-year head start treating instructions as first-class objects. Version control, testing frameworks, code review, IDEs - an entire ecosystem built around manipulating instructions rather than outputs. Claude Code doesn’t just generate code; it generates code that slots into this ecosystem.

Now other domains are catching up fast. napkin.ai isn’t a drawing tool - it’s a parametric design system generator. Browser automation tools don’t perform tasks - they generate observable command sequences. Data platforms create reproducible analysis pipelines, not just query results.

The Evolution Unfolding Now

Phase 1 (2022-2024) saw everyone trying to make LLMs directly produce final outputs. Most attempts failed spectacularly.

Phase 2 (2024-2025) brought the developer breakthrough. We discovered that generating code - already an instruction set - worked better. GitHub Copilot, Claude Code, and Cursor led the charge.

Phase 3 (2025 and beyond) spreads the pattern everywhere. Design gets parametric graphics through napkin. Data analysis gets Python/SQL notebooks. Automation gets browser scripting.

Where This Goes

The next frontier is executable reasoning itself.

Instead of “here’s my analysis,” imagine getting the decision framework as code. Assumptions become adjustable parameters. Analysis re-runs with different inputs. You get reasoning recipes, not just conclusions.

This extends beyond simple generation into how we structure complex thinking itself. The same pattern applies - intermediate representations that preserve optionality and control.

Here’s the builder’s opportunity: Stop asking “What can AI generate?” Start asking “What’s the right intermediate representation for this domain?”

The winners won’t eliminate human control. They’ll amplify it through better recipes. Because the goal was never to replace chefs with robots. It’s giving every cook access to Michelin-star recipes they can make their own.

A friend recently overhauled his entire hiring process. Not because of budget cuts or growth pivots, but because hundreds of fake applicants flooded their pipeline, some even using AI to impersonate real people during calls.

The problem wasn’t that candidates used AI. It was that some hid it when asked directly.

There’s a clear ethical line here. Using AI to augment your work? Smart. Hiding AI use when asked point-blank? Dishonest. Not using AI when everyone else does? Career limiting.

While companies play whack-a-mole with “AI cheaters” their competitors are hiring AI-augmented superstars who are upfront about their tools. The real question: How well do you use AI, and are you honest about it?

The New Baseline

The market has moved. 88% of organizations now use AI regularly, up from 78% a year ago. This isn’t just tech companies. Marketing saw the biggest adoption increases, and 62% of organizations are experimenting with AI agents.

I’ve watched this play out in teams I work with. Strong engineers with solid fundamentals leverage AI to become even more effective. Less experienced developers copy-paste AI-generated code without understanding the implications. The difference? The first group has the judgment to know when to trust AI and when to trust themselves.

This is the amplification effect: strong fundamentals plus AI equals 10x improvement. Weak fundamentals plus AI equals scaling bad judgment. Engineers with liberal arts backgrounds often excel here because they’ve been trained to think critically about context, a meta-skill that matters more than ever.

You’re screening for effective AI users who understand the tool’s role, not just anyone who’s touched ChatGPT. Bonus: people who hide AI use probably lack the judgment you need anyway.

What to Actually Test

Instead of asking “did you use AI?”, test for the four dimensions that matter. Anthropic’s identifies these as Delegation, Description, Discernment, and Diligence.

Delegation: “Tell me about a time AI failed you”

This tests their mental model of AI’s limitations. Watch out for “AI can do everything” (naïve) or “AI is useless” (stubborn). Look for candidates who know when to use automation versus augmentation versus staying hands-on. “I use Claude for planning, Cursor for implementation, but always review architecture decisions myself” is exactly what you want to hear.

Description: “Walk me through your workflow for <relevant task>”

You’re testing orchestration and communication clarity. Good candidates use multiple tools strategically: “I start with Perplexity for research, move to Claude for synthesis, then validate key claims manually.” Watch out for people using one tool for everything or being unable to articulate why they chose specific tools.

Discernment: “Show me something you shipped. What did AI contribute versus you?”

This tests critical evaluation. I call this the $100/month test: ask what AI tools they pay for personally. It shows real commitment beyond casual use. Look for candidates who can explain what they verified and where they added strategic value: “AI drafted the code, but I redesigned the data model because it missed the scaling implications.”

Diligence: “How do you verify AI outputs?”

Ask them to walk through checking an AI-generated technical spec. Strong candidates demonstrate source triangulation (”I check the AI’s claims against official docs”), metacognition (”I know I don’t know X, so I verify those sections more carefully”), and ethical consideration (“I always check if the solution creates accessibility issues”).

Put It Into Practice

For technical roles, give them pre-written AI code with subtle bugs. Tell them: “You have full AI access. Fix this and explain what you’d deploy to production.” This tests whether they can debug, if they blindly trust AI fixes, and if they understand the underlying system.

For non-technical roles, provide AI-drafted content that’s 80% good and ask them to improve it for your target audience. Can they identify what’s generic versus strategic? Do they add business context AI missed?

Universal questions that work across roles:

• “What’s changed in your AI workflow in the last year?” (The field moves weekly; they should move fast too)

• “Name an AI tool you tried and abandoned. Why?” (Shows discernment, not just adoption)

• “How would you help a skeptical colleague adopt AI?” (Tests if they can be an AI champion)

The honesty test is simple: ask what AI tools they used to prepare for the interview. You don’t care if they used AI. You care if they’re honest about it.

Why This Matters Now

Only 33% of organizations have scaled AI beyond pilots, despite 88% using it. The most in-demand roles are software engineers and data engineers, precisely where AI amplification matters most.

Organizations with AI champions see 80% success rates versus 37% without. High performers in McKinsey’s research (those seeing 5%+ EBIT impact) are 3x more likely to have redesigned workflows around AI, not just adopted it.

Here’s the moat: hire one AI-fluent person, they transform 10-20 colleagues, and your whole department levels up.

The Takeaway

Ask how well candidates use AI and whether they’re honest about it. The days of hiding AI use are over.

The next generation of top performers won’t hide their AI fluency. They’ll demonstrate it. Your job is building interview processes that surface this competitive advantage.

Your PM just shipped a working prototype in 48 hours. Your non-technical founder built a PoC that actually impressed investors. The future of software development is here, powered by AI, and everyone can build now.

Except there’s a problem nobody wants to talk about.

After working with multiple teams and talking to friends deep in the trenches, I’ve noticed a pattern. Some call it the ‘vibe coding hangover’. The honeymoon phase is over, and teams are discovering that building an app and shipping an app are two very different beasts.

TL;DR

• Vibe coding democratizes the first 90% of development - tools like Lovable, Replit, and Bolt let non-developers build functional apps

• The last mile is still brutal - deployment, infrastructure, security, and optimization require deep expertise that AI struggles with

• This creates a dangerous gap - working prototypes that can’t become production apps

• The logistics analogy holds - like shipping, the final leg to users is disproportionately complex and expensive

• Solutions exist but require intentionality - know your purpose, align processes, and most importantly, know when to bring in experts

What Is Vibe Coding?

Vibe coding platforms let you describe what you want to build in natural language, and AI generates the code. Think Lovable for full-stack apps, Replit’s AI agents, Bolt for rapid prototyping, and to a lesser extent, Cursor (though it’s built for developers).

Examples are everywhere now. PMs who once wrote PRDs are shipping prototypes. Non-technical founders validate concepts before raising capital. The barrier to entry for software creation has essentially evaporated.

Why Now?

I’ve written before about how AI democratizes access to the median professional. AI makes writing boilerplate super easy. It handles the predictable patterns that make up most code. But here’s the catch: this works best over short time horizons and discrete tasks.

Tasks like simple question answering from a domain expert’s perspective? Perfect for AI. Building a complex, production-ready system? That’s where things get interesting. It’s why Replit has been emphasizing how long its agents can last on a task: they know the challenge isn’t starting, it’s finishing.

The Problems Start Compounding

Eventually, when you’re building something new, you have to start innovating and making contact with the real world - infrastructure, security, performance. A common misconception is that programming is very rote, but the best engineers exercise tremendous creative problem solving. Having “median” code generated by an LLM leads to diminishing returns fast.

Vibe coding can lead to serious security issues because deep understanding of systems matters. You can’t prompt your way around not knowing how authentication actually works.

In teams, the problems multiply:

• Non-technical users throw AI-generated code over the wall to engineering

• Decades of software development best practices get ignored because they’re invisible: automated testing, version control for understanding changes and enabling rollbacks, code reviews that disseminate knowledge

• Nobody wants a 30K line code review, but that’s exactly what vibe coding platforms often produce

Infrastructure issues are particularly brutal for AI. I’ve written about debugging a nightmare deployment with Claude Code.

These problems require systems thinking and a complete mental model. Feedback loops are slow. Setting up a domain name might take hours before you know if it worked. Without a good mental model to debug, you end up in frustrating loops.

The Last Mile Problem

Here’s an old engineering joke: coding is the first 90% of work. Then there’s the other 90%.

In logistics, the final delivery leg from distribution center to doorstep accounts for 40-50% of shipping costs despite being the shortest distance. It’s inefficient due to scattered residential stops, traffic, failed deliveries, and individual address complexities.

In software:

• Development to Production = Warehouse to Doorstep

• “Works on my machine” = Code runs in ideal conditions but fails in real environments

• Disproportionate effort = Last 20% (deployment, debugging, infrastructure) takes 50%+ of total effort

Both reveal the same truth: building the thing is the easy part. Getting it successfully into users’ hands is where the real challenge lies.

Real-World Example

A friend recently built a simple SaaS app using Lovable. The prototype phase was magical - natural language to working app in under three hours. User authentication, a clean dashboard, data visualization, even AI pixie dust on it. It all worked perfectly on his computer. Then came deployment to Railway, one of the most developer-friendly platforms out there.

What should have been simple became a multi-day saga. Environment variables were hardcoded. The SQLite database that worked locally needed PostgreSQL in production, breaking some queries with different syntax. Stripe integration looked perfect but failed on webhook signatures and OAuth redirects.

The killer was performance. What ran instantly locally took 5-10 seconds per page load in production. N+1 query problems everywhere, no pagination, I could go on. Three hours to build, four days to deploy - and this was on Railway, a platform specifically designed to make deployment easy.

How Lovable Is Addressing This

Their recent announcements show they understand the problem:

Lovable Cloud - A managed backend solution. They’re trying to become a Platform-as-a-Service faster than PaaS providers become vibe-coders.

Shopify Integration - Smart move. Shopify is already a well-understood platform with clear constraints. There’s good overlap with the low-code audience. A developer wouldn’t need “text to Shopify” - Shopify is already easy enough.

Imagi/OpenAI Partnership - Bringing Lovable to schools. Education was Replit’s initial audience too. Different use case than professional development.

From the Trenches

I spoke to a friend with FAANG experience who transitioned from professional designer to developer:

“Lovable has super good design sense,” he told me. “But it fails at full backend, production quality code.”

He described needing to correct the AI over and over. The codebase got too big for Lovable’s context window. With Claude Code, you need to direct less since it has codebase access, but you’re still dealing with what I call “a savant with amnesia.”

Niche issues that aren’t well documented are brutal to debug with any AI tool. They need tons of direction. “AI doesn’t know what it doesn’t know, or what you don’t know,” as I put it.

At one point, Claude Code literally told him: “You should contact Supabase support.”

“Read the f***ing docs” prompting has become common. You get stuck in unproductive loops making small edits. “Replit is clunky,” he concluded. “Not meant for professionals.”

Solutions That Actually Work

Be intentional about purpose:

• Is this a prototype to align stakeholders? Keep it in Lovable, share via their preview links

• A proof of concept for the engineering team? Export the code early, review together

• The foundation for a production app? Plan for the handoff from day one

Align processes around purpose:

• Work cross-functionally early and often. Have engineers review vibe coded apps weekly, not after “completion”

• Work with internal and external experts on the last mile - bring in DevOps early, not as firefighters

• Document the evolution of ideas so handoffs are smoother. Have AI generate a decision log after each session

• Know when to “graduate” from one tool to another. Example: PM uses Lovable for wireframing, Engineering rebuilds core logic in Cursor with proper architecture

Create systematic handoff points:

• At 1000 lines of code (coarse but useful measure): Architecture review

• Before any external integrations: Security review

• Before user data touches the system: Infrastructure planning

• When the context window fills up: Time to migrate to professional tools

Budget realistically:

• The “last mile” typically takes 2-3x the initial build time

• If Lovable got you there in 2 days, budget at least a week for production

• Factor in costs for monitoring, error tracking, and actual infrastructure

• Remember: an optimized app can be 10x cheaper to run than a naive implementation

Vibe coding doesn’t eliminate the need for expertise, it just moves it.

The Problem Will Get Worse

Building is easier, but building was always the easiest part. As more building occurs, talent and expertise for the last mile gets spread thinner. For example, today there are more developers than ever, but fewer who understand how to work with databases at a deeper level beyond code.

The gap between what can be built and what can be shipped is widening. Vibe coding platforms are incentivized to encourage creation, not necessarily helping users deliver business value. An optimized app runs faster and cheaper. It mitigates catastrophes like security breaches.

Moving Forward

The vibe coding revolution is real and valuable. It’s democratizing software creation in ways we couldn’t imagine three years ago. But let’s be honest about what it is and isn’t.

It’s a powerful tool for exploration, validation, and rapid prototyping. It isn’t a replacement for the deep expertise required to ship production software that scales, performs, and protects user data.

If you’re experiencing this gap between prototype and production, you’re not alone. The tools are evolving, but in the meantime, knowing when and how to bring in expertise for that crucial last mile makes all the difference.

After all, friends don’t let friends ship vibe coded apps... without a plan for the last mile.

Are you stuck in the gap between a working prototype and production-ready app? I help teams navigate the last mile of vibe coded applications. Get in touch if this resonates with your current challenges.

The fix was three words. The investigation? 1,427 lines across four days, chasing 63 suspicious database connections when only 10 mattered.

I recently debugged a bug that brought my entire application to its knees. What I learned about AI-assisted debugging turned out to be more valuable than the fix itself.

I’ll get technical in places (it’s unavoidable given the nature of the bug), but the broader lessons about AI-assisted debugging apply whether you’re writing code or not.

TL;DR

What I learned:

• Why async/sync boundaries are debugging nightmares

• How to structure AI-assisted investigations

• When human intuition beats AI exhaustiveness

• Why documentation is your debugging superpower

For the curious: You can find the full investigation and lessons learned technical documentation here.

Subscribe now

The Setup

I’m building a product that streams AI responses to users—think ChatGPT’s typing effect. This requires server-sent events (SSEs) because LLMs take seconds to generate responses, and nobody wants to stare at a blank screen. I also designed my architecture around SSEs because I expected all kinds of events being sent; some events would be based on user input, and some would not.

I used Django because I know it well, it has spectacular documentation and powerful database functionality. Because of the nature of this architecture, I decided to be async-first; this would allow efficient scaling and resource usage.

Async I/O (async) is like having a single waiter managing multiple tables instead of one waiter per table standing around waiting for each customer to finish chewing.

Traditional (synchronous) I/O:

Your program asks for data → sits idle waiting → gets the data → continues

Like calling someone and staying on hold until they answer. Your entire program is blocked.

Async I/O:

Your program asks for data → immediately moves on to other work → gets notified when data arrives → handles it

Async I/O lets your app handle way more users and requests simultaneously without buying more servers. It’s the difference between a web server handling 10,000 concurrent users versus 100 with the same hardware.

However, async support within Django and broadly within Python is relatively new, given the maturity and age of both projects1. Async Python was introduced in 2015, and Django added preliminary async support in 2019 (Python’s been around since 1991 and Django since 2005).

There are libraries and projects that are async-first, but the overall ecosystem is bifurcated into synchronous libraries and asynchronous libraries. This was the crux of the problem2: throughout my stack, some components were asynchronous and some were not leading to inexplicable hangs in the interface.

When Everything Froze

Click around too fast and the entire app would freeze. No requests could get through. Only fix? Restart the server. This also did not represent a ton of requests, and it caused concern. While I reproduced the issue by clicking around quickly, my intuition was that this would occur when even a handful of users used the app concurrently. This was the opposite of what I wanted! The whole point of the async-first architecture was supporting many concurrent users efficiently. And the only resolution was restarting the web server.

I knew it was related to async because of the nature of the stack; many components were unproven in an async environment. However, it was unclear where to start. Was it the database? Was it the API serializers? Was it thread pool exhaustion? It was time to debug and find out.

The Investigation Strategy

I started where any reasonable engineer would: the obvious culprits. Database configuration? Check. Thread pool exhaustion? Nope. When these quick wins didn’t pan out, I knew I was in for a proper debugging session. I’d dealt with concurrency bugs before - they’re the software equivalent of trying to catch smoke.

Here’s where it gets interesting: I was already using Claude Code for development, so why not use it for debugging too? But I didn’t just throw the problem at it and hope for magic. I structured the process the same way I would tackle it manually.

Every hypothesis, every test, every outcome got documented in real-time. Think of it as leaving breadcrumbs, except instead of getting eaten by birds, these breadcrumbs would actually help Claude (and future-me) understand what we’d already tried.

The workflow: I’d direct Claude to investigate specific areas. It would run tests, document findings, and I’d steer toward the next hypothesis. I ran it in maximal thinking mode because this wasn’t straightforward—it was a deep integration issue between async and sync components that required careful reasoning.

The real engineering challenge became creating a faster feedback loop. The original bug took several clicks and some waiting to manifest. That’s brutal when you’re testing multiple hypotheses. So part of the process was figuring out how to make the bug appear faster and more reliably. The quicker I could reproduce it, the quicker I could validate whether a fix actually worked.

This hybrid approach - AI doing the grunt work while I provided direction and manual verification - turned what could have been weeks of solo debugging into a more systematic investigation. Claude couldn’t replace my intuition about where to look next, but it could exhaustively check those places and document everything along the way.

Timeline

Here’s how several days of investigation actually played out:

Day 1: The Obvious Suspects

Started with the most likely culprits. First thought: maybe the system was running out of worker threads (think of these like employees handling customer requests). Bumped the limit from 10 to 100. The hang still occurred with only 4 threads busy. Definitely not that.

Next theory: maybe the database was holding onto information too long during user sessions. Disabled that feature. Still hung.

Then I tried something drastic - completely disabled the streaming connections that let users see updates in real-time. If those were the problem, this should fix it. It didn’t. The app still hung even on normal page loads. So much for the easy wins.

Day 2: The Convincing Red Herring

Checked the database during a hang and saw 63 idle connections just sitting there, all from the same security check. This looked damning - like leaving 63 phone lines open instead of hanging up. This has to be it.

Made a change to force the system to hang up after every call instead of keeping lines open. The app worked! Problem solved, right?

Wrong. During the next hang, I checked again: only 9 connections. Then 31 during another test. The database could handle 100 connections easily. This wasn’t about too many connections at all - it just happened to look that way during my first check.

Day 3: The Grind

Tried everything: different security checking approaches, adjusting how the development proxy handled requests, various configurations for managing the async/sync boundary. None worked.

Set up better monitoring tools to watch what was happening in real-time. Claude Code documented each attempt systematically. The investigation file grew longer with each dead end.

Day 4: Finding the Smoking Gun

Decided to test a wild theory: what if I just skipped the security check entirely in the streaming manager? Just let everyone through temporarily to see what happens.

Started rapidly clicking between conversations in the app. Connection count stayed at 3. No hang. No errors. Perfect performance.

Found it. The security check at the start of each stream was grabbing a database connection and never letting go - not for the quick moment it needed, but for the entire time the stream stayed open (potentially minutes or hours). When I rapidly switched between 10 conversations, each stream grabbed its own connection and held it. After 10 switches, all available connections were taken. Request number 11 had to wait, timing out after 10 seconds.

The confusing part? Only 10 connections were actually held, not 63. But those 10 connections were enough to exhaust the pool when they never got released.

The Fix

Added a wrapper function that forces the database connection to close immediately after the security check completes, rather than keeping it open for the stream’s lifetime.

The change: three words added to one line of code.

The investigation: 1,427 lines of documentation across four days.

That’s debugging.

Where AI Falls Short

Claude struggled with distinguishing root causes from legitimate issues. Claude can’t distinguish between “not the problem” and “not a problem.” It marked legitimate bugs as resolved simply because they didn’t explain the hang. I had to circle back manually to fix these issues later.

This speaks to a broader limitation: LLMs can’t maintain a nuanced “mental model” where something can be simultaneously “not the root cause” and “still needs fixing.”

Temporal reasoning remains a weak spot. Claude’s sense of time is consistently off. The investigation documents suggest this took weeks, but it actually took days. This mirrors temporal issues I’ve noticed elsewhere: when planning projects, Claude thinks everything will take weeks. When doing AI-powered research, models struggle with understanding that one source supersedes another simply because it’s more recent. The temporal confusion was most apparent in the “lessons learned” document Claude generated after reading the full investigation - it assumed the ordeal spanned weeks rather than days, presumably because the depth of investigation felt like it should take that long.

Documentation accuracy was directionally right but not always precise. The overall narrative held up, though not everything Claude documented matched exactly what happened. The core issue is how it handles granularity and structure: Claude didn’t quite grasp that hypotheses and solutions are independent “atoms” of investigation. Instead, it treated the entire documentation as one continuous stream, making it harder to distinguish between what we tried, when we tried it, and what we learned from it.

I still had to direct the investigation. Despite Claude’s ability to run tests and document findings, I needed to point it toward the right logs, suggest which areas to investigate next, and make judgment calls about what mattered. It’s like having a brilliant research assistant who can exhaustively check everything you point them to, but can’t quite develop investigative intuition on their own.

Manual reproduction was unavoidable. The debugging process was deeply manual. I had to physically click through the app to trigger the hang, over and over. In the future, I’d love to try something like a Playwright MCP or similar automation to let Claude drive the debugging process itself. But UI automation is notoriously flaky - anyone who’s maintained Selenium tests knows the pain. Perhaps tools like Stagehand, which are AI-native, could help bridge this gap. Until then, the human remains an essential component of the debugging loop.

What This Taught Me About AI-Assisted Development

Context engineering is the difference between a helpful assistant and an expensive parrot. When Claude had good documentation to work from, it could build on previous findings and make useful inferences about where to look next. This is where AI coding agents shine - their uniform ability to reference specific artifacts (@my_file) is crucial. I wish this function laity existed outside of coding assistants. I had Claude document learnings separately and more concisely so they could serve as future reference and best practices. These findings echo the context stack I described earlier this year. Organizing context isn’t glamorous, but neither is debugging. Both require the same discipline.

Human-in-the-loop isn’t a bug, it’s a feature Similar to my learnings with Atticus, current AI technology is too early to be completely autonomous. What worked really well in this workflow was being able to redirect Claude when I saw it going down the wrong path. This is why Claude Code currently has the best coding agent experience: it uses tools I use, in my development environment, and I can supervise and jump in when needed. This design philosophy becomes striking when comparing against other AI coding agents that try to do everything autonomously and end up doing nothing useful.

Artifacts are the infrastructure layer of AI debugging. In this case: code Claude could run, and documentation it could reference. Without easy access to these artifacts, Claude would be left guessing about what’s been tried and how to verify changes. The ability to create, reference, and modify concrete artifacts transforms an AI from a chatbot into a genuine debugging partner.

The broader lesson? AI assistance works best when you treat it like a skilled junior developer who’s really good at following instructions and running experiments, but needs you to provide the investigative intuition and strategic direction.

After months of experimenting with AI assistants across different projects, I've developed what I call a "modular AI context stack" – a systematic approach to managing knowledge that transforms how I work with AI. It's not just about better prompts; it's about building a reusable infrastructure for thought.

But let me be clear upfront: this is messy, manual, and very much a work in progress. I'm not selling you a polished system – I'm sharing an evolving experiment that's been valuable despite its rough edges.

The Multi-Assistant Advantage

I previously wrote about LLMs as thought partners:

I explored how AI can help us think through complex decisions. But here's what I've discovered since: using multiple AI assistants isn't just helpful – it's transformative. I regularly pit Claude, ChatGPT, and Gemini against each other on the same problems, not because I can't pick a favorite, but because their different biases and approaches create productive friction.

Think of it like assembling your own advisory board, where each member brings distinct perspectives. Claude might offer nuanced analysis, ChatGPT provides creative alternatives, and Gemini excels at technical precision. The magic happens when you compare their responses – suddenly you're seeing blind spots and opportunities you'd miss with a single assistant.

The reality? This means having three browser tabs open, manually copying prompts between them, and doing the synthesis work myself. There's no elegant orchestration layer here – just good old-fashioned manual labor. But the insights are worth the friction.

The Atomic Document Revolution

The breakthrough came when I started treating context like code – modular, reusable, and version-controlled. Instead of dumping everything into massive prompts, I create focused, single-purpose documents that live in Google Docs.

For Blazer, my SERP analysis tool, this looks like:

• Brand Guidelines: Visual identity, tone, and messaging principles

• Product Strategy: Market positioning, competitive analysis, and roadmap

• Technical Specifications: API design patterns and data structures

• Customer Insights: Interview summaries and usage patterns

Each document is atomic – it stands alone, focuses on one thing, and acts as the "Directly Responsible Individual" (DRI) for that knowledge domain. Just as microservices revolutionized software architecture, this approach transforms AI context management.

Except, unlike microservices, there's no automatic discovery or loading. I'm manually sharing Google Doc links with Claude, or copy-pasting entire documents when link sharing doesn't work. Sometimes I forget which document has what, leading to a frantic search through my Drive. It's inelegant, but it beats starting from scratch every time.

From Documentation Avoider to Context Architect

Here's a confession: as a manager, writing documentation was always my Achilles' heel. I'd rather build ten prototypes than write one process document. But AI has completely flipped this script.

Why? Because the payback is immediate and multiplicative. When I document a decision framework today, tomorrow's AI conversation starts from that foundation rather than square one. It's like compound interest for knowledge work – every well-crafted document becomes a force multiplier across all future AI interactions.

This shift has been profound. I've gone from avoiding documentation to actively seeking opportunities to memorialize insights. The key difference? I'm not writing for some hypothetical future reader – I'm building tools for my AI collaborators to help me think better, faster, and more consistently.

That said, maintaining this system is its own burden. Documents get stale. I'll reference a strategy doc in an AI conversation only to realize it's three iterations behind my current thinking. The overhead of keeping everything updated is real, and I don't have a good solution yet.

The Power of Portable Context

Storing these atomic documents in Google Docs might seem mundane, but it's strategically brilliant. The documents live in a format that's broadly accessible – I can share them with any AI assistant, paste them into any chat interface, or reference them across platforms. No vendor lock-in, no migration headaches – just pure portability.

This approach also solves the context window problem elegantly. Instead of cramming everything into a single conversation, I can selectively load only the relevant modules. Working on brand design? Pull in the visual guidelines. Discussing technical architecture? Load the API specifications. It's precision context at scale.

The manual nature is both a feature and a bug. On one hand, I have complete control over what context each AI assistant sees. On the other, I'm the integration layer – constantly copying, pasting, and managing which documents go where. When I'm on mobile, the friction multiplies.

Looking ahead, protocols like MCP (Model Context Protocol) could transform this manual process into something more automated – imagine AI assistants directly querying your document repository based on conversation context. But for now, I'm the human API between my knowledge base and AI tools.

Building Your Own Context Stack

The principles for effective atomic artifacts are surprisingly simple:

• Single Focus: Each document should have one clear purpose

• Self-Contained: Minimize dependencies between documents

• Broadly Accessible: Use formats that work everywhere

• Living Documents: Update them as you learn

What makes this approach powerful isn't the individual documents – it's the system. You're essentially building a knowledge management OS that amplifies AI's capabilities while maintaining your unique perspective and expertise.

But here's what they don't tell you: figuring out the right granularity is an art. Too atomic, and you're drowning in documents. Too broad, and you lose the modularity benefits. I'm constantly refactoring my document structure, splitting some, merging others. It's very much a learning process.

The Meta-Skill of Context Management

As AI tools proliferate, the differentiating skill isn't knowing which model to use or how to craft clever prompts. It's becoming a master of context – knowing what information to provide, when to provide it, and how to structure it for maximum leverage.

This modular approach has transformed how I work. Complex projects that once required lengthy briefings now start with a few strategic document references. Consistency across projects has improved dramatically because core principles live in reusable modules rather than my fallible memory.

More importantly, it's made AI feel less like a tool and more like a true collaborator. When your AI assistant has access to your accumulated wisdom, structured thoughtfully, the conversations become richer and the outputs more aligned with your goals.

The Honest Truth

This system is held together with digital duct tape and determination. It's manual where it should be automated, fragile where it should be robust, and constantly evolving as I discover better approaches. Some days I wonder if I'm overengineering something that should be simple.

But despite all the friction, it works. The manual copy-pasting, the constant document updates – they're all worth it for the leverage this approach provides. I'm building a personal knowledge infrastructure one Google Doc at a time, and while it's not pretty, it's mine.

The future of AI isn't just about better models – it's about better collaboration between human expertise and AI capabilities. Building your own modular context stack is the first step toward that future. Start small, document one process or framework, and watch how it transforms your next AI interaction. The compound effects might surprise you.

Just don't expect it to be elegant. Expect it to be useful.

Breaking: Two major developments this week have catapulted MCP from promising experiment to industry standard practically overnight:

1. OpenAI announced they'll be supporting MCP across their products, with integration available today in their agents SDK and support for ChatGPT desktop app and responses API coming soon.

2. Anthropic shared that a new version of the MCP spec was finalized today, introducing critical improvements including an auth framework based on OAuth 2.1, streamable HTTP transport replacing the previous HTTP+SSE implementation, support for JSON-RPC batching, and tool annotations for better describing tool behavior.

In just four months, MCP has evolved from an idea at Anthropic about making integrations easier for developers to becoming the de facto standard for AI app integrations across the industry.

Here are my initial thoughts on this rapidly evolving technology and what it means for both builders and users.

Subscribe now

What Is MCP, Really?

Let's cut through the marketing: MCP is a protocol, not a product. Think of it as today's equivalent of early USB or OAuth standards. It defines a way for AI assistants like Claude to communicate with external applications through specialized "servers" without requiring custom code for each integration.

The promise is compelling: write integration code once as an MCP server, and any compatible AI assistant (or application) can use it. This standardization could eliminate countless redundant implementations and eventually make AI-to-application connections as seamless as authenticating with "Login with Google."

The Builder's Perspective

As someone who's built AI products, I approach MCP with cautious optimism. The protocol doesn't eliminate the need to write code — someone still needs to build those MCP servers. What it does is create the potential for standards that allow that code to be written once and used across multiple AI hosts.

Currently, implementing MCP servers requires more technical savvy than most users possess. You're still editing JSON files and managing API keys. This technical barrier means we're firmly in early adopter territory.

Initially, MCP looked like Anthropic's version of Android to OpenAI's iOS. While OpenAI had built a closed-garden approach with GPT Store and plugins, Anthropic appeared to be betting on an open ecosystem that other AI companies might rally behind to compete collectively against the market leader.

With today's announcement from Sam Altman, this dynamic has shifted significantly. OpenAI's embrace of MCP suggests we may be moving toward a more standardized ecosystem rather than competing walled gardens. This could accelerate developer adoption dramatically, as building MCP servers now potentially grants access to both Claude and ChatGPT users.

The User Experience

For users, MCP's promise is enabling AI assistants to actually do things rather than just talk about them. Instead of asking Claude to write a script to download regulatory documents, you could simply ask Claude to download them directly. That's powerful.

In practice, however, the experience remains hit-or-miss. Even after installing an MCP server, you might find yourself repeatedly prompting "Please use the database server" while Claude seemingly ignores your request. The non-deterministic nature of LLMs means you can't guarantee your MCP server will be called even when it's the perfect tool for the job.

When it works, though, it's genuinely impressive. I've seen demos where Claude analyzes database contents or updates blog tags across platforms. These experiences feel magical compared to traditional interfaces.

The Missing Pieces

Earlier this month, several crucial elements still needed addressing for MCP to reach mainstream adoption:

1. Installation and discovery - There was no centralized marketplace or one-click installation process

2. Authentication and permissions - The protocol lacked built-in OAuth-like functionality for managing access across users

3. Function namespace collision - All functions lived in the same namespace, creating challenges as collections of MCP servers grew

4. Testing and reliability - Building evaluation frameworks was difficult when you couldn't predict which model would process your requests

Remarkably, today's spec update directly addresses several of these concerns. The addition of an OAuth 2.1-based authentication framework resolves one of the most significant barriers to enterprise adoption. Just as I predicted earlier, we're seeing internet protocols building on top of each other - MCP isn't reinventing authentication but leveraging established standards.

This pattern of layered protocols mirrors how the internet itself developed. Each new protocol builds upon existing foundations rather than starting from scratch. MCP represents a new layer connecting probabilistic AI systems to deterministic applications, following the same evolutionary pattern.

Interfaces of the Future

Is MCP bound to be used only in AI assistants, and will conversational interfaces become dominant, or are they just a transitional technology?

While natural language reduces the barrier to discovering functionality (no more remembering exact menu options), the cognitive load of expressing yourself through language remains significant. Most users don't want to craft perfect prompts — they want buttons, visual elements, and guided experiences.

The most likely future combines these approaches: purpose-built applications with clear visual interfaces for common tasks, augmented by conversational capabilities for flexibility and exploration.

MCP is valuable regardless of interface evolution because it standardizes how AI-native applications connect to traditional software, creating a consistent bridge between probabilistic and deterministic systems. Whether accessed through chat, buttons, or voice commands, the underlying protocol enables seamless integration across application types.

Subscribe now

Looking Forward

With both OpenAI and Anthropic firmly behind MCP, the question of industry adoption seems settled – this protocol is now poised to become the standard way AI assistants interact with external applications. The speed of this convergence is unprecedented in protocol development, compressing what would typically take years into mere months.

The rapid evolution of the MCP spec itself is equally impressive. The inclusion of OAuth-based authentication, streamable HTTP transport, and improved tool annotations addresses several of the key limitations I observed just weeks ago. This pace of improvement suggests we may see solutions to remaining challenges like function namespace collision and testing frameworks sooner rather than later.

For builders, this represents a clear signal to invest in MCP development. Building MCP servers now grants access to both Claude and ChatGPT users – covering the vast majority of the consumer AI assistant market. This dramatically changes the ROI calculation for developers who were hesitant to commit resources to an unproven protocol.

These are just my initial impressions as the landscape evolves incredibly quickly. By the time you read this, more details about implementation timelines or additional improvements to the spec may have emerged. The AI ecosystem changes daily, and we're all figuring this out in real-time.

What's clear is that MCP represents a fundamental shift in how AI systems interact with external applications. Just as HTTP standardized web communications and OAuth standardized authentication, MCP is standardizing the bridge between probabilistic AI systems and deterministic applications – creating the foundation for the next wave of AI-native software.

I recently shutdown Atticus, my first consumer AI product that aimed to make legal contracts accessible to everyone. The journey of building and launching it taught me invaluable lessons about AI product development, market dynamics, and the future of technology.

Atticus was born in June 2023 during the initial AI boom when my friend Dylan approached me with a simple question: could AI help analyze his girlfriend's gym contract? Over the next year, we built one of the early consumer-facing AI legal tools. The journey took us through the Vercel AI Accelerator, gave us early access to GPT-4's expanded capabilities, landed us coverage in tech newsletters, and brought paying customers from around the globe.

At its core, Atticus analyzed contracts by breaking them down into plain English, determining if clauses favored the selected party, and assigning risk scores to concerning terms. Users could also engage with their contracts through an open-ended chat interface.

After a year and a half of learning, iterating, and serving users across multiple continents, I made the decision to shut it down - not because it wasn't useful (we were still making sales!), but because the path forward revealed deeper insights about the future of AI that I'll share in this post.

The Genesis Story

Dylan and I had been consulting on an AI project when he approached me about building a tool to analyze consumer contracts. Initially, we envisioned it as a simple consumer product like WordCounter.

I was excited because early findings suggested LLMs were particularly well-suited for legal problems. I wrote:

I flagged that costs might make a freemium model prohibitive, but decided to build it anyway, confident that LLM prices would drop over time. I set a one-week deadline to build and start validating the idea with our networks to mitigate the risk of dropping other opportunities.

We launched within a week and immediately recognized that running Atticus wouldn't be cheap. In June 2023, the LLM price reductions we now take for granted hadn't kicked in. We applied to the Vercel AI Accelerator for free AI credits and access to models that weren't widely available, including Claude (with its then class-leading 100K context window) and GPT-4-32K.

Technical Architecture

I built Atticus' backend with Langchain, using a multi-step pipeline to analyze contracts:

1. Extract text from uploaded documents (PDFs, docx, etc.)

2. Determine contract type and extract key information like dates and parties

3. Generate analysis criteria - the clauses an experienced attorney would look for in that contract type

4. Compare the contract against these criteria, analyzing how favorable each clause was

This approach had several advantages:

• Reduced perceived latency by splitting the analysis into stages

• Improved effectiveness by focusing the model on specific tasks

• Enabled caching by separating contract type analysis from specific instances

• Provided a dynamic, non-deterministic approach that still yielded consistent results

Initially, I spent significant time optimizing our pipeline to reduce costs, trying different RAG techniques and models. Between the Vercel AI credits and my belief that costs would eventually decrease, I pivoted to focus on launching faster.

The Search for PMF

After launch, we found that people in our networks liked the concept but didn't need a tool like Atticus consistently. We explored different ICPs over the next few months - content creators, tech consultants, construction contractors, and startup founders. I even cold emailed ~60 construction companies in NYC (0% response rate 😭).

While our rapid experimentation approach was valuable, in retrospect, we should have gone deeper with each potential ICP rather than quickly switching between them. For instance, with construction companies, phone calls might have been more effective than email outreach.

We got most of our users through:

• LinkedIn, where our "build in public" approach resonated with professional networks

• In-person demos at meetups (though mostly to AI enthusiasts rather than target users)

• Hacker News posts (briefly reaching page 1 of Show HN)

• Tech newsletters like Ben's Bites and There's An AI for That, which drove significant organic traffic

Show Me the Money

In the fall, Lunar and Dylan redesigned Atticus into something that looked like a real product:

Around this time, Dylan noticed we had users returning frequently, so we implemented usage-based billing - 4 analyses for $20. We set up Stripe within a couple of days and got our first paying customer in October 2023! Sales continued to trickle in consistently until the day right before I decided to shut down.

We applied to YC and got an interview, though we didn't get funded. Their feedback centered on Atticus not being venture-scale, which I didn't fully disagree with.

In retrospect, the YC rejection was the death knell for Atticus. I processed the rejection emotionally but moved on without fully internalizing or actively rejecting the feedback - a missed opportunity to make a more deliberate strategic decision.

After scaling down resources to reduce operating costs and switching to more affordable models, Atticus continued to generate sales. It was bittersweet to see my prediction about falling LLM costs play out in real life. 🥲

Why It Didn't Work

A concept like Atticus requires changing people's expectations of what technology can do for them. While technology evolves rapidly, people and institutions change much more slowly. As Tyler Cowen said in a recent interview: "the number one bottleneck to AI progress is humans."

Atticus excited early AI adopters but faced barriers to broader adoption. One model we explored was offering AI-augmented legal services, using AI to handle monotonous low-value work while connecting lawyers with high-value work.

However, lawyers showed little enthusiasm for disrupting their own industry. The legal system is built around billable hours, and persuading professionals to change a system that financially benefits them proved extremely difficult. As Upton Sinclair noted: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."

Potential customers (startup founders and operators) recognized the value, feeling the pain of inefficient legal processes. But we faced a chicken-and-egg problem: delivering the service required lawyers who weren't ready to adopt the technology.

An idea this disruptive could succeed, but it needs a long-term plan (10+ years) with sustainable phases along the way. Think of Amazon, which aimed to be the "everything store" but started with books before expanding category by category. We lacked such a patient approach.

Another challenge: AI products aren't well-suited to traditional software business models. Unlike deterministic software, AI models are probabilistic and sometimes make mistakes. This contradicts customer expectations that software will work the same way every time.

In retrospect, the "services as software" model that has since emerged seems most promising for AI products and agents. But this approach requires domain experts, bringing us back to the challenge of convincing technology-resistant lawyers to participate.

What Did Work

Despite these challenges, several things went right:

Speed and iteration. We got an MVP in front of users within a week and built in public, garnering attention and support. A few months in, we implemented monetization and made our first sale within days of launching Stripe integration.

Usage-based pricing aligned well with our users' needs, as most didn't require ongoing analysis. Personally, making money directly from customers was exhilarating - a milestone "you can just do things" moment after years at early-stage startups.

Ignoring the "thin wrapper" stigma. In 2023, many believed model providers would capture all the value in AI. Our experience proved otherwise - users didn't care how Atticus worked, only that it solved their problem. Several users familiar with ChatGPT still valued our bespoke solution.

Cost projections proving correct. LLM prices plummeted as predicted, making Atticus more viable over time. GPT-4-32K initially cost $60 per million input tokens and $120 per million output tokens. Today, GPT-4o costs just $2.50 and $10 respectively - over 90% lower in less than two years.

Falling Forward: Learning the Hard Way

Building and shutting down Atticus taught me lessons no book could provide. Perhaps most surprising was realizing how different AI product development is from traditional software development - you're constantly navigating probabilities and edge cases rather than deterministic behaviors.

This insight has shaped Ascention, my venture that helps teams adopt AI while building AI-based products. My vision is for Ascention to be to the AI age what 37signals was for the SaaS era – creating products and methodologies that define best practices in this new paradigm.

I've realized that the magic happens at the intersection of deterministic systems and AI capabilities. These systems require fundamentally different approaches to design, development, and user expectations.

The marketing journey revealed unexpected global interest - about 60% of our traffic came from outside the US. Legal documentation is clearly a universal pain point, though this complicated our approach given the jurisdictional complexities of legal assistance.

I confirmed my passion for 0-to-1 building and high-ambiguity environments. Starting with a blank slate and rapidly testing hypotheses energizes me in ways that later-stage product development doesn't. For truly transformative ideas like Atticus, however, you need a 10-year vision with sustainable phases along the way.

Technically, I discovered that frameworks like Langchain eventually became obstacles as I developed deeper familiarity with LLMs. The technology evolved at breakneck speed - features we built with considerable effort later became standard in LLM APIs, and expanding context windows made our document chunking strategies increasingly unnecessary.

The most practical lesson? Experiment quickly, but establish clear criteria for continuation or exit. Monetize early, even imperfectly - revenue validates your concept and provides emotional fuel through inevitable challenges.

Perhaps the most profound discovery was about agency - the power of "just doing things." There's something liberating about taking an idea from concept to reality and having strangers value it enough to pay for it. It reminds me of Theodore Roosevelt's "Man in the Arena" speech: "The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood..." Despite its outcome, I wouldn't trade the Atticus experience and the growth it fostered.

While Atticus didn't become the venture-scale business we envisioned, I remain bullish on AI's transformative potential in legal services and beyond. We were right about where the puck was going but underestimated what it would take to survive until it got there. These lessons have fundamentally shaped how I'll approach building products through Ascention - creating systems that harness AI's potential while acknowledging its unique characteristics.

Subscribe now

Acknowledgements

This journey wouldn't have been possible without my incredible collaborators:

Dylan was the catalyst that set everything in motion. From our initial conversations to driving key decisions around rapid launches and monetization, his technical prowess and entrepreneurial instincts were invaluable. Dylan's ability to transform abstract ideas into working code gave Atticus its initial momentum. He's now channeling his considerable talents into his own venture, and I have no doubt it will be exceptional.

Greg brought strategic organization and clarity amidst the chaos of early-stage building. His remarkable talent for driving alignment and "sharpening thinking" helped us navigate crucial decision points. Greg has that rare ability to synthesize complex information, identify core issues, and orchestrate teams toward coherent action.

Lunar, my friend of over a decade, transformed Atticus from a functional tool into an experience users genuinely loved. Her extraordinary design sensibility elevated everything we built. In a world where skeptics dismiss AI products as "thin wrappers," Lunar's design expertise created an interface so thoughtful that users consistently praised the experience.

The Atticus journey reinforced something I've always believed: extraordinary collaborators make all the difference. I'm profoundly grateful to have worked alongside such talented individuals, and I look forward to watching their continued impact on the technology landscape.

A few months ago, Klarna made waves by announcing they were replacing Salesforce and Workday with AI-built solutions. But this wasn't just another cost-cutting move. It was a signal of a profound shift in how software gets created: the democratization of software development itself.

The Great Software Unbundling

We've seen this pattern before. In the 1980s, the personal computer wrested control from the priesthood of mainframe operators. Before PCs, computing meant costly machines in air-conditioned rooms, operated by specialists in white coats. The PC revolution wasn't just about smaller computers—it democratized computing itself, enabling anyone to write a BASIC program or run VisiCalc.

In the 1990s, desktop publishing transformed an industry dominated by printing presses and typesetters. Tools like PageMaker and laser printers didn't just make publishing cheaper—they enabled entirely new forms of expression and business models. Small newsletters and magazines flourished, giving voice to communities that traditional publishers never served.

Today, we're seeing similar signs in software development:

• A developer builds a workout/diet planner that generates personalized meal plans and exercise routines with a single prompt

• Claude artifacts enable building 14 different tools in a week, from QR code decoders to audio processing apps

• GitHub launches Spark, promising to turn ideas into instantly deployable applications through natural language

But more importantly, look at what these tools are enabling. That workout planner isn't just a static app—it evolves through natural interaction, integrating with nutrition APIs and adapting its interface based on user feedback. The audio processing tool isn't just a simple converter—it interfaces directly with OpenAI's API to enable rich voice interactions. These aren't just faster ways to build the same old apps—they're enabling entirely new categories of personal software.

Coding's YouTube Moment

We're witnessing what many are calling "coding's YouTube moment." Just as YouTube transformed video creation, AI is reshaping software development. The parallels are striking:

Pre-YouTube:

• Video hosting required expensive setup and infrastructure

• Professional editing tools cost thousands

• Quality production needed specialized equipment

Post-YouTube:

• Free hosting and distribution

• Mobile-first editing tools

• High-quality phone cameras

Similarly, software development is undergoing its own transformation:

Before:

• Deployment required complex infrastructure knowledge

• Development needed extensive technical expertise

• Testing and iteration cycles took days or weeks

Now:

• Platforms handle deployment and scaling automatically

• Natural language interfaces lower technical barriers

• Rapid prototyping and iteration happens in minutes

Just as Hollywood initially dismissed YouTube as a platform for "cat videos," some developers today argue that AI-assisted coding can't create "serious" software. But this misses the point entirely. YouTube didn't replace Hollywood—it created entirely new categories of content and creators. Similarly, AI-assisted development won't replace traditional software engineering—it will enable new types of software and developers.

The Missing Piece: Infrastructure

The final piece of this transformation is deployment infrastructure. Companies like GitHub and Replit are racing to build platforms that make deploying and running applications as seamless as the development process itself.

The ideal platform needs to deliver:

1. One-click deployment that's truly free

2. Unlimited bandwidth with a fair revenue-sharing model

3. An approachable interface that hides complexity without limiting capability

GitHub Spark is tackling this with a fully managed runtime environment where deployment happens automatically as you create. Replit offers a similar vision but with more traditional development tools. Claude's artifacts showcase yet another approach through instantly runnable prototypes.

These platforms aren't just making deployment easier—they're reimagining what software distribution could look like in an AI-first world. Instead of separating development, deployment, and distribution into distinct phases, they're creating integrated environments where ideas can flow directly into running applications.

This infrastructure layer is crucial because it completes the democratization cycle. Just as YouTube needed both easy video creation tools AND seamless distribution to transform content creation, software needs both AI-assisted development AND friction-free deployment to enable truly personal software.

Beyond The Code

Furthermore, the cost of writing code dropping to zero doesn't eliminate all challenges. The true costs of software remain:

• Determining which problems are worth solving

• Managing complexity as applications grow

• Maintaining and improving systems over time

• Understanding user needs and feedback

This is why some enterprise software will persist. The dividing line isn't between AI-built and human-built software, but between commodity and strategic applications. Companies must ask: "Is this application core to our competitive advantage?"

The Path Forward

We're entering an era where software becomes truly personal again. Just as the PC revolution transformed computing from a specialized discipline to a universal tool, AI is transforming software development from a technical challenge to a creative endeavor.

This will reshape the industry:

• Development focuses on problem definition rather than implementation

• Enterprise software must compete on strategic value, not just features

• New categories of applications emerge that weren't feasible before

The barriers between users and creators are dissolving. We're moving toward a world where personalizing our software becomes as natural as customizing our desktop environment. This isn't just about making development faster or cheaper—it's about enabling everyone to create tools that precisely fit their needs.

The question isn't whether this transformation will happen, but what new possibilities it will unlock. As with YouTube, the most interesting innovations will likely come not from replicating existing software patterns, but from enabling entirely new categories of applications we haven't even imagined yet.

This week in AI feels like watching several games of chess being played simultaneously, but all the players keep switching sides. Google rolled out search grounding for Gemini, ChatGPT finally unveiled its web search, and Microsoft – in a particularly spicy move – announced it's bringing competing models to GitHub Copilot. Meanwhile, Anthropic quietly shipped desktop apps and voice features, while OpenAI started designing its own chips. What's going on here?

Let's unpack this tangled web of competition and cooperation, because it tells us something important about where the AI industry is headed.

Subscribe now

The Search Wars Heat Up

Remember when Google declared its "code red" over ChatGPT? Well, we're seeing the results of that scramble. Google's announcement of grounding with Google Search is a classic incumbent move – leveraging their moat in search to enhance their AI capabilities. It's a smart play that highlights Google's key advantage: fresh, authoritative web data at scale.

But OpenAI isn't sitting still. ChatGPT's new search capabilities, while still in their infancy, represent a direct challenge to Google's territory. The timing feels deliberate too – almost like OpenAI wanted to front-run Google's announcement. The real question isn't about who announced what first though; it's about who can build the most compelling fusion of search and AI.

The implications extend beyond just these tech giants. Every developer building a knowledge-intensive application – from legal research tools to medical databases – now has multiple options for integrating up-to-date search capabilities. The competition at the top is creating opportunities at every layer of the stack.

Desktop Dreams and Platform Plays

Anthropic joined the native app party this week, following ChatGPT and Perplexity's lead. They also added voice dictation, though it's more "speak-to-text" than ChatGPT's more sophisticated voice mode. These moves might seem minor, but they're part of a larger pattern: AI companies are racing to become your default digital interface.

Think about it – we're watching the early stages of a platform war. Each company is trying to become your go-to AI assistant, whether through web interfaces, desktop apps, or voice interactions. The stakes? Only the future of human-computer interaction.

But here's what's different from previous platform wars: the underlying AI capabilities are becoming increasingly commoditized and accessible. You don't need to be a tech giant to build sophisticated AI applications anymore. The real innovation is happening in the verticals – specialized tools built on top of these foundational models.

The Partnership Paradox

But the most fascinating developments are happening behind the scenes. Microsoft's decision to bring Claude and Gemini to GitHub Copilot is particularly telling. This is the same Microsoft that invested $10 billion in OpenAI. What gives?

The answer lies in a strategy as old as tech itself: commoditize your complements. Microsoft doesn't want to be dependent on any single AI provider, even one they've heavily invested in. They're creating optionality while ensuring no single player becomes too powerful.

Meanwhile, OpenAI's reported chip development efforts with Broadcom and TSMC show they're thinking along similar lines. In the AI world, today's partner could be tomorrow's competitor, and everyone's trying to control their destiny by securing key infrastructure.

This dynamic is creating a fascinating opportunity for startups and developers. As the giants battle it out, they're inadvertently creating a more competitive market for AI capabilities. Need a language model? You've got options. Want to integrate search? Take your pick. Building the next great AI application is becoming less about having privileged access to foundational technology and more about understanding your specific domain and users.

What This Means for the Future

We're witnessing the AI industry's awkward teenage phase – lots of growth, plenty of drama, and constant identity crises. The partnerships and rivalries forming today will shape the AI landscape for years to come.

For developers and users, this competitive chaos is actually good news. More competition means more innovation, better products, and hopefully, lower prices. The trick will be navigating the increasingly complex ecosystem of tools and platforms. Smart developers are already taking advantage of this, building specialized applications that leverage multiple providers rather than betting everything on a single platform.

Subscribe now

One thing's certain: in the AI industry, the only constant is change. Today's alliances are tomorrow's rivalries, and everyone's playing the long game. Keep your friends close, your enemies closer, and your API keys for multiple providers handy. The real winners in this new era? The builders who can navigate this shifting landscape while staying focused on solving real problems for real users.

Imagine if AI could use your computer just like you do - clicking buttons, filling forms, and navigating websites. That's exactly what Anthropic is enabling with 'computer use', announced this week alongside updates to their Claude models. While I've tested the new flagship "Claude 3.5 Sonnet (new)" and found it even more natural-sounding in conversation, it's computer use that represents the most significant advancement. By allowing Claude to understand and interact with existing computer GUIs, this feature fundamentally transforms how AI can interact with software.

This advancement turns foundation models into general purpose software automation platforms. Rather than developing software to fit models in the form of bespoke "tools" (as has been the case for over a year), the model itself has some understanding of existing software. Anthropic aptly describes this as "... the model fit the tools" as opposed to "... made tools fit the model". This is analogous to how LLMs themselves are a general purpose tool for what previously had been several models in NLP, computer vision, etc. As a consequence of this generality, this opens up many possibilities for agents. With computer use, foundation models are able to drive legacy software, rather than requiring programmers write custom integrations. Put another way: existing LLMs supercharges writing new software because they are very good at coding; computer use supercharges using LLMs with the iceberg that is legacy software. It's worth noting that this holy grail of automation has been tackled from different angles over the years; RPA platforms like UiPath have been automating legacy enterprise software for years, and AI-native entrants like Adept have been tackling the same problems.

What makes Anthropic's release interesting is the generality of the solution: the same model that processes human language, computer code, and images can now drive GUIs. Foundation models like Claude are having their iPhone moment - just as Jobs famously introduced what seemed like three devices but was actually one revolutionary product in 2007, LLMs are consolidating multiple AI capabilities into a single, general-purpose system. The iPhone was revolutionary in large part because it aggregated disparate demand for computing in a single, portable device. LLMs have exhibited a similar march over the last 2 years. To start, LLMs aggregated what were previously separate models in NLP including sentiment analysis,  named entity recognition, amongst others. Then, LLMs became multi-modal, with support for images and audio. With computer use, these models gain yet another capability that previously required separate models.

Anthropic notes that computer use is in beta, and has a number of limitations. To get a sense of what's possible I decided to try it out with a simple example. I ran Anthropic's sandbox for computer use, which basically just set up an environment with a Linux desktop, and a browser. I asked Claude to help me get a refund for a fictional Delta flight. Earlier this year I had to go through the process, and while not entirely onerous I found it tedious.

The process entailed 1) performing a web search 2) determining which case I fell under, and 3) filling out a form in a modal with flight and transaction details. The 2nd step is usually the most complex, as it involves a fair amount of reasoning to determine if a refund is even applicable, and if so which form needs to be filled out.

I also liked this example because it's a mini-version of what computer use promises: integrating with legacy software that does not have clean, easy-to-use APIs. Delta's forms were alright but even on a desktop computer, the process can be difficult to navigate.

I had to run the same process a few times because the results were sporadic:

• The first time I ran it, Claude correctly identified what needed to be done. When I asked it to fill out the form, it refused, citing that it cannot work with PII. This is a known limitation of computer use, which Anthropic has documented.

• The 2nd time, Claude concluded that nothing needed to be done, which I don't believe is correct under all circumstances. I think this is a problem with missing context; I provided a very broad prompt, but it would be more helpful for Claude to have more context around my (fictional) situation.

• The 3rd time around, Claude hallucinated a URL but recovered from it. It decided to summarize what manual steps I needed to take in order to get a refund, instead of executing. Again, this is likely due to prompt underspecification.

Key Takeaways from Testing Computer Use:

• Computer use is slow and costly; each run cost me about $1 and my issue wasn't fully resolved. As noted above, with better prompting a full resolution probably would have been reached, but it probably would have been 2x as costly. This implies that developers need to be judicious with the use-cases they apply this new capability to, while the technology and costs improve.

• Anthopic's constraints on passing sensitive data makes sense, but is very limiting. One of the biggest promises of AI agents is the ability for them to act on your behalf, and that involves PII. Human assistants have access to and utilize privileged information all the time, and so I hope there is some evolution on this front.

• I used Anthropic's ready-to-run computer use sandbox, but this is not necessary. Developers can actually control the agent's loop, which opens up a lot of possibilities, like different kinds of input control. Since developers also control the execution of GUI actions, this also allows for different isolation/sandboxing schemes, and application-specific error-handling; basically you're free to implement whatever agent architecture you want.

When I chatted with Claude about setting up the sandbox, I was amused by this very natural response in the new 3.5 Sonnet model:

My initial impression is that Anthropic has trained a model that is vastly superior in vision capabilities, with specific capabilities as it relates to GUIs.  Notably, Anthropic mentions that this model was trained to be able to count individual pixels.  One outcome from this is that Claude may have very good vision capabilities that apply to other domains. For example, it looks like Claude can identify plane models with very few cues. On the agentic front, my initial impression is that computer use, is very useful for interpreting GUIs but it might be better to use other tools for other parts of a workflow. For example, I wouldn't waste time asking Claude to perform a web search. Instead, I would provide screenshots of web pages that have already been filtered for relevance. Similarly, for web applications specifically, I might use Claude to "direct" traditional automation software like Selenium or Playwright, rather than having Claude execute everything itself.

Subscribe now

All that being said, I've been impressed by what computer use has to offer on day one. While current limitations around speed, cost, and handling sensitive data need to be addressed, the potential is clear: we're moving toward a future where AI can seamlessly interact with existing software interfaces, potentially revolutionizing how we think about automation. The evolution of this technology will be fascinating to watch.

An esoteric use-case for which I find myself increasingly turning to LLMs is as a thought partner. I don't mean using AI as a therapist or as a girlfriend, like some other prevalent uses. Instead, I'm leveraging LLMs to help me think through complex decision-making and planning processes. While this use-case might seem odd at first, I believe it will become mainstream in due time. Surprisingly, this application is under-discussed in the broader AI community.

So what do I mean by a thought partner? I use AI (primarily Claude) to answer open-ended questions that help me think through both technical and non-technical problems. Here's a recent example: I was contemplating whether to self-host my media collection or simply pay for a higher storage iCloud tier. I initially consulted Claude because I suspected my decision-making wasn't entirely rational; I have a strong bias towards DIY and self-hosting. However, after an in-depth conversation with Claude, I had a complete change of heart. Claude even created a simple breakeven analysis of my situation:

Despite how productive this conversation was, I still find current products lacking for true thought-partnership. To start, you must explicitly prompt the LLM to consider opposing points of view. I believe a great thought partner illuminates gaps in your thinking, and as such, healthy debate is necessary to reach better ideas. A good thought partner also challenges your thinking and asks you to specify your assumptions. LLMs are often too agreeable, making explicit prompting for debate necessary. For example, I prompted Claude as follows:

I generally agree with this assessment, but give me an in-depth bull case for self-hosting and an in-depth bear case for the cloud.

This type of prompting raises another issue: effective brainstorming is usually a highly non-linear process. The linear chat UX can be frustrating because what I really want is a branching conversation, somewhat like brainstorming on Miro. Branching would also help manage LLM context more effectively, as LLMs tend to forget and get confused during long conversations (not unlike humans 😅). Branching enables sending only the necessary context for a given conversation thread. For example, in my case, the "bull case" and "bear case" would function better as separate branches rather than a single command. In theory, this also allows the model more "time to think."

Branching is crucial for divergent thinking in a brainstorming session. However, great brainstorming sessions also feature convergent thinking, where the ideas discussed are synthesized and unified. To this end, I would love for a thought partner AI to summarize key points at different "checkpoints" in a conversation. These "checkpoints" are fairly arbitrary and abstract because they depend on the specific conversation. To draw an analogy, in a human-led brainstorming session, someone is likely jotting down notes of key points raised and discussed. This is critical, as much of brainstorming involves productive meandering; a lot of the conversation content may be irrelevant to the final outcome but is a necessary part of the process.

A more advanced feature for a thought partner AI that combines both convergent and divergent thinking would be comparing the results of two different models. This is akin to asking your personal board of directors a question and comparing individuals' responses. Currently, I do this manually by providing both ChatGPT and Claude the same prompt. I then compare their answers to determine areas of agreement, disagreement, and which assistant exhibited the best reasoning. Due to the subpar UX for this type of conversation, I often short-circuit the process by continuing the conversation with the most thoughtful assistant.

These are just some initial ideas on this emerging use-case. I'll continue to refine this concept, but I'd love to hear if others are using LLM assistants in this way and if these ideas resonate with them.

Subscribe now

Introduction

I'm thrilled to kick off this series with a service I've grown to love over the last year: Neon. I've recommended it to several people, and today I'll detail why it's so awesome. While my enthusiasm might sound like a paid endorsement, I assure you I'm simply a satisfied paying customer. 😅

What is it?

Neon is a modern, cloud-based database service that offers a serverless version of PostgreSQL, one of the most popular open-source relational database systems. Neon itself is also open-source. Neon is like having your favorite high-end oven (PostgreSQL) in a cloud kitchen. You get all the power and familiarity of Postgres, but without the hassle of maintaining the hardware or infrastructure – just show up and start cooking with your data. Let's explore how this analogy plays out in Neon's unique features and benefits.

What are the benefits?

For the business

Autoscaling to the rescue

Neon is a serverless cloud offering, so the immediate benefit to businesses is that it allows usage-based billing that is "right-sized" to your workload. In practice, rather than paying for a static database server, a business pays only for the compute and storage a Neon database utilizes. Neon's core innovation is separating compute and storage1, and billing for them separately. Compute is what is used at query-time to retrieve data, while storage simply represents all the data in the database and any historical snapshots (more on this later). Separate compute and storage, and separate their separate billing enables businesses to buy "just enough" database rather than under-provisioning to save money, or over-provisioning to mitigate traffic spikes.

Autoscaling saves you more than money

Neon's serverless core offers more than just potential cost savings; it also helps teams maintain operational excellence. As mentioned earlier, Neon can scale up and down based on traffic. This means your product and site can handle sudden traffic spikes, like after being featured in a popular publication. Scalability like this is made all that much sweeter because it's dead simple to configure. The following is the way I've configured my Neon instance (though there are other ways as well).

Because of configuration like this you can also be sure about what your minimum and maximum spend will be for the month; this is a subtle but important distinction, since many teams are frustrated that a lot cloud services do not afford pricing transparency. Autoscaling without cost transparency is like driving a car without side mirrors or seatbelts - risky and potentially disastrous.

Neon helps you to better serve your customers

Neon's scaling benefits extend well beyond a single database instance or application; provisioning new databases can be done programmatically, and its provisioning is fast (a few seconds vs a few minutes in traditional managed database services). This opens up possibilities for businesses, like a multi-tenant architecture where every customer has their own database. Having worked in B2B Saas for over 10 years, a capability like this is music to my ears because oftentimes multi-tenancy is a hard requirement for customers. Furthermore, the autoscaling makes such an architecture cost-effective. For example, your biggest customer can get a beefy database instance that rarely or never suspends, while your lower tier customers can use cheaper autoscaling instances that scale to 0 when not in use.

An insurance policy

Neon supports a killer feature that I consider a huge insurance policy against incidents and anomalies: time travel. Specifically, Neon allows developers to issue queries against a specific point-in-time or database transaction. The importance of this cannot be understated. Imagine if you have a new developer starting with your team and they mistakenly issue a disastrous query that deletes a bunch of customer data. Not only can you create query before the incident to ascertain the damage, you can also restore the database entirely. It's worth noting that PITR (point-in-time-recovery) is a fairly standard feature amongst managed database services. What makes Neon's offering unique are speed and the developer experience. In Neon performing a PITR is painless and fast. Neon also offers a time travel query editor, which is a unique offering I have not seen elsewhere. The retention window for history is also configurable, up to 30 days, depending on tier. To be clear: this is an additional safeguard and is not a replacement for backups. It's a pretty sweet safeguard though, because it's impactful and easy to use. Finally, I'd be remised if I didn't mention that time travel can be used as a poor man's data warehouse since an engineer or analyst can issue historical queries. Similarly, this is not a replacement for a proper data warehouse, but it can be fairly effective in a pinch, especially for a small team.

Closing

In essence, Neon's business benefits extend far beyond cost savings. It's about empowering your team to focus on innovation rather than infrastructure management, providing peace of mind with robust safeguards, and offering the flexibility to adapt to your business's unique needs. Whether you're a startup or an established enterprise, Neon's serverless PostgreSQL can be a game-changer for your database strategy.

Subscribe now

For developers

We've covered how Neon can help drive business outcomes, but what about the developer experience? After all, while the business may benefit from a good strategic decision, it's ultimately developers that will be responsible for maintaining and leveraging a service like Neon.

The Neon team ❤️ Postgres

Here's a hot take: it's crucial that the people building in a technology ecosystem genuinely care about both the ecosystem and the technology itself. The reason is simple: if they're not nerding out about the tech, how can you trust they're putting out the best work? How can you trust you won't eventually be hampered by their decisions? I don't know much about the Neon team itself, but in this respect their actions speak volumes. First, Neon is completely Postgres-compatible, so they're not fighting to replace the venerable database. As mentioned earlier, Neon itself is open source.

The Neon team's ethos is important, but delivering value to customers and developers matters too. And here they excel as well. On launch day, Neon supported the latest and greatest: Postgres 17. That's an impressive turnaround, and I can tell you from experience this is not the norm for managed database services. This is an important indicator in my view because it means that as an engineering team, you can leverage the latest and greatest as soon as possible. Postgres is rock-solid in terms of stability, and so the ability to leverage any new developer features is huge. In the past, I've found new Postgres features are able to improve significant portions of an application almost overnight. New features also typically open up new use-cases. One notable improvement  in Postgres 17 specifically, is the updated MERGE command. This updated command can dramatically improve upsert queries, and move them away from the application, which in turn improves overall stability.

Neon also embraces the broader Postgres ecosystem, so developers are fully empowered to innovate. It supports major open source extensions, including Postgis for geospatial applications, timescaledb for time-series data, amongst others. On the AI front, Neon has stellar support for pgvector, which can serve as a very good vector database. Similar to Postgres major versions, I've observed the Neon team supports new versions of extensions very quickly; this is especially important with regards to pgvector, which is evolving rapidly as teams build out AI applications.

A delightful developer experience

While Neon's robust Postgres support is great, it's the exceptional developer experience that truly sets it apart.

I've discussed history in the context of business continuity but it's also an invaluable developer tool. Neon makes it so frictionless to use historical data that it's very practical to use it to troubleshoot bugs. Imagine this scenario: a bizarre bug crops up at 1:31 AM, and replicating your system's state to reproduce it seems impossible. With Neon, it's a breeze: simply branch your production data as of 1:31 AM and use that new branch in your local development environment. Oh, I forgot to mention, Neon has branching for your data! This is what makes using history so seamless. Basically, like git, different "versions" of a database instance can exist as different branches. Branches can be forked and cloned, and these operations are fast. This extends beyond debugging too. If you're developing a new feature and want realistic data, what's better than just forking production? Each branch gets its own unique endpoint that can be referenced wherever you like. After a branch is forked, the branch histories diverge, just like git; changes in a base branch do not affect the base branch. The ability to treat data like code is such a superpower, I can't overstate its importance.

This feature's potential goes even further. Neon has a Github actions integration that allows you to create branches on a PR-basis. If you have a preview environment capability for your application (which most PaaS support), this effectively means you have a complete replica of your production environment. Branching won't be feasible for all teams out-of-the-box. In particular, some teams may need to scrub sensitive customer data before allowing access to developers (eg financial services, health tech). However, teams can address this with 3rd party vendors that specialize in data scrubbing, or automate it themselves.

The cherry on top of Neon's developer tooling is its comprehensive and well-documented API. If your team has specialized needs, you have the power to implement them. All the aforementioned features, like branching, creating databases, and editing scaling parameters are scriptable via the API.

The final notable developer tooling feature that I want to cover is endpoints. I've mentioned how branches each get their own endpoint, but this is not limited to branches. Each Postgres database (not to be confused with the Neon database instance) gets its own endpoint. Each endpoint can then be separately configured with its own scaling parameters. This combination is powerful because it allows for more flexible architecture choices. For example, if separate database instances is not your preferred architecture for multi-tenancy, you can simply create separate databases within a single instance, each with it's own endpoint (and by extension its own scaling). Similarly, Neon also has a native read replication functionality, which leverages endpoints. So for example, a complex analytics query won't take down your production endpoint, and in fact your read replica can be scaled independently from your production configuration.

One small thing I recommend developers look out for: make sure your database connections are correctly configured in whatever framework or library you're using. I spent some time debugging my Neon instance because it would scale to zero (as I configured), but my backend then tried connecting to a closed connection; my backend automatically caches database connections.

This is trivial to address, but it's not something I've had to do coming from non-serverless databases. In Django, I configured CONN_HEALTH_CHECKS so that Django's ORM first checks the status of the connection before attempting to use it. Similarly, in Sqlalchemy, I configured pool_pre_ping, which works much the same way as Django's setting.

Closing

As you can see, Neon has a complete toolbelt to make developers' lives easier and more productive. From lightning-fast provisioning to intuitive branching and historical querying, Neon has reimagined what a cloud database service can offer. It's not just about storing data; it's about empowering developers to work smarter, faster, and with greater confidence. In a world where developer productivity is paramount, Neon stands out as a tool that truly understands and enhances the developer experience.

Low Barrier, High Leverage Assessment

Now, let's summarize Neon's offering and rate it as a low-barrier, high-leverage solution. I also discuss some areas where I think it could be improved.

Barrier Lowerers

• Neon is open source and Postgres compatible, so getting started is a breeze.

• Autoscaling reduces the amount of decision-making teams need to make around workloads.

• Neon's branching feature allows for easy testing and development without affecting production data, reducing the barrier to experimentation.

• The comprehensive API enables teams to integrate Neon seamlessly into their existing workflows and automation pipelines.

Leverage Lifters

• Neon's developers tools are top-notch.

• Neon's support for the latest Postgres versions and extensions ensures your team can leverage cutting-edge database features as they become available.

• The ability to create separate databases with individual endpoints allows for flexible architectural choices that can evolve with your business needs.

• As your team and business grow, Neon's scalability and branching features support increasingly complex development workflows and multi-tenancy requirements without requiring a fundamental change in your database strategy.

Areas for Improvement

• I didn't cover this earlier, but Neon has connection pooling support. However, it requires using a separate endpoint. Connection pooling is generally considered a best practice, and I would love if this were the default out of the box, and didn't require separate configuration.

• I love Neon's history feature because it serves as both an insurance policy and as a developer power-up. I wish history could be configured completely independently of the tiering. Currently, the first paid tier tops out at 7 days of retention, and the next tier up is good for 30 days. I would like to pay for this independently of the other features, but I understand why this tiering makes sense from a customer segmentation perspective.

• Neon currently supports instances in AWS, and Azure support just launched in beta. I think it's inevitable but I look forward to having Neon on all major clouds, so it's a no-brainer for any team. I know of at least one team that cannot use it because they are using GCP. Technically, anyone can use the existing instances, but using a database in a different cloud is a suboptimal.

Scoring

I'm scoring Neon a 9/10 for having a low barrier to entry, and a 10/10 for the leverage it affords teams, for a combined score of 9.5/10.

I subtracted a point for lowering the barrier to entry because of the areas where Neon can improve. Neon is a startup, so I expect these minor shortcomings to be addressed in time (with the exception being paying for the history feature independently, which is more of a pricing strategy complaint).

Conclusion

I've been itching to dive deep into Neon for a while now, and for good reason. It's a rare gem in the database world that combines the robustness of PostgreSQL with cloud-native innovations like autoscaling, branching, and time travel. From its business benefits of cost-efficiency and operational excellence to its developer-friendly features that boost productivity, Neon is reshaping how I think about and interact with databases. I've recommended it to numerous teams, and the feedback has been consistently positive. Whether you're a startup looking for a scalable solution or an enterprise seeking to modernize your database infrastructure, Neon deserves a serious look. It's not just a database service; it's a catalyst for better, faster, and more innovative software development.

Remember Intel's famous tick-tock model? Big changes ("tocks") followed by refinements ("ticks"). While "ticks" seemed minor, they drove rapid progress. I see OpenAI's DevDay as their "tick" – a slew of developer-facing changes that will ripple through applications in the coming months.

Here's my quick recap of the announcements and initial impressions.

Realtime API

OpenAI's new Realtime API is a game-changer. It lets developers create their own versions of OpenAI's Advanced Voice Mode, enabling low-latency voice interactions. The possibilities are vast – from customer support to AI tutors. But what caught my eye was the pricing:

This equates to approximately $0.06 per minute of audio input and $0.24 per minute of audio output.

Let's crunch the numbers: at ~$15 per hour (based on output pricing), it's already competitive with many call center roles. Having tinkered with advanced voice mode, I can attest to its impressive capabilities. The Realtime API's support for function-calling is the cherry on top – it allows for dynamic information retrieval and action-taking based on conversation flow. We're one step closer to building assistant experiences straight out of Her.

Prompt Caching

OpenAI has joined the prompt caching party, following Anthropic and Google's Gemini. Their implementation stands out for its automatic nature – no explicit requests needed. It's an Apple-esque move, prioritizing simplicity for developers. However, the cost savings (50%) pale in comparison to Anthropic's "up to ~90%", and latency improvements remain unclear. Let's break it down:

• OpenAI: Automatic, 50% cost savings, unspecified latency improvements

• Anthropic: Explicit requests, variable savings (50-90%)

• Google: Most flexible, with configurable TTL for cached prompts

While I'd love to see configurable TTL across the board, flexilibity is tricky to balance with simplicty.

I expect the proliferation of prompt caching to have substantial but subtle downstream effects:

1. UX is going to improve

2. They'll be a wider proliferation of "chat with data" use cases, and AI more broadly.

Prompt caching is set to revolutionize AI app UX through reduced latency. Picture this: an entire book cached, ready for instant Q&A. While OpenAI's specific latency improvements are yet to be quantified, this addresses a key pain point in AI-native apps.

The cost reduction from prompt caching opens doors for more "chat with data" features across applications. While RAG with dynamic data might not benefit, simpler implementations – where document corpora are injected into prompts – are ripe for caching. As context windows expand, it's becoming increasingly cost-effective to load extensive context into a single prompt.

Prompt caching isn't just a minor upgrade – it's a quality-of-life improvement that could dramatically expand the horizons of what developers can create.

Model Distillation

Model distillation is a technique in machine learning where knowledge from a larger, more complex model (often called the 'teacher' model) is transferred to a smaller, more efficient model (the 'student' model). This is typically done by training the student model to mimic the outputs or intermediate representations of the teacher model, rather than training directly on the original dataset. Think of it as a master chef (the larger model) teaching a novice cook (the smaller model) to prepare gourmet dishes by demonstrating techniques and sharing insights, rather than just handing over a recipe book.

While developers have been distilling models for a while, OpenAI's latest announcement is a game-changer. It streamlines the process by integrating multiple steps – including automatic prompt saving and evals for fine-tuning smaller models – directly into their platform. This integration isn't just convenient; it's a catalyst for rapid iteration and experimentation.

This addition implies a few things:

1. OpenAI is continuing to differentiate itself as not just a model provider, but a developer platform.

2. As a consequence of 1, many AI infra startups are in trouble. There's still use-cases that necessitate this tooling outside of model providers, but the bar has been raised.

As LLMs become commodities, OpenAI is pivoting towards becoming a full-fledged developer platform. This strategy echoes the early 2010s cloud evolution:

• OpenAI's journey:

  • 2023: Launched Assistants API, offering higher-level abstractions

  • Now: Integrating model distillation and fine-tuning

• Parallel with AWS's evolution:

  • Started with basic services (blob storage, virtual servers)

  • Moved "up the stack" with managed services like RDS

The rationale? Just as cloud services became commoditized in the 2010s, LLMs are following suit in the 2020s. OpenAI's platform approach not only differentiates their offering but also increases stickiness and workload share. The Assistants API, while not as flexible as custom solutions, opens the LLM playground to a broader developer base.

OpenAI's developer platform endangers AI infrastructure startups that are superficial wrappers on top of the model providers. It's the classic case of bundling and defaults. Why would a developer or organization choose an external vendor for fine-tuning and distillation, when they get it all for "free" with OpenAI?

I don't think the cost-benefit analysis is universally in OpenAI's favor, but the inclusion of this suite of features raises the bar for startups and the completeness of their offerings. One obvious reason developers will continue to use external vendors for this kind of work is to avoid vendor lock-in, and have uniform tooling for their portfolio of models, since advanced AI applications typically use more than one model provider. 

There's yet another cloud analogy: as multiple cloud providers rose to prominence, and the vision of the cloud came into fruition, tools like terraform enabled teams to seamlessly manage multi-cloud deployments. However, tools like terraform made a name for themselves by creating substantial value beyond simply wrapping the cloud providers. I expect something similar will happen with AI infrastructure startups: the best ones will deliver on a vision that's so compelling, you'll use them because you want to, not just to avoid vendor lock-in.

Vision Fine-Tuning

OpenAI also announced it's now possible to fine-tune GPT-4o for both text and vision. My excitement for vision fine-tuning is second only to the Realtime API because it will enable substantially better performance on vision-related tasks. OpenAI's models (and other leading LLMs) were already remarkably good at computer vision tasks, and it didn't require training with a custom dataset. With vision fine-tuning, developers now have a clear path for improving performance of these tasks.

Some use-cases I suspect will take off:

• Better structured extraction from documents.

• Automation of (currently) manual data labeling tasks, which facilitates training smaller, more efficient models.

• UI automation (browser + desktop) because the model can be trained on specific sites + apps.

I think vision fine-tuning will essentially "unlock" vision use-cases, the same way GPT-4 was enough of a leap from GPT3.5 that it enabled a litany of new applications. And what's even more remarkable is that OpenAI suggests the performance of many tasks can improve substantially with as little as 100 samples!

Conclusion

Phew! We've covered a lot, but there's even more: free limited-time fine-tuning (clever customer acquisition!), the summer's cheaper model becoming the default, and a slew of new developer tools in the OpenAI playground.

Can you feel my excitement? These updates are transformative. The Realtime API and vision fine-tuning pave the way for agents with voice, hearing, and sight. As you scale, model distillation and prompt caching offer powerful optimization tools. Buckle up – we're in for an exhilarating season of AI innovation 🤓.

I've been exploring OpenAI's o1 over the last couple of weeks, particularly its potential for reasoning. o1 takes a variable amount of time to "think" through problems, making it well-suited for complex reasoning tasks. In my last post, I delved into how this represents system 2 thinking - deliberative and time-consuming.

One consequence of this idea, is enabling more interesting architectures for agents. I also covered agents in my last post, but to recap:

An AI agent is software that can perform tasks or make decisions on behalf of a user, similar to how a person might delegate responsibilities to a trusted representative.

Now, let's explore what a software architecture might look like with o1 at an agent's core. Buckle up!

Components

In brief, my proposed AI agent architecture would consist of the following components:

• A preamble prompt that summarizes the agent's purpose, the user's goals, and relevant interactions.

• A reasoning engine, which would be built around a model like o1.

• Tools/functions for task-specific functionality.

• A customer experience model, whose entire purpose is to translate the agent's thoughts, actions and outputs to the user in an intelligible and personable way.

• General purpose memory that the agent, tools and other components can use to query relevant information.

Let's double click into each component to understand how they each fit into the agent.

Preamble Prompt

An agent frequently needs to recall several key pieces of information for almost any user interaction: its purpose, the user's details, and a summary of past interactions.

Let's start with purpose: role-playing is a powerful prompting technique crucial for an agent to understand its role. Without it, an agent defaults to a generic personality dependent on the model provider (e.g., generic ChatGPT for OpenAI's APIs). Technically, role-playing serves as contextual priming for LLMs, activating specific knowledge domains and behavioral patterns. By specifying a role, we narrow the model's vast knowledge base to a specific context, potentially improving response efficiency and accuracy.

Next, let's consider the user. At a basic level, an agent needs to know a user's name and other useful identifiers (e.g., IDs). Depending on the application, this could extend to include a comprehensive user profile.

The final part of the preamble prompt is more fluid: a summary of interactions. Think of this as short-term working memory. Just as you might not remember every lunch with a colleague but recall the recent ones, this summary is application-specific and requires experimentation. While not strictly necessary, it's worth considering what frequently-accessed data could benefit the agent.

This information is a preamble because it should precede any prompt, leveraging the model's tendency to pay closest attention to the beginning and end of data.

At the end of the day, the preamble prompt is just a dynamically generated text string; I'm not proposing something very sophisticated. However, it is useful because you'll likely need the same basic piece of context over and over again. Rather than having all components re-implement this time and time again, it pays dividends to design this upfront.

Reasoning Engine

Now let's circle back to my recent obsession:  o1 as a reasoning engine. I conceptualize a reasoning engine as a model capable of routing requests to appropriate tools with appropriate parameters. Simple enough? Well, in practice this is not so. I've previously leveraged function/tool calling in OpenAI's models to serve this purpose. However, I've found performance to be lacking, mainly because calling the right tools at the right time requires a fair amount of reasoning over context.

Let me illustrate with a real-world example. Recently, while ordering chicken and waffles for breakfast, I considered adding lemonade. I inquired about it, and the cashier added it to my order without my knowledge. To correct this, I said, "I don't want both... just the chicken and waffles." Surprisingly, this was misinterpreted as keeping only the lemonade (🤦🏽).

This mixup was the result of a few things:

1. Long-lived context. The interaction included several messages, and meaning of later partial messages was influenced by earlier messages.

2. An understanding of preference. My hesitancy around adding the lemonade should have provided context around what I wanted to keep in the order.

While o1 isn't quite ready to man a cash register at your local eatery, I believe its reasoning capabilities bring us much closer to "getting it right" in multi-step and ambiguous situations.

Note: o1 currently lacks native function/tool calling, but as the model evolves this capability is inevitable. Furthermore, function/tool calling is not strictly necessary and is more often than not a developer convenience (a much welcomed one at that!).

TLDR: a model like o1 serves as both the brain and air traffic control for an agent.

Tools

An agent isn't very interesting if it can't do stuff. That's where I imagine tools coming in. Tools serve as a very broad label for subroutines that do stuff. A tool can be as simple as a function call, or as a complex as a separate RAG pipeline that has several inference steps. It's a very broad spectrum. I'm excited to see that tool interfaces are standardizing around JSON schema specifications, with OpenAI and Anthropic, the leading model providers essentially using the API. Standardization like this will make it easier to share best practices, and hit the ground running for AI application developers. Standardization also makes switching providers easier, what's not to love?

When designing tools I think about a few characteristics. First, tools should follow the Unix Philosophy, namely:

Make each program do one thing well.

AI app development is challenging enough, and tools offer a great way to manage overall complexity. Moreover, I believe tools are the most impactful component for introducing determinism into an agent. LLMs are inherently probabilistic, flipping traditional programming on its head - the same input can lead to different outputs, unlike traditional software. Tools provide an opportunity to inject more determinism through entirely deterministic code. However, this isn't guaranteed, as tools can also call LLMs and other models for additional inference.

Finally, tools are yet another opportunity to imbue agents with system 2, deliberative thinking. When leveraging LLMs in tools, it's advantageous to break up complex steps into multi-step RAG pipelines, essentially affording any underlying models more time to "think". For example, suppose you wanted to use an LLM model with vision capabilities to provide users an estimate of macronutrient content in a picture of a meal (ahem ). Instead of performing this as a single step, such a tool could be decomposed into the following:

1. Determine whether the picture consists of food in the first place

2. Identify the foods in the picture

3. Identify the portion sizes in the picture

4. For each food and portion size estimate, estimate macronutrient content

5. Combine the results for the entire meal

By breaking the task down, the model essentially has more time to "think", à la o1, which leads to better quality results.

The design space for tools is very big, and tools are a pillar of this proposed architecture. As such, I believe this component has the most room for creative software design and engineering.

CX model

So far, we've covered fairly technical, behind-the-scenes aspects of an overall agent architecture. Most agents will need to communicate with humans, whether internal or external to an organization. Human communication differs significantly from inter-component communication, even when both use natural language. For instance, communication between the food logging tool and the reasoning engine might be programmed as: "A sandwich with 40g carbs, 20g protein and 10g fat was logged on behalf of the user". This terse, impersonal message suffices for software components that understand human language, like the reasoning engine.

When communicating with a person, the requirements are likely to be different:

1. You likely want to ensure the response matches your brand voice. You do not want to sound like a generic ChatGPT wrapper.

2. You likely want the model to be more affable. While people appreciate brevity, they also appreciate feeling heard.

3. On a technical level, you may want to have a model that responds fast. After inference is complete, you likely do not want the response to become a bottleneck. Relatedly, this is an opportunity for cost-optimization: this model can be a smaller fine-tuned variant of a simpler model. Think: fine-tuned gpt-4o-mini rather than gpt-4o.

While I believe a fine-tuned small model is best,  I also think separating this into a separate component allows developers to leverage models that are better communicators out-of-the-box. Personally, I enjoy and prefer using Claude for writing, and would consider using it here if a fine-tuned model is not initially feasible.

For now, I'm calling this the "customer experience model", but I'm open to more catchy names, especially since it's not exclusive to customers - any human interaction could benefit from such a model. Got any ideas? I'm all ears!

Memory

This is where things get a bit hazy (pun intended!). While memory is intuitively core to any AI application or agent, it's currently the most ill-defined and least understood component. A slew of startups are attempting to build this as a separate service, with varying degrees of success.

In my experience, a good memory component would accomplish the following:

1. Allow "querying" for disparate facts. Querying can be highly structured (think: SQL) or unstructured (eg "what important things do I know about this user?"). Folks are using knowledge graphs to bridge the gap between structured and unstructured data. It remains to be seen if this is a viable long-term solution.

2. It should be usable by all components, as presumably all components benefit from contextual understanding of what's going on.

As alluded to in "querying", this component would likely be backed by several databases:

• Potentially a knowledge graph, like Neo4j,  for more flexible querying of facts.

  • Example Query: "Who are known associates of this user?"

• A relational database like PostgreSQL for highly structured querying.

  • Example Query: "When was the last time this user logged in?"

• A vector database for semantic searches

  • Example Query: "The user asked about the health of their product line, retrieve all documents related to sales and revenue."

Memory is also the subject of a lot of interesting research, some of which at least tries to emulate human memory. Most notably, Generative Agents: Interactive Simulacra of Human Behavior discusses a novel architecture that compresses memory and adds a time-decay to them. In brief:

Generative agents are AI characters that can simulate believable human behavior in interactive environments. The key to their abilities is a novel memory architecture that stores experiences as natural language, retrieves relevant memories, and synthesizes them into higher-level reflections to guide behavior. This allows the agents to form relationships, spread information, and coordinate activities in emergent ways. While not perfect, the architecture enables more coherent and context-aware behavior compared to previous approaches, opening up new possibilities for interactive AI characters in games, simulations, and other applications.

Memory is a domain I'm watching closely, as it holds the potential to be just as transformative for AI agents as advanced reasoning capabilities.

Putting it all together

To summarize, this is the proposed architecture:

Wrapping Up: Where Do We Go From Here?

Let's bring this home. We've just taken a whirlwind tour of a potential AI agent architecture, from preamble prompts to reasoning engines and everything in between. But we're still feeling our way through the dark. This isn't the definitive blueprint for AI agent design - far from it. We're in the "throw spaghetti at the wall and see what sticks" phase, and that's thrilling.

This reminds me of the wild west days of web development. Remember when we were all writing PHP spaghetti code, mixing business logic and presentation with reckless abandon? Then frameworks like Rails and Django swooped in, making MVC the new standard. They didn't invent MVC, but they certainly democratized it.

I can't help but wonder if we're on the brink of a similar watershed moment for AI agents. Perhaps this architecture, or something like it, will bring some order to the chaos. Or maybe it'll be something entirely different that we haven't even conceived yet. That's the thrill of innovation, isn't it?

One thing's certain: the AI landscape is evolving at breakneck speed. Today's cutting-edge could be tomorrow's old news. So let's stay flexible, keep experimenting, keep building, and above all, keep sharing our insights.

Now, I'm eager to hear your thoughts. Have you ventured into building AI agents? What triumphs or spectacular failures have you encountered? Does this architecture seem promising, or am I way off base? Share your insights in the comments - let's spark a discussion. After all, collective wisdom is how we'll navigate this new frontier.

Last week, I discussed my initial impressions of OpenAI's new model family: o1. I deliberately used a toy example to put o1 through its paces, knowing full well that it might not be satisfactory for all use cases. This week, I'm diving into what I believe this enables: agents.

OpenAI's o1 and o1-mini models represent a potential paradigm shift in AI capabilities, bridging the gap between reactive language models and truly agentic AI systems. By providing superior reasoning and planning abilities, these models could catalyze the development of autonomous AI assistants capable of handling complex, multi-step tasks.

Quick Update: Improved Usage Limits

First, a quick update: the usage limits on o1 and o1-mini have been substantially improved for paying users. This is a great benefit as everyone learns to leverage these models.

The increased limit for o1-mini is particularly impressive, essentially enabling usage that's on par with normal models.

Speaking of o1-mini, I suspect it will be a sleeper hit. It's substantially cheaper but performs similarly to full-blown o1:

Source

However, it's worth noting that o1-mini has a specific focus:

Due to its specialization on STEM reasoning capabilities, o1-mini’s factual knowledge on non-STEM topics such as dates, biographies, and trivia is comparable to small LLMs such as GPT-4o mini.

It remains to be seen how limiting this specialization truly is, but I'm optimistic that RAG can help fill in any knowledge gaps.

Defining AI Agents: More Than Just Chatbots

People have been discussing "AI agents" for well over a year now, and I would argue that their promise has largely been unfulfilled, except in very constrained use cases like customer support chatbots. To further complicate matters, there are numerous definitions of what constitutes an "agent".

Here's my attempt to define it as simply as possible:

Examples of human agents include sales representatives, travel booking agents, and customer support representatives. AI agents can be thought of similarly: software designed to perform tasks typically associated with specific job roles. A key question to consider is:

Can this AI software effectively carry out the responsibilities of an existing professional role?

There are differing opinions on what constitutes an AI agent, but I like this definition for how parsimonious, predictive, and explanatory it is. It's straightforward to identify and evaluate an agent when it's getting the same work done as a person. I also think this is one of the holy grails of AI: having technology do impactful work for people.

One small wrinkle: I believe "impactful work" can happen entirely in the digital world and doesn't have to translate to the physical world. However, I think this expectation will change as robotics improves.

Prior Art and Current Developments: The o1 Advantage

With definitions out of the way, let's dive into how o1 might differ from existing LLMs. People have, in fact, been attempting to design agentic systems with LLMs for some time. Papers like ReACT proposed designs as early as October 2022. And as I mentioned previously, prompting techniques like chain of thought (CoT) has been widely adapted in AI-native applications for some time as well. What's different about o1 is that the "thinking" is performed natively within the model. This is somewhat analogous to multimodality in models: instead of needing text to be fed into them, some models can natively support audio and image input and output.

o1 learns "productive" CoT through reinforcement learning1:

Similar to how a human may think for a long time before responding to a difficult question, o1 uses a chain of thought when attempting to solve a problem. Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones.

Having CoT natively  confers a few advantages for applications as far as I can see: performance and latency. I suspect that OpenAI's massive user base through ChatGPT helped it in compiling useful data for its CoT reinforcement learning. This likely translates to better performance out-of-the-box than  that which most organizations could achieve, simply because OpenAI has a data advantage.

CoT occurring at the model-layer may avoid performing RAG with additional context, and its associated database lookups. This might reduce the number of roundtrips to OpenAI's API, and by extension perceived latency. However, it should be noted that the o1 family of models does not have consistent performance; different requests/queries may lead to different overall latencies.

These advantages notwithstanding, I intuitively believe most applications would still benefit from domain-specific CoT. The reason is straightforward: a lot of the o1 model family's CoT learning has been centered around specific technical domains. OpenAI emphasized benchmarks and reasoning ability for STEM. However, I'm excited for more research into broader reasoning ability. It's possible these models would benefit from transfer learning; having strong analytical skills in a handful of technical domains may translate to reasoning competency in other domains. This is similar to humans: your friend who's a talented musician may also be a proficient programmer.

From Language Models to Autonomous Agents

While LLMs have shown impressive capabilities, they often fall short when it comes to complex reasoning and planning. This is where OpenAI's o1 models could be truly transformative. To understand why, let's dive into a useful mental model: system 1 vs system 2 thinking.

System 1 vs System 2: A Tale of Two Thinkers

You might be familiar with the concept of system 1 and system 2 thinking, popularized by Daniel Kahneman in his book "Thinking, Fast and Slow." If not, here's a quick primer:

Think of System 1 as your brain's autopilot. It's quick, intuitive, and doesn't require much effort. It's what you use when you're driving a familiar route or recognizing a friend's face. It's all about snap judgments and gut feelings.

System 2, on the other hand, is more like your brain's analytical powerhouse. It kicks in when you need to focus, solve complex problems, or make careful decisions. It's slower but more deliberate and logical. This is what you're using when you're doing your taxes or learning a new skill.

The cool thing is, we use both systems all the time, often without even realizing it. They work together to help us navigate the world. Sometimes System 1 can lead us astray with biases or rushed judgments, and that's when it's good to consciously engage System 2 to double-check our thinking.

LLMs: The System 1 Thinkers of AI

Traditional LLMs, in their raw form, excel at System 1-like thinking. They're great at rapid-fire responses, pattern recognition, and generating human-like text. This is why they're so good at tasks like casual conversation, creative writing, or quick information retrieval.

However, when it comes to tasks requiring deep analysis, step-by-step reasoning, or complex problem-solving, these models often fall short. We've been engineering workarounds, like chain-of-thought prompting, to coax more System 2-like thinking out of them. But it's been an uphill battle.

Enter o1: The System 2 Powerhouse

This is where o1 could be a game-changer. By incorporating chain-of-thought reasoning directly into its architecture, o1 is poised to become a powerful System 2 thinker right out of the box. It's not just spitting out pre-learned patterns; it's actively reasoning through problems.

Think about the best colleagues you've worked with. They weren't just good at executing specific tasks, right? The truly exceptional ones excelled at planning, prioritizing, and exercising good judgment. They didn't just have knowledge; they knew how to apply it effectively.

That's what o1 brings to the table. It's like having a highly effective manager on your AI team. It can break down complex problems, plan out approaches, and guide other AI components to execute those plans.

A Symphony of Models

Here's where things get really interesting. I don't think o1 is going to replace other models entirely. Instead, I see a future where we leverage multiple specialized models in concert:

1. o1 as the "reasoning engine" and planner

2. GPT-4o or even 4o-mini as the "front office" for conversation and user interaction

3. Specialized models for specific tasks (e.g., code generation, data analysis, document processing)

4. Dedicated RAG pipelines for injecting domain-specific knowledge

Imagine o1 as the conductor, orchestrating this AI symphony to tackle complex, multi-step tasks that were previously out of reach for autonomous systems.

The Path to True AI Agents

This multi-model approach, with o1 at its core, could be the missing link in creating truly autonomous AI agents. We're talking about systems that can:

1. Understand a user's high-level goals

2. Break those goals down into actionable steps

3. Determine which specialized tools or models to use for each step

4. Execute the plan, adjusting course as needed

5. Deliver results in a user-friendly manner

It's not hard to see how this could revolutionize fields like personal assistants, customer service, or even software development. The AI agent of the future might use o1 to plan out a complex task, GPT-4o to handle user communication, and a variety of specialized models to execute individual steps.

Of course, we're still in the early days. There will be challenges to overcome, from model integration to handling edge cases. But o1's ability to bring robust, built-in reasoning capabilities to the table is a significant step forward on the path to truly autonomous AI agents.

What is o1?

First, why a new model series. OpenAI explains:

Essentially, these models are designed around giving the model time to "think" rather than answer immediately. The "internal chain of thought" (CoT) is notable in that CoT has been utilized in prompting for a while, now it comes natively to the model itself. OpenAI does not expose the internal chain of thought, but does provide a summary in ChatGPT (more on this later).

OpenAI highlights some key points about o1:

These enhanced reasoning capabilities may be particularly useful if you’re tackling complex problems in science, coding, math, and similar fields.

And:

For applications that need image inputs, function calling, or consistently fast response times, the GPT-4o and GPT-4o mini models will continue to be the right choice.

So these new models are not a complete replacement for other models, and are not strictly superior to previous model. Instead, they can be used in a more specialized role for planning and reasoning. Planning and reasoning is something that so far, application developers have had to engineer into their applications, so it's a big upgrade to have the model handle some of this (though I suspect most applications will still benefit from engineering bespoke CoT workflows ).

Along with the base o1 model, OpenAI also introduced o1-mini, which trades off some capability for better speed and latency. Speaking of cost, o1 is about 5-6x more expensive than GPT-4o. This further drives home the point that this model is not a drop-in replacement for all inference needs.

You can also try o1 and o1-mini through ChatGPT, though take note, the quota is much more limited:

Both o1-preview and o1-mini can be selected manually in the model picker, and at launch, weekly rate limits will be 30 messages for o1-preview and 50 for o1-mini.

Build me an app in 5 mins

To put the new o1 model through its paces, I decided to "speed run" creating a simple app as quickly as possible. I especially wanted to try this out because of o1's purported improvements in coding ability. Because o1 is more expensive and deliberative, this is not how I would actually design an application, but this is ok for fun little experiment.

Here's my prompt to Claude, which I used to refine my requirements for o1:

o1's limited quota makes every query feel very substantial, like firing a .50 cal gun. So I did some prep work in other LLMs to get o1's prompt just right. To quickly summarize: I wanted to create an app that could take a picture and output an ICS file (a standard calendar file format) that I can later import into my calendar (coincidentally Apple is bringing this feature to the new iPhone 16 through its visual intelligence feature). I'd be using GPT-4o and its new structured output feature to extract structured data from images. Structured outputs, released in August 2024, allow for more predictable and easily parseable responses from the model. As I'm often reminded, being on the cutting edge often leads to bleeding (more on this later).

After some back and forth with Claude, I got a quick requirements document that I then passed to o1. o1 did a good job initially, taking ~40 seconds to devise a plan and generate the necessary. I had a further back and forth with o1 in refining the specs, once the initial implementation was complete. The full conversation can be found here.

In the end o1 got about 90% to a working solution. It was very impressive, but still required my human input to refine its work. Besides the planning and prep work, o1 actually only took a few minutes. And o1 was exhaustive (when prompted appropriately), having generated a complete working structure:

Here's a screenshot of the working app:

I say it only got 90% there because the frontend had trouble sending data to the backend. Specifically, the date and time formats used on the frontend differed from the backend. This was exacerbated by subpar prompting that didn't extract very structured data. Ironically, the integration between frontend and backend is a common gotcha for flesh-and-blood developers too. Integration work remains a pain! Better data modeling and thinking about architecture would have avoided these issues, and those are precisely the skills great software engineers possess.

Observations and Learnings

I spent about an hour getting this all to work, and I was impressed with the initial results. That being said, it was great to see where there may be limitations for this new class of model.

Here are my initial impressions and insights:

• As prescribed by OpenAI, o1 is best used as reasoner/planner, not for grunt work. Perhaps in my next experiment, I'll have it generate a detailed spec for GPT-4o to follow. My use of o1 as the "individual contributor" seems suboptimal. I suspected as much going in but it was good to experience first-hand.

• Problem specification still matters a lot.

  • Using Claude / GPT-4o to refine your thinking and preparing it for a more capable reasoning model like o1 seems like a good tradeoff.

  • Using Claude / GPT-4o as a "thought partner" for refining your problem and the associated solution seems especially useful. I used these models not only to provide the final specification, but also to think through what it was I wanted out of my little app.

  • This part actually took the longest; it's why I think software engineering is going to evolve. Not everyone is going to want to sit down and deconstruct a problem. The skills of the best "traditional" software engineers and the software engineers of tomorrow are largely the same in this regard.

• Like all models, o1 has a knowledge cutoff that hampered its ability to use the latest and greatest.

  • For this reason, RAG still has a very important role to play, even as models get more powerful.

  • Ironically, o1 experienced egregious hallucinations regarding AI model names and API calls. It kept wanting to use gpt-4, despite strong prompting indicating newer models with newer capabilities, like multimodality were available. I even provided snippets of documentation and told it to reference that.

    • It struggled with structured outputs especially, even when an example was specified.

• While getting to a working prototype is very useful, iteration remains king.

  • I integrated some specific knowledge about LLM calls, like tuning the temperature parameter to better serve the task at hand. This is not something that's easy to discern early on.

• "Last mile" knowledge like how to deploy an app is still immensely valuable and hard to prompt for.

• Latency is very task dependent, which was cool to see. This makes sense, because like humans, some tasks require more thinking than others.

  • One response took ~40 seconds, another ~20 seconds.

  • This can also be problematic for real-time applications, but as mentioned before, o1 excels as reasoning, so real time is not the best use-case.

• The speed at which o1 can generate a working prototype suggests that the role of software developers may evolve more towards problem definition, architecture design, and fine-tuning rather than writing every line of code from scratch. I did not review or care for any of the code generated.

Unrelated to anything I tried, but pointed out by a friend, o1 and o1-mini provide the most output token capacity of any OpenAI models (up to 65k tokens):

I speculate that o1 excels at "self-contained" reasoning work, as demonstrated by OpenAI's benchmarks in competitive coding, math and science. Software engineering entails a lot more than coding, including integration between different systems, requirements-gathering, and testing. With this lens, the shortcomings I observed are not surprisingly. Integrating HTML, JS and Python is often times annoying "integration" work that requires experimentation. Many times, after building a prototype you find that you underspecified your requirements, but the only way to find out is by doing.

All this being said, I'm less than 24 hours into using o1 so I'm sure I'll learn more about its optimal use over time. 😅

Subscribe now

Conclusion

o1 represents a significant step forward in AI reasoning capabilities, but it's not a one-size-fits-all solution. Its strengths lie in complex problem-solving and planning, rather than rapid-fire tasks or real-time applications. As we continue to explore and experiment with o1, we'll likely uncover more optimal use cases and integration strategies. The future of AI-assisted development looks promising, but it's clear that human insight and expertise will remain crucial in navigating the complexities of software engineering.

Hey there, fellow knowledge enthusiasts! Remember that one teacher who actually got it? Mine was a chemical engineering whiz in high school. His golden nugget of wisdom? "You don't need to memorize this stuff to be a great engineer, you just have to remember where to look it up." That little gem has been living rent-free in my head ever since.

Fast-forward to 2024, and while that advice still slaps, I'd give it a modern twist: "You don't need to know how to do something, you can learn it just-in-time." Let me break it down for you.

Just-in-time learning is like having a super-smart friend on speed dial. Need to know something? Bam! You learn it right when you need it, no advance cramming required. It's all about learning on the fly and putting that knowledge to work pronto. And guess what? LLMs are turning this into a reality for knowledge workers everywhere.

Subscribe now

From Factory Floors to Code Galore

Just-In-Time (JIT) isn't the new kid on the block. It first made waves in manufacturing back in the 60s and 70s, with Toyota leading the charge. They figured out that making stuff only when needed was way cooler than piling up inventory like there's no tomorrow. Less waste, lower costs, and quicker production? Yes, please!

But JIT wasn't content with just revolutionizing factories. In the 90s and early 2000s, it gate-crashed the software development party. Enter Lean and Agile methodologies, bringing that sweet JIT energy to coding. The Agile Manifesto dropped in 2001, and suddenly everyone was all about iterative development and rolling with the punches.

Now, JIT is making its grand entrance into the world of knowledge work. In an era where we're drowning in information, being able to pluck out exactly what you need, when you need it, is like having a superpower. And with LLMs as our sidekicks? We're talking superhero league stuff here.

LLMs: Your 24/7 Tutor on Steroids

Gone are the days when you had to remember when to use Avogadro's number (shoutout to my chem teacher). Now, you can just ask an LLM not only when to use it, but also get a crash course on why it matters. It's like having a genius buddy who never sleeps and is always pumped to explain stuff.

Take Anthropic's Claude, for instance. This AI whiz took tech Twitter by storm faster than you can say "viral tweet." Folks were learning new tricks and shipping MVPs like it was going out of style. And Claude's Artifacts feature? Chef's kiss It's like having your code and explanations served up on a silver platter.

The hype got so real that there's now a whole meme subculture about using Cursor (an AI-powered IDE) with Claude to build apps at warp speed. Some jokers are even calling it the dawn of the "idea guy" era. Got an idea and some gumption? Boom! You're a tech mogul in the making.

I've even jumped on this bandwagon for my photography hobby. As someone with more gear than skill (we've all been there, right?), I asked Claude to help me plan an unboxing video. Check out this snazzy diagram it whipped up:

The results? Let's just say I looked like I knew what I was doing, even though I learned it all on the spot. Talk about fake it 'til you make it!

The JIT Jedi: Mastering the Force of Instant Knowledge

Now, before you go thinking you can just coast by as an "idea guy," let me hit you with some truth bombs. To really rock this JIT learning thing, you need to level up a few key skills:

1. Be a Swiss Army Knife: In this LLM-powered world, being a jack-of-all-trades is your ticket to success. The more adaptable and curious you are, the more you'll thrive.

2. Speak LLM: Clear communication is your new superpower. The better you can explain what you need, the better results you'll get. It's like learning to speak "computer," but way cooler.

3. Channel Your Inner Bulldog: Sometimes, getting what you want out of an LLM takes persistence. Embrace your inner tenacious terrier and keep digging until you strike gold.

4. Sharpen Your Mental Katana: While LLMs can feed you info, you still need to know how to slice and dice it. Keep those problem-solving and critical thinking skills razor-sharp.

5. Know Thyself (and Thy Learning Style): Understanding how you learn best is key. It's like having a mental GPS that always knows which knowledge route to take.

Subscribe now

The Future is JIT: Are You Ready to Roll?

There's never been a better time to be a curious cat with a can-do attitude. As a proud T-shaped generalist, I'm telling you, these LLMs feel like pure magic. They're expanding our mental playgrounds faster than we can say "AI revolution."

All it takes is a spark of courage to get started. The entry barrier for learning new skills or tackling gnarly problems has never been lower. With LLMs as your always-on study buddy and collaborator, the sky's the limit.

But hey, don't throw out your expertise just yet. Mastery still matters, especially when it comes to the brain basics like problem-solving and smooth talking. LLMs can help with the nitty-gritty, but that human touch? Still irreplaceable.

JIT learning, powered by our AI pals, isn't about making experts obsolete. It's about supercharging our brains and making knowledge as easy to access as your smartphone. It's turning us into learning ninjas, ready to take on any challenge that comes our way.

So, my friends, embrace the JIT mindset. Stay curious, stay hungry, and don't be afraid to dive into the unknown. Remember, the knowledge you need is just a clever prompt away. Now go out there and show the world what you've got!

Controversial take incoming: If you're building a generative AI product, you might want to think twice before rushing to hire that machine learning specialist. After months of interviewing AI professionals across the industry, from scrappy startups to tech giants, a surprising pattern has emerged. The skills that made someone a rockstar in traditional ML often don't translate directly to success in the new world of large language models (LLMs) and generative AI.

In fact, for early-stage gen AI product development, a traditional ML background might not just be unnecessary – it could even be a hindrance. Let's unpack why.

The Inverted Process of Gen AI Development

Developing products with generative AI turns the traditional ML workflow on its head. Instead of starting with data collection, model training, and then prototyping, gen AI often begins with rapid prototyping using commercial LLMs. Only later do teams worry about fine-tuning, gathering domain-specific data, or potentially training custom models.

This inversion is possible because off-the-shelf LLMs like GPT-4 are astonishingly capable out of the box, especially for natural language tasks. They often outperform specialist models that would have taken months to develop just a few years ago. This shift fundamentally changes the skills needed in early product development.

Prompting: The New Critical Skill

Enter prompting – the art and science of coaxing the desired behavior out of LLMs through carefully crafted instructions. Effective prompting has emerged as perhaps the most critical skill in gen AI development, and it requires a very different mindset from traditional ML.

The best prompters I've encountered share some key attributes:

1. Strong verbal and analytical skills

2. An ability to break complex problems into smaller, well-defined steps

3. Metacognition – the capacity to articulate tacit knowledge and "think about thinking"

Interestingly, these skills often align more closely with the strengths of generalists – product managers, UX researchers, or even philosophers – than with highly specialized ML engineers.

This creates what I've started calling an "impedance mismatch" between traditional ML skills and the needs of gen AI development. ML specialists are used to thinking in terms of data distributions, loss functions, and model architectures. But with gen AI, the focus shifts to natural language, cognitive task analysis, and understanding the nuances of human instruction.

The Role of Traditional Software Engineers

Of course, it's not just ML specialists facing challenges. Traditional software engineers often find themselves in unfamiliar territory with gen AI. The deterministic, rules-based thinking that serves them so well in most domains can be at odds with the probabilistic, fuzzy nature of LLM outputs.

Yet, solid software engineering practices remain crucial for building reliable, scalable gen AI products. The ability to design robust systems, handle errors gracefully, and create intuitive user interfaces is as important as ever. The best teams I've seen pair AI-savvy engineers with strong prompters, creating a powerful combination of technical rigor and creative problem-solving.

The Emerging AI Engineer Role

This new landscape is giving rise to a hybrid role that some are calling the "AI Engineer." The recent AI Engineer Summit highlighted the unique blend of skills required:

• A strong foundation in software engineering

• Familiarity with ML concepts and LLM capabilities

• Expertise in prompt engineering and RAG (retrieval-augmented generation)

• Product thinking and UX design sensibilities

It's a tall order, combining technical chops with creativity and strategic thinking. But for those who can master this skill set, the opportunities are immense.

When Traditional ML Backgrounds Become Valuable

Don't get me wrong – there's definitely a time and place for deep ML expertise in gen AI. As products mature and scale, skills like building robust evaluation pipelines, fine-tuning models on domain-specific data, and optimizing for latency and cost become crucial. This is where those with traditional ML backgrounds can really shine.

Additionally, some applications – like highly regulated industries or specialized scientific domains – may require custom models from the start. Here, domain knowledge combined with ML expertise is invaluable.

Implications and Future Outlook

For business leaders, this shift means rethinking hiring criteria for AI roles. Rather than focusing solely on ML credentials, consider candidates with strong analytical thinking, clear communication, and the ability to rapidly prototype and iterate.

Aspiring AI professionals should broaden their skill sets beyond just technical ML knowledge. Develop your prompt engineering abilities, study cognitive science and UX design, and cultivate a product mindset.

On an industry level, we may see a democratization of AI development. As the barriers to entry lower, a more diverse range of thinkers and problem-solvers can contribute to shaping the future of AI products.

However, this democratization also raises ethical concerns. As it becomes easier for anyone to create powerful AI systems, how do we ensure responsible development and deployment? The need for ethical guidelines and best practices in AI engineering has never been greater.

Subscribe now

Conclusion

The world of gen AI is evolving at a breakneck pace, and our notions of what makes an "AI expert" need to evolve with it. While deep ML knowledge will always have its place, the immediate future belongs to those who can bridge the gap between human needs and AI capabilities.

So before you rush to hire that ML specialist for your gen AI product, take a step back. The skills you really need might be found in unexpected places. The next great AI innovator could be a linguist, a psychologist, or even a particularly insightful product manager. Keep an open mind, and you might just stumble upon the perfect blend of talent to bring your AI vision to life.

a walk down memory lane

To put this in perspective, let's revisit a classic example: computer storage. Remember when digital memories came with a hefty price tag? I certainly do.

Back in 2005, I shelled out $100 for a 1 gigabyte flash drive—and felt like I was living in the future. Fast forward to 2024, and that same Benjamin can snag you a micro SD card smaller than your fingernail, packing a whopping 1 terabyte of storage. We're talking a 1000x increase in capacity for the same price, in a fraction of the size, in just two decades. Mind-boggling, right?

Subscribe now

the price is (rapidly) right

Now, let's turn our attention to LLMs. We're witnessing a similar price plunge, but at a pace that makes even Moore's Law look sluggish:

• 2022: Each ChatGPT query cost several cents

• 2023: Costs dropped to less than 1 cent per query

• 2024: We're approaching models that are almost too cheap to meter

The following graph tells a compelling story. It plots each model's cost per million tokens against its LMSys Elo ranking (a chess-like rating system for AI language models). The trend is clear: newer models are rapidly becoming both better and cheaper.

A quick guide to reading the graph:

• X-axis: Cost per million tokens (reverse scale, cheaper to the right)

• Y-axis: LMSys Elo rating (higher is better)

• Colored lines: Different cohorts of models

• Ideal position: Top-right corner (high performance, low cost)

The exciting part? Newer models are consistently pushing towards that ideal top-right corner, month after month. In an ideal world, a model would occupy the rightmost and topmost point on the graph (make number go up!). And that's exactly where newer models are headed, on a month-to-month basis. Up and to the right, baby!

Credit: swyx

beyond model makeovers

It's not just about overhauling entire models. Sometimes, it's the subtle engineering improvements that make a big difference. Take Anthropic's recent beta release of prompt caching (following Google's Gemini Pro from Spring '24). While not a universal cost reduction, it represents a significant saving for many use-cases:

These tweaks might not grab headlines like a brand new model, but they're quietly revolutionizing how we use AI.

the ripple effects

These rapid price drops aren't just about saving a few bucks. They're reshaping the landscape of what's possible:

1. Democratization of AI: This technology can now reach and benefit a broader spectrum of society.

2. Unlocking New Frontiers: Applications that were once cost-prohibitive are now within reach.

3. Accelerating Innovation: As AI becomes more accessible, it fuels further technological advancements. Imagine using Claude as your junior software engineer 100x more often—that's a lot more software getting written.

a paradigm shift

So what does all this mean for the future? While it's impossible to predict with certainty, these plummeting costs and skyrocketing capabilities are likely to reshape industries in ways we're only beginning to imagine. Let's consider one particularly intriguing possibility:

We're potentially witnessing a reverse industrial revolution in software. Instead of moving from bespoke goods to mass production, we're transitioning from mass-distributed software to personalized, bespoke solutions for everyone.

Imagine a world where everyone is a software engineer, armed with LLMs too cheap to meter. The result? Software purpose-built for individual needs becomes the norm, not the exception.

That's not just an incremental change—it's a paradigm shift that could reshape the very structure of our society. And it's all being driven by these plummeting AI costs.

So, as we watch these prices fall, remember: we're not just seeing numbers change on a graph. We're watching the future unfold, one cheaper token at a time.

Subscribe now

Déjà Vu All Over Again: Tech's History of Jumping the Gun

This situation actually follows a common pattern in tech. New technologies often see heavy investment in infrastructure before useful applications catch up - we've seen this seemingly over-the-top spending many times in tech history. 

Fiber Frenzy: When the Internet Got Way Too Much Pipe

Let's look back at the late 90s during the dot-com boom. Many companies spent huge sums laying fiber optic cables, betting on a surge in internet use. At the time, most home users had dial-up connections crawling at just 56 kbps. The problem? There weren't enough uses for all this new bandwidth yet. This led to a 'bandwidth glut'. Eventually, applications like video streaming did make use of this infrastructure, but only years later – after the dot-com bubble had burst.

Data Center Dilemma: Build It and They Will... Eventually Come

In the mid-2000s, we saw a similar trend, but this time with data centers – and it was all about business customers. As companies rushed to digitize their operations, tech giants poured billions into constructing massive data centers. They anticipated a surge in demand for cloud computing and data storage. However, just like with fiber optics, the applications lagged behind. Many of these data centers sat partially empty for years, running at low capacity. It wasn't until the late 2000s and early 2010s that cloud services, big data analytics, and eventually AI workloads began to fill these spaces. Once again, the infrastructure was built well before its full potential could be realized.

Even the 1800s Had Tech FOMO

Now for a curveball – let's dial back to the 1800s. Think this overbuilding trend is just a computer-age thing? Hold onto your top hat. Telegraph companies went on a wire-stringing spree, laying thousands of miles of lines alongside railroads. At first, these wires mostly carried business news – your average Joe wasn't exactly itching to tap out morse code. It took years for telegraph use to boom and turn profitable. This 19th-century example shows that jumping the gun on infrastructure isn't just a modern quirk – it's practically a tech tradition.

Overbuilding Through the Ages

This cycle of building big before the demand catches up is a recurring theme in tech history. We've seen it with railroads in the 1800s, radio networks in the 1920s, and even the early days of electricity distribution. More recently, we've witnessed similar patterns with 3G and 4G mobile networks, and the rush to build satellite internet constellations. Each time, visionaries build infrastructure anticipating future demand, often facing skepticism and financial strain before their bets pay off. AI's current trajectory? It's just the latest verse in a very familiar song.

Big Tech's Billion-Dollar Bet: Spend Now or Regret Later

While history gives us one lens to view the AI investment boom, there's another crucial factor at play: the cutthroat world of big tech competition. In this high-stakes game, the mantra seems to be "build it before your rival does." 

These tech giants are sitting on mountains of cash, but here's the twist: strict antitrust regulations are making traditional acquisitions tricky. So instead, we're seeing creative workarounds like the deals with Inflection and Character – think of them as acqui-hires on steroids.

But why the rush? It's all about staying in the race. Imagine being the company that misses the next big AI breakthrough because you skimped on computing power. Or watch helplessly as a competitor with beefier AI capabilities starts eating your lunch across various markets. Even worse, picture having to rely on your rival's AI services because you didn't build your own. For this reason, Meta has made open source AI a strategic priority.

This isn't just about keeping up with the Joneses. It's about avoiding a technological gap that could take years to close. In the fast-paced world of AI, falling behind even for a few months could mean game over. So while the spending might look excessive now, for these companies, it's a calculated bet on staying relevant – or even dominant – in an AI-powered future.

The Nuances of AI Investment: Where to Put Your Chips

Now, while this cycle of overbuilding infrastructure before demand catches up is as old as the telegraph, the AI landscape isn't just one big homogeneous blob. Some areas are already bursting at the seams with investment, while others are still crying out for more attention. Let's break it down:

Hardware: The Chip Race Heats Up

On the hardware side, we're seeing a gold rush in dedicated chips for AI inference. Why? Because these bad boys can seriously slash the cost of running AI models. Take Groq, for instance – they're making waves with their lightning-fast inference engines. But they're not alone in this race.

The cloud giants aren't sitting on their hands either. Google's been flexing with their Tensor Processing Units (TPUs) for a while now, and Amazon and Microsoft are jumping into the custom chip game too. This is actually great news for all of us – more players mean more competition, and that means we're not just stuck with Nvidia calling all the shots.

Observability: Too Many Cooks in the Kitchen?

Now, if you want a classic case of overinvestment, look no further than LLM observability tools. It's the tech equivalent of the California Gold Rush – everyone and their dog is trying to sell picks and shovels to the AI miners. The problem? There's not much to differentiate one from another, and existing players like Sentry and Datadog are already muscling in on the action. It's starting to look a bit crowded in here.

Credit: Bessemer 

Applications: Still Finding Their Feet

When it comes to actual AI applications, we're still in the "figuring it out" phase. These apps are grappling with the current limitations of the tech, like struggles with long-term planning and reliability. But don't write them off just yet.

Take Devin, for example – an AI agent for software engineering from Cognition. It's early days, but as foundation models improve, these kinds of tools are only going to get more useful. We're watching the first wobbly steps of what could become a revolution in how we work.

Foundation Models: The Big Squeeze

Now, here's where things get really interesting. The space for foundation models is slowly but surely collapsing into an oligopoly. I actually called this one in my earlier article, "A Bird's-Eye View of the Competitive Landscape in AI". Those recent deals with Inflection and Character? They're not just splashy headlines – they're symptoms of this consolidation in action.

Tomorrow's Tech Today

So, what's the takeaway here? While we're definitely seeing some of that classic tech overspending, it's not a simple story of "AI bubble about to pop". Some areas are overheated, sure, but others are just warming up. The key is to look beyond the hype and see where the real value is being created. As always in tech, it's a wild ride – but that's what makes it exciting, right?

Look, we're still in the early days of AI, and if history has taught us anything, it's that the real magic happens after the initial gold rush. Want to see where this is all heading? Keep your eyes on the early adopters – they're blazing the trail. The bottom line? It's time to build. The AI revolution isn't just coming; it's here, and it's picking up steam. So whether you're a tech giant or a garage startup, the message is clear: jump in, the water's fine (and the chips are hot). Buckle up, folks – the future's looking wild, and I, for one, can't wait to see what's next.