What building Shippy taught us about building agents

The world of AI is moving fast, and one of the most exciting areas right now is the rise of AI agents. These aren't just chatbots; they're autonomous systems designed to perceive, reason, plan, and act to achieve complex goals. Building these agents, however, comes with its own set of challenges. The journey of creating tools like "Shippy," an AI agent developed by Skylight for real-time maritime domain awareness, offers valuable lessons for anyone looking to build reliable and effective AI agents.

This tutorial will walk you through the practical insights and best practices for building AI agents, drawing on the experiences and lessons learned from projects like Shippy. Whether you're a seasoned developer or just starting with AI, understanding these principles will help you create more robust and trustworthy agentic systems.

What Exactly Are AI Agents?

At its core, an AI agent is an autonomous software system that uses large language models (LLMs) to understand its environment, make decisions, and take actions to achieve specific objectives. Unlike a simple chatbot that responds to direct prompts, an agent can break down complex tasks, use various tools (like APIs or databases), maintain memory over time, and even learn from its interactions.

Think of it as a digital assistant that can not only answer questions but also proactively perform multi-step operations. For example, Shippy, the AI agent built by Skylight, can fuse data from multiple sources, generate citations, and provide maritime intelligence in natural language, helping analysts query vessel behaviors and interpret vessel track data.

The Agentic Loop: How Agents Work

Most AI agents operate on a continuous loop, often referred to as the "Thought-Action-Observation" loop or "Perception, Reasoning, Action, Learning." Here's a simplified breakdown:

• Perception/Observation: The agent takes in information from its environment, which could be user input, data from a database, or responses from external tools.
• Reasoning/Planning: Using an LLM, the agent processes this information, understands the current state, and plans the next steps to achieve its goal. This might involve breaking a large task into smaller sub-tasks.
• Action: The agent executes the planned steps. This could be calling a function, querying an API, generating text, or interacting with another agent.
• Learning/Reflection: The agent observes the outcome of its actions and updates its internal state or adjusts its future plans based on the feedback. This loop allows agents to adapt and improve over time.

Key Lessons Learned from Building Agents (Inspired by Shippy)

Building an AI agent, especially for high-stakes domains like maritime awareness, highlights that success hinges on robust system design, not just the underlying LLM. Here are some crucial lessons:

1. Define Clear Goals and Scope

One of the first and most important lessons is to have crystal-clear objectives for your agent. Vague goals can lead to agents that "wander or fixate on the wrong problem." For Shippy, the goal was specific: provide real-time maritime domain awareness by fusing data and generating cited intelligence.

• Use Case: Instead of "improve customer experience," aim for "resolve at least 80% of user requests without human intervention."
• Tip: Start with a narrow task and expand only when the core functionality is reliable.

2. Choose Your Foundations Wisely: Frameworks and Tools

While you can build agents from scratch, frameworks offer pre-built components that simplify development. They provide abstractions for LLM integration, tool management, memory handling, and orchestration.

• LangChain: An open-source framework (MIT-licensed) for developing LLM-powered applications. It's excellent for chaining LLMs with external tools, memory modules, and APIs. LangChain is widely adopted and supports integration with over 100 different language models.
  • Pricing: The core LangChain framework is free. However, its observability platform, LangSmith, offers a free Developer tier (5,000 traces/month for one seat) and a Plus tier at $39/seat/month (10,000 traces/month included, then $2.50 per 1,000 additional base traces). Enterprise pricing is custom.
• AutoGen: Developed by Microsoft Research, AutoGen is an open-source framework for building AI agents, particularly multi-agent systems, and facilitating cooperation among them. It supports asynchronous messaging and is modular and extensible.
  • Note: AutoGen is currently in maintenance mode. Microsoft recommends new users start with the Microsoft Agent Framework, which builds on AutoGen's lessons with enterprise-grade support. AutoGen Studio provides a no-code interface for prototyping.
• LlamaIndex: This framework focuses on connecting LLMs with your custom data sources for retrieval-augmented generation (RAG). It's available as a free open-source library and a commercial SaaS platform called LlamaIndex Cloud.
  • Pricing: The open-source library is free. LlamaIndex Cloud uses a credit-based system (1,000 credits = $1.25). The Free plan includes 10,000 credits/month. The Starter plan is $50/month for 40,000 credits, with pay-as-you-go options.
• CrewAI: A framework for building and operating coordinated teams of AI agents that execute complex workflows autonomously. It's known for multi-agent orchestration.

3. Design for Deterministic Tooling with Non-Deterministic Agents

AI agents, powered by LLMs, are inherently non-deterministic; they don't always behave the same way twice. A critical lesson from Shippy was to use purpose-built, deterministic tooling to interact with external systems. Shippy communicates with Skylight's API through a custom, deterministic command-line interface (CLI), preventing errors that arose when the model directly constructed raw API calls.

Example (Conceptual Python Tool Integration):

# Define a deterministic tool function
def get_vessel_location(vessel_id: str) -> dict:
    """
    Retrieves the current location of a vessel.
    Args:
        vessel_id (str): Unique identifier for the vessel.
    Returns:
        dict: A dictionary containing 'latitude', 'longitude', and 'timestamp'.
    """
    # This would call a well-defined internal API or database
    # Example: return api_client.fetch_location(vessel_id)
    if vessel_id == "ABC123":
        return {"latitude": 34.0522, "longitude": -118.2437, "timestamp": "2026-07-16T10:00:00Z"}
    else:
        return {"error": "Vessel not found"}

# Agent uses this tool via function calling
# (Conceptual pseudo-code for an LLM agent)
# agent.add_tool(get_vessel_location)
# response = agent.run("Where is vessel ABC123?")
# The LLM would call get_vessel_location("ABC123") and process its structured output.

4. Memory is Crucial, but Needs Design

Agents need memory to maintain context over conversations and tasks. This involves both short-term memory (the LLM's context window) and long-term memory (often implemented with vector databases).

• Short-term: The immediate conversation history. Managing token limits is key.
• Long-term: For knowledge retrieval (RAG) and persistent state. Tools like Pinecone, Weaviate, or ChromaDB are popular for storing and retrieving embeddings.
• Lesson: "Context and memory need to be designed, not dumped in." Design what gets remembered and how it's used.

5. Build for Resilience and Self-Correction

Real-world agents will encounter unexpected inputs, API errors, or ambiguous situations. Robust error handling and self-correction mechanisms are vital.

• Error Handling: Implement retry logic for external tool calls and specific error messages for the agent to interpret. Avoid simply retrying agent output, as it's non-deterministic.
• Reflection: Allow the agent to reflect on its actions and outcomes, identifying when a task failed and how to adjust its plan.
• Guardrails: Define clear boundaries and fail-safe behaviors. For Shippy, the "soul" (system prompt) explicitly defines what the agent will and won't do, like not making legal determinations.

6. Isolated Sessions for Privacy and Reliability

For multi-user or sensitive applications, ensuring data privacy and preventing cross-contamination is paramount. Shippy addresses this by running every conversation in its own ephemeral, isolated session. This involves provisioning dedicated infrastructure (like Kubernetes deployments) for each user.

Conceptual Architecture for Isolated Sessions:

User Request -> Load Balancer -> Session Manager (e.g., Mothership for Shippy)
    -> Provision Isolated Environment (e.g., Kubernetes Pods)
        -> Agent Runtime (LLM, Tools, Memory)
        -> User-specific Data/Credentials
    -> Execute Agent Task
    -> Return Result
    -> Decommission Environment

7. Comprehensive Evaluation is Non-Negotiable

How do you know if your agent is working correctly? Rigorous evaluation is key. Shippy employs a "whole-agent eval" pipeline where experts write weighted rubrics, and scenarios run against live data. A version that regresses doesn't get deployed.

• Evaluation Datasets: Build robust datasets with success cases, edge cases, and failure scenarios (at least 30 evaluation cases per agent).
• Metrics: Evaluate not just accuracy of the outcome but also reasoning, traceability, adaptability, and tool-use success.
• Observability: Log and monitor everything: every step, tool call, token usage, and context consumed. Tools like LangSmith are designed for this.

8. Human-in-the-Loop is Essential

For critical tasks, incorporating human oversight is not optional. Agents can escalate complex or uncertain situations to a human for intervention or approval. This creates a "trust but verify" system, which is core to reliability in business-critical AI systems.

A Practical Approach to Building an Agent (Step-by-Step)

Let's outline a general tutorial for building a basic AI agent, incorporating the lessons above.

Step 1: Define Your Agent's Purpose and Persona

Start by clearly defining what problem your agent will solve and what its role or "persona" will be. This forms the agent's "soul."

# Example: A customer support agent for a SaaS product
# Goal: Resolve common user queries about product features and troubleshooting.
# Persona: Friendly, knowledgeable, and efficient support specialist.
system_prompt = """
You are a helpful and friendly AI customer support assistant for NerdsTool SaaS.
Your primary goal is to assist users with questions about our product features,
troubleshooting common issues, and guiding them to relevant documentation.
Always maintain a polite and professional tone.
If you cannot resolve a query, escalate it to a human support agent.
Do NOT speculate or provide information outside of your knowledge base.
"""

Step 2: Select Core Components (LLM, Memory, Tools)

Choose your LLM, decide on your memory strategy, and identify the tools your agent will need to interact with the world.

• LLM: Pick a suitable model (e.g., OpenAI's GPT-4o, Anthropic's Claude, Google's Gemini).
• Memory: For short-term, rely on the LLM's context. For long-term, consider a vector database.
• Tools: List the specific actions your agent needs to take.
  • search_documentation(query: str): Searches the product documentation.
  • create_support_ticket(issue_description: str, user_email: str): Creates a new ticket in the support system.
  • get_product_status(): Checks the current status of NerdsTool services.

# Conceptual Python code for tool definition
from typing import Dict, Any

def search_documentation(query: str) -> str:
    """Searches the NerdsTool official documentation for the given query.
    Returns relevant snippets of information."""
    # In a real scenario, this would call a RAG system or an API
    if "pricing" in query.lower():
        return "NerdsTool offers a free tier, Pro at $29/month, and Enterprise plans. Visit our pricing page for details."
    elif "troubleshooting login" in query.lower():
        return "For login issues, please try resetting your password or clearing browser cache. If problems persist, contact support."
    else:
        return "Could not find specific documentation for your query. Please rephrase or create a support ticket."

def create_support_ticket(issue_description: str, user_email: str) -> str:
    """Creates a new support ticket in the helpdesk system.
    Returns a ticket ID."""
    # In a real scenario, this would call an external API like Zendesk or Freshdesk
    ticket_id = f"NT-{hash(issue_description + user_email) % 10000}"
    return f"Support ticket created successfully with ID: {ticket_id}. A human agent will contact you shortly at {user_email}."

# Assume other tools like get_product_status are similarly defined.

Step 3: Implement the Agentic Loop (Using a Framework)

Utilize an AI agent framework (like LangChain or AutoGen) to orchestrate the LLM, memory, and tools within the perception-reasoning-action loop.

Conceptual LangChain Agent Setup:

from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain.tools import tool

# Wrap your functions as LangChain tools
@tool
def search_docs(query: str) -> str:
    """Searches the NerdsTool official documentation for the given query."""
    return search_documentation(query)

@tool
def create_ticket(issue_description: str, user_email: str) -> str:
    """Creates a new support ticket in the helpdesk system."""
    return create_support_ticket(issue_description, user_email)

# Define your LLM
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Define the prompt for the agent
prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}") # For agent's internal thoughts and tool calls
])

# Create the agent
tools = [search_docs, create_ticket]
agent = create_tool_calling_agent(llm, tools, prompt)

# Create the agent executor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Example usage
# response = agent_executor.invoke({"input": "What is the pricing for NerdsTool?"})
# print(response["output"])

# response = agent_executor.invoke({"input": "I can't log in, and I tried resetting my password. My email is user@example.com"})
# print(response["output"])

Step 4: Testing, Evaluation, and Refinement

Continuously test your agent with a variety of inputs, including edge cases and failure scenarios. Use observability tools (like LangSmith) to trace its execution and understand its decision-making.

• Unit Tests for Tools: Ensure your deterministic tools work perfectly.
• Agent-level Tests: Create test cases for end-to-end user interactions.
• Human Feedback: Collect feedback from real users and use it to refine prompts, add new tools, or adjust agent behavior.
• Performance Monitoring: Keep an eye on latency, token usage, and API costs. Optimize where possible, perhaps by using smaller models for simpler tasks or caching results.

Who Should Try Building AI Agents?

• Software Developers and AI Engineers: If you're looking to build intelligent, autonomous applications that go beyond simple chatbots.
• Researchers: For exploring advanced AI concepts like multi-agent systems, complex reasoning, and adaptive behaviors.
• Product Managers: To understand the capabilities and limitations of agentic AI for designing new product features.

Who Should Skip This?

• Non-technical Users: If you're looking for off-the-shelf, no-code solutions without diving into the underlying technical details. While some tools offer no-code interfaces (like AutoGen Studio), understanding the principles is still beneficial.
• Those Expecting Perfect Autonomy: AI agents are powerful but require careful design, oversight, and iteration. They are not "set-it-and-forget-it" solutions, especially in production.

Frequently Asked Questions

What are the biggest challenges when building AI agents?

The biggest challenges include defining clear goals, managing non-deterministic behavior, ensuring data privacy and isolation for multiple users, building robust error handling, and implementing comprehensive evaluation and observability.

Which AI agent frameworks are most popular today?

Popular open-source frameworks include LangChain, Microsoft AutoGen, LlamaIndex, and CrewAI. These frameworks provide essential building blocks for orchestrating LLMs, tools, and memory.

Is building AI agents expensive?

The core frameworks like LangChain and LlamaIndex are open-source and free to use. However, costs can accrue from using commercial platforms like LangSmith or LlamaIndex Cloud for observability and managed services, as well as API costs for the underlying Large Language Models (LLMs) and any external tools or vector databases you integrate.

How important is human oversight in AI agent systems?

Human oversight, often referred to as "human-in-the-loop," is crucial for ensuring reliability, safety, and alignment with user needs, especially in high-stakes applications. Agents should be designed to escalate complex or uncertain situations to human operators.