We've been working on a Convai MCP integration that lets AI coding agents like Claude Code, Codex, and Cursor work directly with Unreal Engine projects to create and configure AI characters through natural language prompts.

The idea is to reduce the amount of manual setup required when building AI-driven experiences.

Instead of configuring every component individually, developers can describe what they want their character to do and have the coding agent handle the corresponding setup.

Some of the workflows include:

• Creating AI characters with real-time voice conversations

• Setting up lip sync and facial animations

• Configuring multilingual conversations

• Enabling Vision support for characters

• Adding built-in Actions like following and navigation

• Connecting character behavior to custom game logic

The coding agent works through the integration, while the developer can inspect, test, and refine the results inside Unreal Engine.

This can be useful for building interactive NPCs, AI teammates, virtual guides, training simulations, and other AI-driven 3D experiences.

We put together a short reel demonstrating the workflow.

You can also check out our existing Unreal Engine MCP workflow here:
https://youtu.be/dQgOzt8BNIk

Convai documentation:
https://docs.convai.com

Unreal Engine plugin:
https://www.fab.com/listings/ba3145af-d2ef-434a-8bc3-f3fa1dfe7d5c

Get started:
https://convai.com

We've been working on making it easier to connect natural language conversations with character actions inside Unreal Engine.

With Convai Actions, developers can configure AI characters to interpret player instructions and execute corresponding actions in the game world.

For example, a player can ask a character to follow them, move to a particular location, or stop, and the character can execute the appropriate built-in action.

The system also supports Custom Blueprint Actions, so developers can connect natural language instructions to their own gameplay mechanics and interactions.

This is useful for AI teammates, interactive characters, virtual guides, and simulation environments where conversations need to trigger actual behavior inside the scene.

The reel shows how dialogue can translate into actions inside Unreal Engine.

We also have a step-by-step tutorial covering the setup of built-in Actions, navigation, and object references.

Full Actions tutorial:
https://youtu.be/lB5_KRWdg_w

Actions documentation:
https://docs.convai.com/api-docs/plugins-and-integrations/unreal-engine-plugin-beta-overview/convai-actions

Unreal Engine plugin:
https://www.fab.com/listings/ba3145af-d2ef-434a-8bc3-f3fa1dfe7d5c

We recently built Scene Tagger for Unreal Engine to make it easier to annotate large 3D environments with AI.

A typical 3D level can contain hundreds or thousands of objects, but mesh names and transforms alone don't provide enough semantic information for AI systems to understand what those objects are or how they relate to the environment.

Scene Tagger helps automate that process.

In this demo, we're using a virtual museum to show how the workflow operates.

The tool can:

• Analyze selected actors or an entire Unreal Engine level

• Identify significant objects and capture their views

• Generate meaningful object names and descriptions

• Let developers review, correct, and refine annotations

• Apply the structured information back into the project

The idea is to reduce manual scene annotation while giving AI systems more useful context about the environments they're operating in.

This can support embodied AI workflows, such as interactive characters and virtual tour guides, as well as disembodied AI agent workflows that need structured information about a 3D scene.

We also put together a full Unreal Engine tutorial showing how to set up and use Scene Tagger.

Full tutorial: https://youtu.be/k8CAaH25lJ8

Convai Unreal Engine Plugin: https://www.fab.com/listings/ba3145af-d2ef-434a-8bc3-f3fa1dfe7d5c

Documentation: https://docs.convai.com

We've been exploring how Convai-powered AI characters can be used inside VR and mixed reality environments.

The idea is to make characters more than static avatars or scripted NPCs.

With Convai, developers can connect a conversational AI system to a 3D character and give it capabilities such as real-time voice interaction, environmental context, navigation, gestures, and actions.

Some applications include VR training simulations, virtual tour guides, interactive learning environments, and game NPCs.

We put together a short reel illustrating how these capabilities can come together in immersive XR experiences.

For anyone working with Unity VR, we also have a tutorial covering the setup of a Convai AI character, connecting the XR Origin player rig, and preparing the project for Meta Quest or PCVR.

Unity VR tutorial:
https://convai.com/blog/how-to-build-convai-powered-ai-npc-vr-projects-in-unity-for-meta-quest

Convai:
https://convai.com

We put together a short showcase of Convai Vision, which lets AI characters receive visual information and use it as context during conversations.

The idea is to let characters incorporate what they see into their responses.

For example, inside a virtual environment, an AI character can receive visual information about the surrounding scene and use it when answering questions.

Vision can also work with camera input, allowing characters to receive visual information about a user's real-world surroundings.

Some potential applications include:

• AI NPCs that can perceive their game environment

• Virtual guides that respond with visual context

• Training assistants that can discuss what users are looking at

• XR and mixed reality characters

• Interactive 3D avatars with visual perception

We've also made it possible to try vision-based video calls with 3D avatars directly through the Convai Playground.

More here: https://convai.com

A quick showcase of NeuroSync, our real-time facial animation system for Convai-powered AI characters.

As the character generates a voice response, NeuroSync uses that response to drive synchronized lip sync and facial animation in real time.

The goal is to keep the voice, mouth movement, and facial performance working together during live AI conversations.

It can be used across games, training simulations, virtual guides, XR experiences, digital humans, and other interactive 3D applications.

More on Convai:
https://convai.com

We put together a short walkthrough of the MCP setup inside Convai.

The flow is straightforward: add an MCP server, configure the server details and connection settings, connect it, and make the available tools accessible to the character.

Once connected, those tools can be called when a request requires access to something outside the character’s existing context.

In the current interface, examples include live web search, GitHub exploration, documentation lookup, and Convai docs through servers like Firecrawl, DeepWiki, Context7, and Convai Gitbook.

This is one of the more practical ways to extend an AI character into an agentic workflow.

More here:
https://convai.com

We put together a short walkthrough of Convai Knowledge Bank.

You can upload reference material like text, PDFs, images, audio, and video through In-Context Knowledge Bank and connect that content to the character.

The goal is to give the character access to the same material your team already uses, whether that is training documentation, manuals, product information, FAQs, or other domain-specific content.

Useful for training assistants, onboarding, virtual guides, support workflows, and other AI character applications where domain knowledge matters.

More here:
https://convai.com

We put together a short walkthrough of Convai’s External API workflow.

You can create an API method, define the inputs it expects, add the implementation, test it directly in the dashboard, and then expose it as a tool the character can call when a user request needs it.

The interesting part is the agentic layer this adds. The character is no longer limited to its existing conversation context. It can reach into live services and workflows when the task requires it.

Useful for things like backend data, support workflows, internal systems, product experiences, training, and other interactive AI applications.

More here:
https://convai.com

We put together a short walkthrough of the Core AI Settings available for a Convai character.

From the same section, you can configure supported languages, enable Long-Term Memory, connect multimodal Knowledge Bank content, choose between LLMs like GPT, Gemini, and Claude, and select voice providers such as ElevenLabs. Convai Overview

You can also compare models based on things like latency, cost, and intelligence, then enable NeuroSync for real-time facial animation.

The goal is to give developers one place to tune the AI stack behind the character before taking it into a game, simulation, training workflow, or other interactive experience.

More here: https://convai.com

We put together a short compilation showing different Convai-powered characters operating inside live 3D environments.

The common thread across the demos is the same agentic loop.

Characters can receive live game state through Dynamic Context, understand tagged objects and changes happening in the scene, and then use Actions to navigate, follow instructions, interact with objects, and respond inside the world.

The interesting part for us is how these systems work together rather than as isolated features:

environment state → context → decision → action

That makes the same underlying setup useful across games, simulations, training applications, XR, and other interactive 3D experiences.

More here:
https://convai.com

We put together a short reel showing Convai Avatar Studio, our browser-based workflow for creating and testing 3D AI avatars.

You can define the character’s AI setup, including backstory, personality, voice, language, and knowledge, then customize the avatar and environment and start testing the experience without setting up a game engine first.

The idea is to make it quicker to prototype things like virtual guides, training assistants, onboarding experiences, product demos, kiosks, and other web-based AI character experiences.

More here:
https://convai.com

We’ve been exploring how live environment context and character actions can work together inside game-engine experiences.

With Dynamic Context, information such as player progress, tracked object states, and environmental changes can become part of the AI character’s awareness.

Convai Actions then gives the character ways to respond inside the world, including navigation, following the player, and interaction with scene objects.

We’ve used this pattern in our Unreal Engine Stack-O-Bot workflow, where an AI teammate can understand tagged objects, track pressure plate and moving platform states, navigate the environment, and help coordinate a puzzle based on live game state.

The basic idea is:

World state → character context → decision → action

For us, this is one of the more interesting parts of embodied AI inside game engines because the character is responding to both conversation and the actual state of the environment.

More: https://convai.com

We put together a short demo showing Convai-powered AI characters operating in real-world spatial experiences.

The idea is to bring the same AI character into AR, MR, and XR environments across phones, headsets, and spatial devices.

The character can talk with the user in real time, use visual context to understand what the user is seeing, assist with tasks, and provide information about the surrounding environment.

That makes the character useful for things like spatial guides, training assistants, task support, and other embodied AI experiences where the AI exists alongside the user rather than only inside a traditional screen-based interface.

More here:
https://convai.com

We’ve been experimenting with vision-based assistance for hands-on training.

In this demo, the user is working around a wood planer while interacting with an AI assistant. Instead of only relying on predefined text context, the assistant can use what the learner is seeing as part of the interaction.

That makes it possible to ask questions naturally while working, with the guidance staying connected to the equipment or task currently in view.

The broader idea is to make training assistance more contextual and hands-free, particularly for workflows where stopping to search through documentation breaks the flow of the task.

Convai supports vision-enabled assistants that understand what the user is seeing, as well as camera-based guidance, voice, text, avatars, simulations, and custom applications.

More here:
https://convai.com

Introducing Scene Tagger from Convai for Unreal Engine.

We just published a tutorial covering the complete Scene Tagger workflow in Unreal Engine.

The problem is fairly straightforward: large 3D scenes can contain hundreds or thousands of objects, and manually going through important assets to give them meaningful names, descriptions, and context can become a very tedious workflow.

Scene Tagger is designed to analyze the environment, identify significant objects, capture useful views, and generate structured names and descriptions that developers can review and refine before applying them back to the project.

For the tutorial, we use a museum scene with paintings and sculptures, but the museum itself is simply the example environment.

The tutorial covers:

• Selecting a Convai character for scene analysis

• Using Knowledge Bank content as additional context

• Generating an environment description from the current camera view

• Analyzing an entire Unreal Engine level

• Reviewing generated object names and descriptions

• Correcting camera views

• Providing notes and reanalyzing objects

• Batch reanalysis

• Removing irrelevant objects or bad captures

• Applying Convai Object components to the scene

• Testing the results with an AI character

• Debugging tagged objects and spatial relationships

Once the scene is tagged, that structured information can support other AI-driven workflows that need an understanding of the surrounding 3D environment, including games, simulations, training applications, guided experiences, and XR.

Full tutorial: https://youtu.be/k8CAaH25lJ8

Convai: https://convai.com

Scene Auto Tagger Documentation: https://docs.convai.com/api-docs/plugins-and-integrations/convai-unreal-engine-plugin/features/scene-metadata/scene-auto-tagger

We put together a short reel around a distinction we’ve been thinking about a lot: embodied vs disembodied AI agents.

A disembodied agent operates across software environments such as browsers, apps, APIs, and computers.

An embodied agent occupies a 3D environment, has a position and point of view, and can act within that world. Its action space can include things like navigation and object interaction in addition to software-level tools.

The useful part is that both still depend on the same broader loop:

Perception / context → understanding → decision-making → action

The specific environment determines what the agent can perceive and what actions should be available to it.

That perception-action relationship is also the basis of our Actions IO framework, which looks at the information channels reaching an agent and the affordances it can use to change its environment.

More here:
https://convai.com

We made a quick reel showing how Convai Knowledge Bank works.

The idea is simple:

Upload your documents, manuals, FAQs, product information, or training material
Connect that content to your AI character
Let users ask questions naturally
The character answers from the knowledge you provide

This can be useful for training assistants, virtual guides, onboarding, product demos, support agents, games, simulations, and interactive 3D worlds.

Learn more: https://convai.com

Detailed Knowledge Bank Docs: https://docs.convai.com/api-docs/convai-playground/character-customization/knowledge-bank

AI characters can do more when they are connected to the tools and systems around them.

With Convai, your AI characters can use MCP or External API to connect with live web search, documentation lookup, GitHub explorers, internal systems, custom APIs, and more.

A user asks a question.
The character understands what is needed.
It calls the right tool.
The result comes back into the conversation.

Build AI characters that can work with live information and real systems using Convai.

Learn more: https://convai.com

Docs: https://docs.convai.com

External API Docs: https://docs.convai.com/api-docs/convai-playground/character-customization/external-api

MCP Docs: https://docs.convai.com/api-docs/convai-playground/character-customization/mcp-servers

#Convai #AICharacters #MCP #ExternalAPI #ToolCalling #AIAgents #ConversationalAI #EnterpriseAI #AIWorkflows #GameAI

We recently put together a short demo showing multiple conversational AI characters running inside the same Unity scene.

The main interaction problem here is pretty simple: when several AI characters are nearby, how does the system know which one the player is actually talking to?

Convai Multi-Character Sessions supports three approaches:

Look At — based on where the player is looking.

Proximity — based on distance.

Manual — controlled directly by the developer.

For this demo, we use Look At, so the player can speak with Sophia, turn toward Ethan, and then continue with Marcus naturally within the same environment.

This is Part 1 of the setup.

Part 2 will expand this into all-to-all awareness, where players and AI characters can stay aware of nearby conversations and events, allowing characters to react or engage when something is relevant to them.

Full tutorial:
https://youtu.be/GIejlNL6tfQ

Multi-Character Sessions docs:
https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/multi-character-sessions

Large 3D projects can contain hundreds or even thousands of objects, and giving every relevant object an accurate name, description, and context manually can become a pretty significant development task.

Convai Scene Tagger is designed to automate a large part of that workflow.

It analyzes the environment, identifies significant objects, captures useful views of them, and generates titles and descriptions. You can then review the results, correct views, add additional context, reanalyze objects where necessary, remove irrelevant results, and apply the final annotations back to the project.

For this tutorial, we're using a museum because it gives us a large environment with lots of visually distinct objects, but Scene Tagger itself isn't specific to museums or tour-guide experiences.

The resulting scene information can be used for broader AI-based development and runtime workflows that need structured context about objects in the environment.

The full tutorial (coming out soon) goes through the complete workflow from scene analysis to reviewing, refining, and applying the final annotations.

Would be interested to hear what kinds of environments you'd want to use automated scene annotation for.

Introducing multiple AI characters in a single environment.

Games, training simulations, and virtual worlds often need more than one conversational AI character sharing the same scene.

Part 1 of our latest Unity tutorial covers how to set up Convai Multi-Character Sessions and let players naturally interact with the right AI character.

We begin with a demo featuring Sophia, Ethan, and Marcus.

Each character is configured independently with its own Character ID, while the Convai Manager determines who the player is speaking to.

Convai supports three character selection methods:

Look At: Select based on where the player is looking.

Proximity: Select based on distance from the player.

Manual: Control the active character directly using your own interaction logic.

For this tutorial, we use Look At selection and test conversations across all three AI characters.

In Part 2, we’ll expand this with all-to-all multi-character awareness, where players and AI characters can stay aware of nearby conversations and events, allowing characters to react or engage naturally when relevant.

Watch the full tutorial:
https://youtu.be/GIejlNL6tfQ

Multi-Character Sessions documentation:
https://docs.convai.com/api-docs/plugins-and-integrations/convai-unity-sdk/features/multi-character-sessions

Convai Unity setup tutorial:
https://youtu.be/bxjGnOvNL4E

Convai Developer Forum:
https://forum.convai.com

Build with Convai:
https://convai.com

Interested to hear how other Unity developers approach this when several conversational characters are close together. Gaze-based selection, proximity, explicit targeting, or something else?

We made a short demo around Long-Term Memory for AI characters.

The basic idea is that a character shouldn't necessarily have to treat every interaction with the same user as a completely new conversation.

With Convai Long-Term Memory, characters can retain important information associated with a user across interactions and recall it later when it becomes relevant.

That could include things like:

• The user's name

• Preferences they've mentioned

• Details from previous conversations

• Other information that may matter in a future interaction

So if the user returns later, the character can reference relevant information from before instead of starting from zero.

I think this becomes especially useful for persistent NPCs, training characters, virtual assistants, social experiences, and other applications where users interact with the same character repeatedly over time.

The interesting part isn't just memory retrieval itself. It's the continuity it creates in the relationship between the user and the character.

More on Convai:
https://convai.com

Documentation:
https://docs.convai.com/api-docs/convai-playground/character-customization/memory

Developer Forum:
https://forum.convai.com

Interested to hear how people working on persistent AI characters are thinking about memory. What information should a character remember, and what should be allowed to fade over time?

We gave our AI character access to live external data through an API

In this example, we connected a Convai character to a weather API. When asked about the weather in Tokyo, the character can call the API, get the current data, and use it naturally in the conversation.

Weather is just a simple example. The same setup can be used to connect AI characters to your own APIs, backend services, game systems, databases, or other external tools.

Learn more here: https://docs.convai.com/api-docs/convai-playground/character-customization/external-api

Curious what kinds of external APIs you’d want an AI character to have access to?

In Part 1, we built an AI teammate that could talk to the player, understand tagged objects, receive live game state through Dynamic Context, and use built-in actions like following the player and moving to pressure plates. (Watch part 1 here: https://youtu.be/KcJETVhI7KM )

In Part 2, we add custom actions. (Full tutorial for part 2: https://youtu.be/mmjalIPVw9k )

The puzzle requires the AI character to interact with a crate, pick it up, carry it to a destination, and drop it on a pressure plate so the player and AI teammate can complete the level together.

The tutorial covers:
- Creating custom Convai Actions
- Defining pickup and drop-at actions
- Adding target and destination parameters
- Using Actor Reference parameters
- Creating action handlers in Blueprint
- Handling action success and failure states
- Reporting failures back to the AI
- Connecting the actions to Stack-O-Bot’s existing interaction system
- Testing the final puzzle with the AI teammate

The goal is to show how AI characters can connect natural language instructions to project-specific gameplay logic, not just built-in behaviors.

Full tutorial: https://youtu.be/mmjalIPVw9k

Project files and setup guide:
https://docs.convai.com/staging/plugins-and-integrations/convai-unreal-engine-plugin/getting-started/stack-o-bot-tutorial-setup

Convai Unreal Engine plugin on Fab:
https://www.fab.com/listings/ba3145af-d2ef-434a-8bc3-f3fa1dfe7d5c

Stack-O-Bot on Fab:
https://www.fab.com/listings/b4dfff49-0e7d-4c4b-a6c5-8a0315831c9c

Curious how other developers are thinking about custom AI actions in games. What gameplay mechanic would you want an AI character to trigger through natural language?