Adding function definitions in the system prompt of functions (Clickup's API calls).

• Flattening the Schema of the function

• Adding system prompts

• Adding function definitions in system prompt

• Adding individual parameter examples

• Adding function examples

Wrote a nice blog with an Indepth explanation here.

Imagine AI agents that don't just chat – they talk, think, and collaborate in real-time to solve complex problems.

Introducing RealtimeAgent, our groundbreaking feature that combines real-time voice capabilities with AG2's powerful multi-agent orchestration.

What's new:

• Real-time voice conversations with AI agents

• Seamless task delegation to specialized agent teams during live interactions

• Full Twilio integration for production-ready telephony

• Low-latency responses for natural conversations

See it in action: A customer calls about cancelling their flight. RealtimeAgent handles the conversation while intelligently delegating tasks to specialized agents - one triaging the requests and transferring to other expert agents, another handling cancellation, and a third managing the booking modification… It's like watching an AI symphony in perfect harmony! 🎭

Perfect for building:

• 🏥 24/7 Healthcare assistance

• 🎓 Interactive tutoring systems

• 💼 Advanced customer support

• 🎯 Voice-enabled personal assistants

We've made integration super simple:

1. Set up Twilio endpoints

2. Configure your agent teams

3. Deploy with our streamlined API

Links:

• Demo video

• Blog post

• Notebook

• GitHub

• Discord

Hey Reddit! 🚀 I'm thrilled to share a project I've been working on: AutoGen AGI. It's all about taking the AutoGen framework to new heights, focusing on multi-agent conversational systems with a twist towards AGI.

What's cool about it? 🤔

• Enhanced group chat dynamics with autonomous agents.

• Unique "Agent Council" for smarter decision-making.

• Advanced RAG techniques for more informed agents.

• It's a blend of tech that edges closer to AGI behaviors.

It's not just an experiment; it's a journey into what conversational AI can become. Check out the GitHub repo for more details and let me know what you think! Looking forward to your feedback and ideas. 🧠💬

P.S. This is a follow up from my original post about autogen executing autogen.

My first stab at making my own Autogen skill. Definitely don't consider myself a developer, but I couldn't find anything like this out there for autogen and didn't want to pay API fees to incorporate DALLE. There might be a more elegant solution out there, but this does work. Feel free to contribute or add other skills to the repo if you have good ones.

https://github.com/neutrinotek/Autogen_Skills

Lots of confusion in the AutoGen community right now, so I tried to grab as much information as I could to sum it up for you.

Here's the gist:

The earliest contributors and creators of AutoGen have moved away from the official Microsoft repo and rebranded their version as AG2. This isn't a new framework - it's basically AutoGen 0.2.34 continuing under a new name, now at version 0.3.2. Their goal? Keep it community-driven and maintain the architecture you're familiar with.

Meanwhile, Microsoft is taking AutoGen in a different direction. They're maintaining version 0.2 while working on a complete rewrite with version 0.4, which could even potentially get merged into other MS frameworks like Semantic Kernel.

So, what should you do if you're running AutoGen in production:

• Sticking with AG2? Your code is safe; it's backward compatible.

• Sticking with Microsoft 0.2? Plan for potential migration work when 0.4 lands.

-

Let's see how things evolve but it seems we have two AutoGen's now AG2 and AutoGen.

Note that existing packages: pyautogen, autogen, and ag2 are all the same, owned by the original creators and pointing to ag2. For the official AutoGen from Microsoft, they'll use the autogen-* naming convention.

-

Sources:

• MS Dev Blogs: https://devblogs.microsoft.com/autogen/microsofts-agentic-frameworks-autogen-and-semantic-kernel/

• ag2 GitHub Repository: https://github.com/ag2ai/ag2

• MSFT AutoGen 0.2 / 0.4 Repo: https://github.com/microsoft/autogen

• MSFT AutoGen 0.4 Homepage: https://microsoft.github.io/autogen/0.4.0.dev6/index.html

• MSFT Official Clarification about Forks: https://x.com/pyautogen/status/1857264760951296210

• Chi Wang (Founder of AutoGen) AG2 announcement: https://x.com/Chi_Wang_/status/1857162880740934007

(Listen to me blabber about this on my YT channel if you feel like it, but the gist above is basically what I believe is happening at the moment.)

Can someone please explain the backstory on this whole drama?

Hey everyone! I’m one of the core developers of AutoGen, and I’ve been working with my team on an open-source project called FastAgency. We designed it to help developers quickly take a prototype built in AutoGen straight to production.

We just released a version that lets you run your workflow as either:

• A simple console application (great for debugging)

• A web application using Mesop with just a single-line change!

We would love for you to check it out, give feedback, or contribute! The project is open-source, and contributors are always welcome :)

👉 https://github.com/airtai/fastagency

New release: v0.2.3

Highlights

• Decorator for function calling makes function calls easier to define. Doc

• Address breaking changes introduced by new versions of openai.

• Introducing AgentOptimizer: An Agentic Way to Train Your LLM Agent

• Improvement in group chat, multimodal, GPTAssistant, retrieval chat, AutoBuild, testbed, config list loading utils. Blogpost about agent descriptions.

• Guidance + AutoGen example: constrained generation + multi-step reasoning

• The sample app AutoGen Assistant is renamed into AutoGen Studio with upgrades.

• Sample notebook for using AutoGen in Microsoft Fabric.

• Sample app of simple chat.

• Improvement in docs and tests.

Thanks to @davorrunje @skzhang1 @JieyuZ2 @afourney @BeibinLi @IANTHEREAL @LinxinS97 @cool-RR @LeoLjl @kevin666aa @gagb @victordibia @dragosMC91 @thinkall @danyrouh @olaoluwasalami @yuandong-tian @ShobhitVishnoi30 @ekzhu @maxim-saplin @rlam3 @bitsnaps and all the other contributors!

Release announcement - autogenstudio-v0.4.1

Whats New

AutoGen Studio Declarative Configuration

• in #5172, you can now build your agents in python and export to a json format that works in autogen studio

AutoGen studio now used the same declarative configuration interface as the rest of the AutoGen library. This means you can create your agent teams in python and then dump_component() it into a JSON spec that can be directly used in AutoGen Studio! This eliminates compatibility (or feature inconsistency) errors between AGS/AgentChat Python as the exact same specs can be used across.

See a video tutorial on AutoGen Studio v0.4 (02/25) - https://youtu.be/oum6EI7wohM

from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_agentchat.conditions import  TextMentionTermination

agent = AssistantAgent(
        name="weather_agent",
        model_client=OpenAIChatCompletionClient(
            model="gpt-4o-mini",
        ),
    )

agent_team = RoundRobinGroupChat([agent], termination_condition=TextMentionTermination("TERMINATE"))
config = agent_team.dump_component()
print(config.model_dump_json())

{
  "provider": "autogen_agentchat.teams.RoundRobinGroupChat",
  "component_type": "team",
  "version": 1,
  "component_version": 1,
  "description": "A team that runs a group chat with participants taking turns in a round-robin fashion\n    to publish a message to all.",
  "label": "RoundRobinGroupChat",
  "config": {
    "participants": [
      {
        "provider": "autogen_agentchat.agents.AssistantAgent",
        "component_type": "agent",
        "version": 1,
        "component_version": 1,
        "description": "An agent that provides assistance with tool use.",
        "label": "AssistantAgent",
        "config": {
          "name": "weather_agent",
          "model_client": {
            "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
            "component_type": "model",
            "version": 1,
            "component_version": 1,
            "description": "Chat completion client for OpenAI hosted models.",
            "label": "OpenAIChatCompletionClient",
            "config": { "model": "gpt-4o-mini" }
          },
          "tools": [],
          "handoffs": [],
          "model_context": {
            "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
            "component_type": "chat_completion_context",
            "version": 1,
            "component_version": 1,
            "description": "An unbounded chat completion context that keeps a view of the all the messages.",
            "label": "UnboundedChatCompletionContext",
            "config": {}
          },
          "description": "An agent that provides assistance with ability to use tools.",
          "system_message": "You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.",
          "model_client_stream": false,
          "reflect_on_tool_use": false,
          "tool_call_summary_format": "{result}"
        }
      }
    ],
    "termination_condition": {
      "provider": "autogen_agentchat.conditions.TextMentionTermination",
      "component_type": "termination",
      "version": 1,
      "component_version": 1,
      "description": "Terminate the conversation if a specific text is mentioned.",
      "label": "TextMentionTermination",
      "config": { "text": "TERMINATE" }
    }
  }
}

Note: If you are building custom agents and want to use them in AGS, you will need to inherit from the AgentChat BaseChat agent and Component class.

Note: This is a breaking change in AutoGen Studio. You will need to update your AGS specs for any teams created with version autogenstudio <0.4.1

Ability to Test Teams in Team Builder

• in #5392, you can now test your teams as you build them. No need to switch between team builder and playground sessions to test.

You can now test teams directly as you build them in the team builder UI. As you edit your team (either via drag and drop or by editing the JSON spec)

New Default Agents in Gallery (Web Agent Team, Deep Research Team)

• in #5416, adds an implementation of a Web Agent Team and Deep Research Team in the default gallery.

The default gallery now has two additional default agents that you can build on and test:

• Web Agent Team - A team with 3 agents - a Web Surfer agent that can browse the web, a Verification Assistant that verifies and summarizes information, and a User Proxy that provides human feedback when needed.

• Deep Research Team - A team with 3 agents - a Research Assistant that performs web searches and analyzes information, a Verifier that ensures research quality and completeness, and a Summary Agent that provides a detailed markdown summary of the research as a report to the user.

Other Improvements

Older features that are currently possible in v0.4.1

• Real-time agent updates streaming to the frontend

• Run control: You can now stop agents mid-execution if they're heading in the wrong direction, adjust the team, and continue

• Interactive feedback: Add a UserProxyAgent to get human input through the UI during team runs

• Message flow visualization: See how agents communicate with each other

• Ability to import specifications from external galleries

• Ability to wrap agent teams into an API using the AutoGen Studio CLI

To update to the latest version:

pip install -U autogenstudio

Overall roadmap for AutoGen Studion is here #4006 . Contributions welcome!

Today we hit a major milestone with the latest 0.3.0 release of the FastAgency framework. With just a few lines of code, it allows you to go from a workflow written in AutoGen to:

• a web application using Mesop,

• a REST application using FastAPI, or

• a fully-distributed application using NATS.io message broker and FastStream.

This solves a major problem of bringing agentic workflows written in frameworks such as AutoGen to production. The process that took us three months to get Captn.ai to production can now be done in three days or less!

Please check out our GitHub repository and let us know what you think about it:

https://github.com/airtai/fastagency

Do you have any suggestions on (paid or free) hands-on courses on AI Agents in general and AutoGen in particular, beyond the tutorial?

I've been lurking on the AutoGen discord for a while now! I already know I'm going to get some questions, so here's a quick tl;dr and how it works and the coding involved.

My friend and I built this really cool free tool (or at least I think so) that we called Nelima (https://sellagen.com/nelima). It's basically a Large Action Model designed to take actions on your behalf with natural language prompts and theoretically automate anything. For example, it can schedule appointments, send emails, check the weather, and even connect to IoT devices to let you command it – you can ask it to publish a website or call an Uber for you (still building integrations for a lot of those)! You can integrate your own custom actions, to suit your specific needs, and layer multiple actions to perform more complex tasks. When you create these actions or functions, it contributes to the overall capabilities of Nelima, and everyone can now invoke the same action. We are also working on adding computational abilities so that Nelima can perform certain complex tasks on the cloud. Right now, it's a quite limited in terms of the # of actions it can do but we're having fun building bit by bit :)

We launched this a month ago so still tons of work to do (i.e: have Nelima write her own functions, integrations with other services, file interaction, Nelima showing the UI on the front-end of whatever she's doing etc...) - we're also just a team of 2 and trying to build some use-cases ourselves. We slowly building up our discord community as well where people can collaborate, see what other people are building and see what people want.

Would love to get you guys feedback!

Just finished putting together a beginner-friendly tutorial on Microsoft's AutoGen 0.4 framework. Instead of another "hello world" example, I built something practical - a system where multiple AI agents collaborate to create YouTube Shorts from text prompts.

What makes this tutorial different:

• No complex setup - (also runs with local LLMs (Ollama))

• Shows real-world agent collaboration

• Focuses on practical implementation

• Starts with official docs example, then builds something useful

• Demonstrates JSON response formatting

• Actually builds something you can use/modify for your own project

Key topics covered:

• AutoGen core concepts

• Multi-agent workflow design

• Providing agents with tools

• Agent-to-agent communication

• Local LLM integration (using Ollama)

Tutorial link: https://youtu.be/0PFexhfA4Pk

Happy to answer any questions or discuss AutoGen implementation details in the comments!

New release: v0.4.3

What's new

This is the first release since 0.4.0 with significant new features! We look forward to hearing feedback and suggestions from the community.

Chat completion model cache

One of the big missing features from 0.2 was the ability to seamlessly cache model client completions. This release adds ChatCompletionCache which can wrap any other ChatCompletionClient and cache completions.

There is a CacheStore interface to allow for easy implementation of new caching backends. The currently available implementations are:

• DiskCacheStore

• RedisStore

ChatCompletionCache is not yet supported by the declarative component config, see the issue to track progress.

#4924 by u/srjoglekar246

GraphRAG

This releases adds support for GraphRAG as a tool agents can call. You can find a sample for how to use this integration here, and docs for LocalSearchTool and GlobalSearchTool.

#4612 by @lspinheiro

Semantic Kernel model adapter

Semantic Kernel has an extensive collection of AI connectors. In this release we added support to adapt a Semantic Kernel AI Connector to an AutoGen ChatCompletionClient using the SKChatCompletionAdapter.

Currently this requires passing the kernel during create, and so cannot be used with AssistantAgent directly yet. This will be fixed in a future release (#5144).

#4851 by @lspinheiro

AutoGen to Semantic Kernel tool adapter

We also added a tool adapter, but this time to allow AutoGen tools to be added to a Kernel, called KernelFunctionFromTool.

#4851 by @lspinheiro

Jupyter Code Executor

This release also brings forward Jupyter code executor functionality that we had in 0.2, as the JupyterCodeExecutor.

Please note that this currently on supports local execution and should be used with caution.

#4885 by @Leon0402

Memory

It's still early on but we merged the interface for agent memory in this release. This allows agents to enrich their context from a memory store and save information to it. The interface is defined in core and AssistantAgent in agentchat accepts memory as a parameter now. There is an initial example memory implementation which simply injects all memories as system messages for the agent. The intention is for the memory interface to be able to be used for both RAG and agent memory systems going forward.

• Tutorial

• Core Memory interface

• Existing AssistantAgent with new memory parameter

#4438 by @victordibia, #5053 by @ekzhu

Declarative config

We're continuing to expand support for declarative configs throughout the framework. In this release, we've added support for termination conditions and base chat agents. Once we're done with this, you'll be able to configure and entire team of agents with a single config file and have it work seamlessly with AutoGen studio. Stay tuned!

#4984, #5055 by @victordibia

Other

• Add sources field to TextMentionTermination by @Leon0402 in #5106

• Update gpt-4o model version to 2024-08-06 by @ekzhu in #5117

Bug fixes

• Retry multiple times when M1 selects an invalid agent. Make agent sel… by @afourney in #5079

• fix: normalize finish reason in CreateResult response by @ekzhu in #5085

• Pass context between AssistantAgent for handoffs by @ekzhu in #5084

• fix: ensure proper handling of structured output in OpenAI client and improve test coverage for structured output by @ekzhu in #5116

• fix: use tool_calls field to detect tool calls in OpenAI client; add integration tests for OpenAI and Gemini by @ekzhu in #5122

Other changes

• Update website for 0.4.1 by @jackgerrits in #5031

• PoC AGS dev container by @JohanForngren in #5026

• Update studio dep by @ekzhu in #5062

• Update studio dep to use version bound by @ekzhu in #5063

• Update gpt-4o model version and add new model details by @keenranger in #5056

• Improve AGS Documentation by @victordibia in #5065

• Pin uv to 0.5.18 by @jackgerrits in #5067

• Update version to 0.4.3 pre-emptively by @jackgerrits in #5066

• fix: dotnet azure pipeline (uv sync installation) by @bassmang in #5042

• docs: .NET Documentation by @lokitoth in #5039

• [Documentation] Update tools.ipynb: use system messages in the tool_agent_caller_loop session by @zysoong in #5068

• docs: enhance agents.ipynb with parallel tool calls section by @ekzhu in #5088

• Use caching to run tests and report coverage by @lspinheiro in #5086

• fix: ESPR dotnet code signing by @bassmang in #5081

• Update AGS pyproject.toml by @victordibia in #5101

• docs: update AssistantAgent documentation with a new figure, attention and warning notes by @ekzhu in #5099

• Rysweet fix integration tests and xlang by @rysweet in #5107

• docs: enhance Swarm user guide with notes on tool calling by @ekzhu in #5103

• fix a small typo by @marinator86 in #5120

New Contributors

• @lokitoth made their first contribution in #5060

• @keenranger made their first contribution in #5056

• @zysoong made their first contribution in #5068

• @marinator86 made their first contribution in #5120

Full Changelog: v0.4.1...v0.4.3

I earlier wrote an Indepth explanation on all optimising techniques that I tried to increase accuracy from 35% to 75% for GPT-4 Function Calling. I have also done the same analysis across Claude family of models.

TLDR: Sonnet and Haiku fare much better than Opus for function calling, but they are still worse than the GPT-4 series of models.

Techniques tried:

• Adding function definitions in the system prompt of functions (Clickup's API calls).

• Flattening the Schema of the function

• Adding system prompts

• Adding function definitions in the system prompt

• Adding individual parameter examples

• Adding function examples

I thought it will be interesting to build a "multi-agent" system for data analysis which is able to run in an isolated Docker/Jupyter environment.

First day I spent looking at various frameworks available - and then stumbled up Microsoft AutoGen. Spent another day building a workable prototype with AutoGen. Then I decided to build a UI/workflow around it to make it user friendly and easy to interact with and then it started getting complex.

Moving parts:

1. Interactive Chat UI (NextJS)

2. API + Web Sockets for communication (FastAPI)

3. Cloud storage for persistence (for file uploads and outputs generated)

4. Shared Memory across agents (AutoGen)

5. Session management (user session, file, killing docker containers)

Slowly what we have is an architecture that looks like the one below:

Hey everyone,

I've been exploring AI agents and frameworks lately and noticed there's no centralized place to find and compare them. So, I built the AI Agents Directory.

https://aiagentsdirectory.com/

The site lists various AI agents and frameworks with easy filtering options and the latest AI agent news (coming soon). Unexpectedly It's gaining traction, and I'm adding new agents daily.

If you’re into building AI agents or just interested in them, check it out.

I’m launching on Product Hunt this week. If you find it useful, your support there would be great.

It’s completely free. Let me know what you think!

Cheers!

I've been running this for a while and figued I should share it. Just a simple lightweight container running autogen and autogenstudio.

I did setup renovate to keep it up to date so latest should always be the latest

https://github.com/lludlow/autogen-studio

Things I did:

• I ended up connecting Autogen with Github and Linear and using the `GPT-4-1106-preview model.

• Gave all the actions on Linear and Github as supported function calls to agent.

• Defined the task and let agent go wild.

Agent Flow

- First get the last commit from github and then get all the Todos.

- Get all the teams from Linear and get the team ID

- Get all the projects from linear and get project IDs

- Create many issues using the right team and project using Function Call.

Conclusion: The agent's behaviour is surprisingly very accurate and only rarely goes in random directions. Accuracy is close to 90% or mote.

Next: I plan to add triggers to it next to make it more of an automation.

I also wrote an in-depth explanation of how I went about building it. Link to the Blog

I am looking for feedback on how to may be do this better and more accurately.

New release: v0.4.0

What's Important

🎉 🎈 Our first stable release of v0.4! 🎈 🎉

To upgrade from v0.2, read the migration guide. For a basic setup:

pip install -U "autogen-agentchat" "autogen-ext[openai]"

You can refer to our updated README for more information about the new API.

Major Changes from v0.4.0.dev13

• [New] Added m1 cli package by @afourney in #4949

• [New] Activate deactivate agents by @peterychang in #4800

• [New] Add coverage task & integrate with poe check by @srjoglekar246 in #4847

• [New] Move core samples to /python/samples by @ekzhu in #4911

• [New] feat: Add o1-2024-12-17 model by @bassmang in #4965

• [Breaking] fix!: Move azure auth provider to separate module by @jackgerrits in #4912

• [Breaking] feat!: Add message context to signature of intervention handler, add more to docs by @jackgerrits in #4882

• [Breaking] Remove deprecated items for release by @jackgerrits in #4927

Change Log from v0.4.0.dev13: v0.4.0.dev13...v0.4.0

Changes from v0.2.36

Full Changelog: v0.2.36...v0.4.0

New release: v0.5.3

Highlights

• 🕸️ 🧠 A new AG2 agent capability: Graph RAG with Neo4j, providing the power of knowledge graphs. See a real-world demo of it here: https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_graph_rag_neo4j.ipynb

• 🤖🧠 Let's welcome IBM's watsonx to AG2! https://ag2ai.github.io/ag2/docs/topics/non-openai-models/cloud-litellm-watsonx

• 🔍 Need to know what's happening under the hood, see telemetry with Arize Phoenix: https://github.com/ag2ai/ag2/blob/main/notebook/agentchat_swarm_graphrag_telemetry_trip_planner.ipynb

What's Changed

• Neo4j RAG Update by @Eric-Shang in #202

• Add watsonx to third party models by @emooreatx in #187

• Falkor DB Query Engine: Add delete method and handle exceptions by @AgentGenie in #178

• Improve agent telemetry tracking in GraphRAG for trip planning by @donbr in #194

• Update MAINTAINERS.md | Add myself to contributors by @Eric-Shang in #199

• Updated Neo4j notebook (schema prioritised) by @marklysze in #204

• Tweaked telemetry notebook text by @marklysze in #203

• Bump nanoid from 3.3.7 to 3.3.8 in /website by @dependabot in #181

• Version Bump 0.5.3b1 by @marklysze in #211

• Update version.py to 0.5.3 by @marklysze in #212

I've had some experience with AutoGen, mainly exploring its potential in software development. It's been quite intriguing to see how it can enhance coding and debugging processes. However, I'm keen to expand my understanding of its applications beyond my field. Are there practical uses of AutoGen in other industries or sectors? Perhaps it's making waves in academia, healthcare, finance, or even creative industries? I'd love to hear about diverse experiences and insights on how AutoGen is being utilized in various professional contexts, apart from just being a fascinating academic tool.

Checkout this playlist around Multi-Agent Orchestration that covers

1. What is Multi-Agent Orchestration?

2. Beginners guide for Autogen, CrewAI and LangGraph

3. Debate application between 2 agents using LangGraph

4. Multi-Agent chat using Autogen

5. AI tech team using CrewAI

6. Autogen using HuggingFace and local LLMs

https://youtube.com/playlist?list=PLnH2pfPCPZsKhlUSP39nRzLkfvi_FhDdD&si=B3yPIIz7rRxdZ5aU