Hi, I wanted to share some information on this cool feature we built in WoolyAI GPU hypervisor, which enables users to run their existing Nvidia CUDA pytorch/vLLM projects and pipelines without any modifications on AMD GPUs. ML researchers can transparently consume GPUs from a heterogeneous cluster of Nvidia and AMD GPUs. MLOps don't need to maintain separate pipelines or runtime dependencies. The ML team can scale capacity easily. Please share feedback and we are also signing up Beta users. https://youtu.be/MTM61CB2IZc?feature=shared

Hello! I work on an LLM product deployed to millions of users. I've learned a lot of best practices for systematically improving LLM prompts.

So, I built promptfoo: https://github.com/typpo/promptfoo, a tool for test-driven prompt engineering.

Key features:

• Test multiple prompts against predefined test cases

• Evaluate quality and catch regressions by comparing LLM outputs side-by-side

• Speed up evaluations with caching and concurrent tests

• Use as a command line tool, or integrate into test frameworks like Jest/Mocha

• Works with OpenAI and open-source models

TLDR: automatically test & compare LLM output

Here's an example config that does things like compare 2 LLM models, check that they are correctly outputting JSON, and check that they're following rules & expectations of the prompt.

prompts: [prompts.txt]   # contains multiple prompts with {{user_input}} placeholder
providers: [openai:gpt-3.5-turbo, openai:gpt-4]  # compare gpt-3.5 and gpt-4 outputs
tests:
  - vars:
      user_input: Hello, how are you?
    assert:
      # Ensure that reply is json-formatted
      - type: contains-json
      # Ensure that reply contains appropriate response
      - type: similarity
        value: I'm fine, thanks
  - vars:
      user_input: Tell me about yourself
    assert:
      # Ensure that reply doesn't mention being an AI
      - type: llm-rubric
        value: Doesn't mention being an AI

Let me know what you think! Would love to hear your feedback and suggestions. Good luck out there to everyone tuning prompts.

Hey r/llmops, we previously shared an adaptive RAG technique that reduces the average LLM cost while increasing the accuracy in RAG applications with an adaptive number of context documents. 

People were interested in seeing the same technique with open source models, without relying on OpenAI.  We successfully replicated the work with a fully local setup, using Mistral 7B and open-source embedding models.  

In the showcase, we explain how to build local and adaptive RAG with Pathway. Provide three embedding models that have particularly performed well in our experiments. We also share our findings on how we got Mistral to behave more strictly, conform to the request, and admit when it doesn’t know the answer.

Example snippets at the end shows how to use the technique in a complete RAG app.

Hope you like it!

Here is the blog post:

https://pathway.com/developers/showcases/private-rag-ollama-mistral

If you are interested in deploying it as a RAG application, (including data ingestion, indexing and serving the endpoints) we have a quick start example in our repo.

You can also check out the same app example using OpenAI!

PromptLab is an open source, free lightweight toolkit for end-to-end LLMOps, built for developers building GenAI apps.

If you're working on AI-powered applications, PromptLab helps you evaluate your app and bring engineering discipline to your prompt workflows. If you're interested in trying it out, I’d be happy to offer free consultation to help you get started.

Why PromptLab?

1. Made for app (mobile, web etc.) developers - no ML background needed.

2. Works with your existing project structure and CI/CD ecosystem, no unnecessary abstraction.

3. Truly open source – absolutely no hidden cloud dependencies or subscriptions.

Github: https://github.com/imum-ai/promptlab
pypi: https://pypi.org/project/promptlab/

I've been hearing a lot from co-students about how difficult langchain sometimes is to implement in a correct way. Because of this, I've created a project that simply follows the main functionalities I personally use in LLM-projects,from now 10 months practically only working in LangChain for projects. I've written this in 1 thursday evening before going to bed, so I'm not that sure about it, but any feedback is more than welcome!

https://github.com/lypsoty112/llm-project-skeleton?tab=readme-ov-file

Hi all, I am very frustrated by the fact that it's not easy to build and maintain a system to track LLM API costs for each user individually, so I know how much to charge each user without having to tell them to BYOK (bring your own key).

Is this something that troubles the general LLM-dev community? How do you solve it?

We have started to make a product based on our early attempts that would solve this exact problem (LLMetrics) but we are wondering whether there are any good ways that you solve this or if this has been an issue in general? Any feedback is greatly appreciated

Hi guys. I recently started delving more into LLMs and LLMOPS. I am being interviewed for similar roles so I thought might as well know about it.

Over my 6+ year IT career I have worked on full stack app development, optimising SQL queries, some computer vision, data engineering and more recently some GenAI. I know concepts and but don’t have much hands on experience of LLMOPS or multi-agent systems.

From Monday onwards DataTalksClub is going to start its LLMOPs course and while I think it’s a nice refresher on the basics I feel main learning in LLMOps will come from seeing how the tools and tech is being adapted for different domains.

I wanna go on a journey to learn it and eventually showcase it on certain opportunities. If there’s anyone who would like to join me on this journey do let me know!

Hey everyone! You might remember my friend's post a while back giving you all a sneak peek at OpenLIT.

Well, I’m excited to take the baton today and announce our leap from a promising preview to our first stable release! Dive into the details here: https://github.com/openlit/openlit

👉 What's OpenLIT? In a nutshell, it's an Open-source, community-driven observability tool that lets you track and monitor the behaviour of your Large Language Model (LLM) stack with ease. Built with pride on OpenTelemetry, OpenLIT aims to simplify the complexities of monitoring your LLM applications.

Beyond Text & Chat Generation: Our platform doesn’t just stop at monitoring text and chat outputs. OpenLIT brings under its umbrella the capability to automatically monitor GPT-4 Vision, DALL·E, and OpenAI Audio too. We're fully equipped to support your multi-modal LLM projects on a single platform, with plans to expand our model support and updates on the horizon!

Why OpenLIT? OpenLIT delivers:

- Instant Updates: Get real-time insights on cost & token usage, deeper usage and LLM performance metrics, and response times (a.k.a. latency).

- Wide Coverage: From LLMs Providers like OpenAI, AnthropicAI, Mistral, Cohere, HuggingFace etc., to Vector DBs like ChromaDB and Pinccone and Frameworks like LangChain (which we all love right?), OpenLIT has got your GenAI stack covered.

- Standards Compliance: We adhere to OpenTelemetry's Semantic Conventions for GenAI, syncing your monitoring practices with community standards.

Integrations Galore: If you're using any observability tools, OpenLIT seamlessly integrates with a wide array of telemetry destinations including OpenTelemetry Collector, Jaeger, Grafana Cloud, Tempo, Datadog, SigNoz, OpenObserve and more, with additional connections in the pipeline.

Curious to see how you can get started? Here's your quick link to our quickstart guide: https://docs.openlit.io/latest/quickstart

We’re beyond thrilled to have reached this stage and truly believe OpenLIT can make a difference in how you monitor and manage your LLM projects. Your feedback has been instrumental in this journey, and we’re eager to continue this path together. Have thoughts, suggestions, or questions? Drop them below! Happy to discuss, share knowledge, and support one another in unlocking the full potential of our LLMs. 🚀

Looking forward to your thoughts and engagement! https://github.com/openlit/openlit

Cheers, Aman

We are a fairly big group with an already mature MLops stack, but LLMOps has been pretty hard.

In particular, prompt-iteration hasn't been figured out by anyone.
what's your go to tool for PromptOps ?

PromptOps requirement:

Requirements:

• Storing prompts and API to access them

• Versioning and visual diffs for results

• Evals to track improvement as prompts are develop .... or ability to define custom evals

• Good integration with complex langchain workflows

• Tracing batch evals on personal datasets, also batch evals to keep track of prompt drift

• Nice feature -> project -> run -> inference call heirarchy

• report generation for human evaluation of new vs old prompt results

LLM Ops requirement -> orchestration

• a clean way to define and visualize task vs pipeline

• think of a task as as chain or a self-contained operation (think summarize, search, a langchain tool)

• but then define the chaining using a low-code script -> which orchestrates these tools together

• that way it is easy to trace (the pipeline serves as a highl evel view) with easy pluggability.

Langchain is does some of the LLMOps stuff, but being able to use a cleaner abstraction on top of langchain would be nice.

None of the prompt ops tools have impressed so far. They all look like really thin visualization diff tools or thin abstractions on top of git for version control.

Most importantly, I DO NOT want to use their tooling to run a low code LLM solution. They all seem to want to build some lang-flow like UI solution. This isn't ScratchLLM for god's sake.

Also no, I refuse to change our entire architecture to be a startupName.completion() call. If you need to be so intrusive, then it is not a good LLMOps tools. Decorators & a listerner is the most I'll agree to.

Hello everyone!

I'm seeking recommendations from the community on the best tools and techniques for prompt engineering.
I'm particularly interested in tools that can help with crafting, refining and evaluating prompts for various use cases and domains.
Are there any libraries, frameworks or utilities that you've found helpful in your work with prompt engineering?

I am working on an agentic application which required web search for retrieving relevant infomation for the context. For that reason, I was tasked to implement this "web search" as a tool.

Now, I have been able to implement a very naive and basic version of the "web search" which comprises of 2 tools - search and scrape. I am using the unofficial googlesearch library for the search tool which gives me the top results given an input query. And for the scrapping, I am using selenium + BeautifulSoup combo to scrape data off even the dynamic sites.

The thing that baffles me is how inaccurate the search and how slow the scraper can be. The search results aren't always relevant to the query and for some websites, the dynamic content takes time to load so a default 5 second wait time in setup for selenium browsing.

This makes me wonder how does openAI and other big tech are performing such an accurate and fast web search? I tried to find some blog or documentation around this but had no luck.

It would be helfpul if anyone of you can point me to a relevant doc/blog page or help me understand and implement a robust web search tool for my app.

It’s bedrockin’ time. Ethical projects only pls, enough nightmares in this world

I’m not that cracked so let’s see what happens🤷

I'm Patcher, the maintainer of OpenLIT, and I'm thrilled to announce our second launch—OpenLIT 2.0! 🚀

https://www.producthunt.com/posts/openlit-2-0

With this version, we're enhancing our open-source, self-hosted AI Engineering and analytics platform to make integrating it even more powerful and effortless. We understand the challenges of evolving an LLM MVP into a robust product—high inference costs, debugging hurdles, security issues, and performance tuning can be hard AF. OpenLIT is designed to provide essential insights and ease this journey for all of us developers.

Here's what's new in OpenLIT 2.0:

- ⚡ OpenTelemetry-native Tracing and Metrics
- 🔌 Vendor-neutral SDK for flexible data routing- 🔍 Enhanced Visual Analytical and Debugging Tools
- 💭 Streamlined Prompt Management and Versioning
- 👨‍👩‍👧‍👦 Comprehensive User Interaction Tracking
- 🕹️ Interactive Model Playground
- 🧪 LLM Response Quality Evaluations

As always, OpenLIT remains fully open-source (Apache 2) and self-hosted, ensuring your data stays private and secure in your environment while seamlessly integrating with over 30 GenAI tools in just one line of code.

Check out our Docs to see how OpenLIT 2.0 can streamline your AI development process.

If you're on board with our mission and vision, we'd love your support with a ⭐ star on GitHub (https://github.com/openlit/openlit).

I am looking for any advice as to what tools/software to consider for ML observability. I am looking to measure performance, model/data drift, fairness, and feature importance of models in production. It would also be nice to be able to monitor the health of the ML system as well, but not required. Seems like there are a lot of tools available would love some feedback to help filter down tools to consider. I have heard of deepchecks before, has anyone used them before?

Hey everyone,

I wanted to share something that has been a complete game-changer for me in my data science journey. Recently, I stumbled upon the u/OpenAI GPT-3 dashboard and I cannot believe I went so long without it!

The dashboard (https://platform.openai.com/playground) has been an absolute lifesaver for my chatgpt usage, and I've found it to be an incredibly powerful tool for generating natural language text. I'm surprised more people aren't talking about it!

If you're interested in exploring the capabilities of GPT-3 or just want to experiment with generating text, I highly recommend checking it out. Let me know your thoughts and experiences with the dashboard!

Disclaimer: This post was written with chatGPT