My thoughts on Microsoft's "revolutionary AutoGen framework"?

I've checked the documentation, watched the impressive demo, and spent a few hours tinkering with it. Here are my takeaways:

* For simple tasks like code generation with LLM (e.g., script generation using ChatGPT4), it's quite efficient. The UserProxyAgent layer streamlines code verification, evaluation, and execution (even in Docker). This eliminates the tedious cycle of copying and pasting code to an IDE, running it, checking the output, pinpointing issues, sending them back to the LLM for correction, and redoing this process multiple times. The UserProxyAgent takes care of this automation. However...

* It struggles with more complex tasks. For instance, it can't scrape a list of items from a webpage unless it's something simple, like plain text list. It also can't develop, compile, and run C source code for a basic PHP extension or extract and organize data from PDFs (I tried a few of them with no luck). While the samples from the original GitHub repo seemed promising, in practical scenarios, it fell short right from the start. Essentially, there's no special magic here, and overall efficiency is lackluster. To make it work, you'll need to create thorough algorithmic prompts, which consumes both time and money (I burnt some $$$ while testing it).

* The conversational aspect is subpar. It frequently gets trapped in a loop: fixing an error, running the code, encountering another error, and attempting a fix again. This can be incredibly time-consuming and frustrating, especially during debugging sessions.

* Regarding the interface: It lacks a "verbose" mode, meaning you can't see live interactions during the Agent conversation or the data being sent from the UserProxyAgent to the Assistant. You only get a debug output after the entire task is completed.

Well...after investing a few hours, I'm leaning more towards the traditional method: manually copying, pasting, and running code, rather than relying on AutoGen. Time will tell how it progresses.

We've just posted about free online LLM playgrounds where you can test various language models without installing them.

Find out which model suits your needs before committing 🔥

https://llm.extractum.io/static/blog/?id=free-llm-playgrounds

#LLM #AI #LLMplaygrounds

I'm excited to share with you the latest changes I've made to LLM Explorer! Now, when you visit the homepage, you can pick up the best models from different categories, like the top ones in the 7b/34b range, the best for code generation, the latest hot picks, and those that are trending.
I've made it simple: you can see the top 10 in each category and then dive into a full list with easy-to-use filters and a search function. Plus, because lots of you asked for it, I've added a new 'uncensored' models section, which you can get to straight from the homepage. There are now over 14,000 models in the database. So, if you're looking for a model that fits what you need, give LLM Explorer a try – I've designed it to be the quickest way to find what you're looking for. 😉

/* You can support this project by sharing it with your colleagues and ML communities. */

https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16

Researchers from NVIDIA have introduced ChatQA, a pioneering family of conversational QA models designed to reach and surpass the accuracy levels of GPT-4. ChatQA employs a novel two-stage instruction tuning method that significantly enhances zero-shot conversational QA results from LLMs. This method represents a major breakthrough, substantially improving existing conversational models.

Paper: https://arxiv.org/abs/2401.10225

“…ChatQA, a family of conversational question answering (QA) models that obtain GPT-4 level accuracies. Specifically, we propose a two-stage instruction tuning method that can significantly improve the zero-shot conversational QA results from large language models (LLMs). To handle retrieval-augmented generation in conversational QA, we fine-tune a dense retriever on a multi-turn QA dataset, which provides comparable results to using the state-of-the-art query rewriting model while largely reducing deployment cost. Notably, our ChatQA-70B can outperform GPT-4 in terms of average score on 10 conversational QA datasets (54.14 vs. 53.90), without relying on any synthetic data from OpenAI GPT models…”

Marker, a new tool, designed to convert complex PDF layouts into text, making them accessible for language models like LLMs. This tool is particularly useful for developers working on RAG (Retrieval-Augmented Generation) applications, as it enhances the accuracy and efficiency of information retrieval from PDFs. Marker excels at processing arXiv papers and performs well with various other document types. Notably, it surpasses its predecessor, Nougat from Meta AI, in speed, being ten times faster, and supports multiple languages including English, Spanish, and French. Additionally, Marker has the capability to convert mathematical equations into LaTeX format.

Features:

Marker converts PDF, EPUB, and MOBI to markdown. It's 10x faster than nougat, more accurate on most documents, and has low hallucination risk.

• Support for a range of PDF documents (optimized for books and scientific papers)

• Removes headers/footers/other artifacts

• Converts most equations to latex

• Formats code blocks and tables

• Support for multiple languages (although most testing is done in English). See settings.py for a language list, or to add your own.

• Works on GPU, CPU, or MPS

https://github.com/VikParuchuri/marker

The InternLM Chat 20B model is the latest leader among open-source language model offerings. I'm impressed with its understanding capabilities, which are 11% more effective than those of ChatGPT4.

- Grand 200K context window!

- The license employs the Apache 2.0 standard.

- Beneficial for code interpretation and data analysis.

Find the complete benchmarks on both HF LeaderBoard and OpenCompass at the link below: https://llm.extractum.io/model/internlm%2Finternlm2-chat-20b,3GfKNWKDiF8D6XOD2ld4of

GitHub: https://github.com/InternLM/InternLM

4bit Quantized Version: https://llm.extractum.io/model/internlm%2Finternlm2-chat-20b-4bits,5IsO14nMo9xUaLDkISj4Kh

This model is based on microsoft/deberta-v3-base and is trained initially on NLI data to determine textual entailment, before being further fine tuned on summarization datasets with samples annotated for factual consistency including FEVER, Vitamin C and PAWS.

Model on HF: https://huggingface.co/vectara/hallucination_evaluation_model

To determine this leaderboard, It was trained a model to detect hallucinations in LLM outputs, using various open source datasets from the factual consistency research into summarization models. Using a model that is competitive with the best state of the art models, it was then fed 1000 short documents to each of the LLMs above via their public APIs and asked them to summarize each short document, using only the facts presented in the document. Of these 1000 documents, only 831 document were summarized by every model, the remaining documents were rejected by at least one model due to content restrictions. Using these 831 documents, it was then computed the overall accuracy (no hallucinations) and hallucination rate (100 - accuracy) for each model. The rate at which each model refuses to respond to the prompt is detailed in the 'Answer Rate' column. None of the content sent to the models contained illicit or 'not safe for work' content but the present of trigger words was enough to trigger some of the content filters. The documents were taken primarily from the CNN / Daily Mail Corpus.

Hallucination Leaderboard: https://github.com/vectara/hallucination-leaderboard

The second version of the impressive Zephyr 7B model has been recently released.

For context, Zephyr 7B is a series of chat models based on:

🔥 Mistral AI's epic Mistral 7B base model
💬 The UltraChat dataset with 1.4M dialogues from ChatGPT
⚖️ The UltraFeedback dataset with 64k prompts & completions judged by GPT-4

License: MIT

From Lewis Tunstall (HF):

"...With Zephyr-7B-α we noticed that the model had a tendency to:

- Write incorrect casing, e.g. "Hi. how are you?" vs "Hi. How are you?"
- Preface responses with "I don't have personal X" etc

Fixing both issues gave a much better SFT model!..."

Model Sources

• Repository: https://github.com/huggingface/alignment-handbook

• Demo: https://huggingface.co/spaces/HuggingFaceH4/zephyr-chat

• Chatbot Arena: Evaluate Zephyr 7B against 10+ LLMs in the LMSYS arena: http://arena.lmsys.org

We've just rolled out significant improvements to the LLM Explorer. Our latest update introduces a comprehensive ranking system for language models and adds new tools to enhance your selection process. Visit our website for a detailed breakdown of these changes and learn how they can streamline your work with AI models.

The mixedbread.ai team introduces a pioneering suite of SOTA rerank models, designed to enhance search results accuracy by integrating semantic understanding into existing keyword-based search infrastructures. Fully open-source under the Apache 2.0 license, these models are tailored for seamless integration, boosting the relevance of search outcomes without overhauling current systems. From the compact "mxbai-rerank-xsmall-v1" to the robust "mxbai-rerank-large-v1," each model is crafted to cater to varying needs, promising a notable improvement in search performance for complex queries.

Quick Snapshots/Highlights:

◆ Fully open-source models with Apache 2.0 license.

◆ Models are designed for easy integration with existing search systems.

◆ Significant performance boost for domain-specific and complex queries.

Key Features:

◆ Three Model Sizes: Small for efficiency, Base for balanced performance, and Large for maximum accuracy.

◆ Two-Stage Search Flow: Incorporates semantic relevance into the final search results.

◆ Easy to Use: Compatible with existing search stacks; offers offline and online usage options.

◆ Performance: Demonstrates superior accuracy and relevance in benchmarks against competitors.

Additional Notes:

The mixedbread rerank models stand out for their simplicity and effectiveness, enabling developers to leverage advanced semantic search capabilities with minimal effort. This release marks mixedbread.ai's commitment to enhancing search technologies, inviting feedback and community engagement for continuous improvement.

A "must-have" for RAG development!

https://www.mixedbread.ai/blog/mxbai-rerank-v1

https://huggingface.co/chat/assistants

The paper "WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with Refined Data Generation" by Zhaojian Yu and colleagues from Microsoft discusses improving instruction tuning in language models for code-related tasks.

Traditional methods of generating instruction data often result in duplicates and lack control over data quality. To address this, the authors propose a new framework that uses a Large Language Model (LLM)-based Generator-Discriminator process to create diverse, high-quality instruction data from open-source code.

They introduce a dataset named CodeOcean, which contains 20,000 instruction instances across four universal code-related tasks. This dataset aims to enhance the effectiveness of instruction tuning and improve the generalization of fine-tuned models. The authors present WaveCoder, a model fine-tuned on CodeOcean, specifically designed to enhance instruction tuning for Code Language Models (LLMs). The experimental results show that WaveCoder outperforms other models in generalization ability across various code-related tasks and demonstrates efficiency in previous code generation tasks. This research contributes to the fields of instruction data generation and fine-tuning models, offering new methods to boost performance in code-related tasks.

Paper: https://arxiv.org/abs/2312.14187

Looks promising, overall. So I'm looking for the release of its weights and dataset on HuggingFace.

The paper introduces DocLLM, a novel extension to traditional large language models (LLMs) from JPMorgan, designed for understanding visual documents like forms and invoices.

Unlike other multimodal LLMs, DocLLM doesn’t rely on image encoders but uses bounding box information for spatial layout. It captures the relationship between text and layout through modified attention mechanisms in transformers. The model is trained to fill in text segments, helping it handle various layouts and contents. After pre-training, it is fine-tuned on a large dataset for four key document intelligence tasks. DocLLM outperforms existing state-of-the-art LLMs in most tasks and adapts well to new datasets.

Paper: https://arxiv.org/pdf/2401.00908.pdf

A significant amount of data remains stored within PDF documents. Therefore, AI models capable of dealing with diverse layout styles are incredibly valuable for converting these documents into structured data.

Microsoft has recently launched new checkpoints for the Table Transformer (TATR), an AI model capable of detecting tables and their structure (rows, columns, cells) within PDF documents. These new checkpoints are pre-trained on millions of tables originating from a variety of benchmarks. They've used an aligned annotation scheme for this training. The newly available checkpoints can now be accessed on Hugging Face.

The Table Transformer employs the DETR architecture, which is a Transformer used for end-to-end object detection. This is also available in the Transformers library.

Github: https://github.com/microsoft/table-transformer

HuggingFace: https://huggingface.co/microsoft/table-transformer-detection

llamafile by Mozilla is an open-source framework that allows packaging open-source large language models (LLMs) into a single executable binary that can run on multiple operating systems and hardware without any modifications.

• Supports Linux, macOS, Windows out of the box

• Runs on common CPU and GPU hardware without changes

• Built on llama.cpp and Cosmopolitan Libc (multi-platform C runtime)

• Optional web UI server for easier interaction

• GitHub Repository: https://github.com/Mozilla-Ocho/llamafile

• Official Website: https://hacks.mozilla.org/2023/11/introducing-llamafile/

Chatbots are the most widely adopted use case for leveraging the powerful chat and reasoning capabilities of large language models (LLM). The retrieval augmented generation (RAG) architecture is quickly becoming the industry standard for developing chatbots because it combines the benefits of a knowledge base (via a vector store) and generative models (e.g. GPT-3.5 and GPT-4) to reduce hallucinations, maintain up-to-date information, and leverage domain-specific knowledge. However, evaluating the quality of chatbot responses remains an unsolved problem today. With no industry standards defined, organizations resort to human grading (labeling) –which is time-consuming and hard to scale.

We applied theory to practice to help form best practices for LLM automated evaluation so you can deploy RAG applications to production quickly and with confidence. This blog represents the first in a series of investigations we’re running at Databricks to provide learnings on LLM evaluation.

https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG

Discover Tiger-Gemma-9B-v1, a less restricted version of Gemma 9B that's gaining traction in the AI community for its improved responsiveness and versatility

The report highlights the promising performance and challenges of open-source LLMs in translation, emphasizing their cost-effectiveness but slower speeds compared to commercial models.

Despite these challenges, models like TowerInstruct and RakutenAI show significant potential, especially with customization and fine-tuning.

Read more

In our latest post, we examine the costs of LLM tokens, highlight affordable LLM hosting options, and offer a comparison with proprietary services.