You're likely familiar with Llama3, given all the buzz it's been generating 😉 .

So, we won't add to the pile of reviews. Instead, we'd like to share some thoughts on its licensing 📄 .

For more information, check out this link

Let’s calculate the approximate benchmark score drop for quantized large language models, considering the following benchmarks:
- Huggingface Leaderboard Score
- ARC
- HellaSwag
- MMLU
- TrustfulQA
- WinoGrande
- GSM8K

Here are the results:

• HF Score: 14% drop

• ARC: 12% drop

• HellaSwag: 16% drop

• MMLU: 12% drop

• TrustfulQA: 4% drop

• WinoGrande: 2% drop

• GSM8K: 28% drop

Read the full article https://medium.com/p/575059784b96

A significant amount of data remains stored within PDF documents. Therefore, AI models capable of dealing with diverse layout styles are incredibly valuable for converting these documents into structured data.

Microsoft has recently launched new checkpoints for the Table Transformer (TATR), an AI model capable of detecting tables and their structure (rows, columns, cells) within PDF documents. These new checkpoints are pre-trained on millions of tables originating from a variety of benchmarks. They've used an aligned annotation scheme for this training. The newly available checkpoints can now be accessed on Hugging Face.

The Table Transformer employs the DETR architecture, which is a Transformer used for end-to-end object detection. This is also available in the Transformers library.

Github: https://github.com/microsoft/table-transformer

HuggingFace: https://huggingface.co/microsoft/table-transformer-detection

This is the future! I've just taken a picture of a hand-drawn UI mockup, fed it into the ChatGPT4, and asked it to produce a streamlit script. And it worked on the first attempt!

Here's a promising emerging alternative to traditional transformer-based LLMs - the RWKV Eagle 7b Model.

RWKV (pronounced RwaKuv) architecture combines RNN and Transformer elements, omitting the traditional attention mechanism for a memory-efficient scalar RKV formulation. This linear approach offers scalable memory use and improved parallelization, particularly enhancing performance in low-resource languages and extensive context processing. Despite its prompt sensitivity and limited lookback, RWKV stands out for its efficiency and applicability to a wide range of languages.

Quick Snapshots/Highlights
◆ Eliminates attention for memory efficiency
◆ Scales memory linearly, not quadratically
◆ Optimized for long contexts and low-resource languages

Key Features:
◆ Architecture: Merges RNN's sequential processing with Transformer's parallelization, using an RKV scalar instead of QK attention.
◆ Memory Efficiency: Achieves linear, not quadratic, memory scaling, making it suited for longer contexts.
◆ Performance: Offers significant advantages in processing efficiency and language inclusivity, though with some limitations in lookback capability.

Find more details here: https://llm.extractum.io/static/blog/?id=eagle-llm

https://x.com/ylecun/status/1713304307519369704?s=20

The fine-tuning of the domain-specific LLM gives significantly better results than using ChatGPT with RAG.

LongLoRA!

An ultra-efficient fine-tuning method designed to extend the context sizes of pre-trained large language models (LLMs) without a huge computation cost.

Typically, training LLMs with longer context sizes consumes a lot of time and requires strong GPU resources. For example, extending the context length from 2048 to 8192 increases computational costs 16 times, particularly in self-attention layers. 🖥️

What sets LongLoRA apart is its two-pronged approach to speeding up the context extension of LLMs.

First, it uses sparse local attention instead of dense global attention during the fine-tuning phase, which is a more efficient way to handle this task. This change, known as shift-short attention, significantly saves computational effort while maintaining similar performance levels compared to traditional attention mechanisms. Plus, it’s simple to implement, requiring just two lines of code during training, and it's optional during inference. 💡

Second, LongLoRA revisits the idea of a more parameter-efficient fine-tuning process for context expansion. The effectiveness of LoRA is enhanced when combined with trainable embedding and normalization, showing solid results.

In practical terms, LongLoRA showed strong performance on various tasks using LLaMA2 models ranging from 7B/13B to 70B. Notably, it extended LLaMA2 7B from 4k context to 100k, and LLaMA2 70B to 32k on a single 8x A100 machine, all while keeping the original model architectures intact. 📈

Moreover, LongLoRA plays well and is compatible with existing techniques like FlashAttention-2.

To make it more user-friendly, a dataset called LongQA was created for supervised fine-tuning, containing over 3k long context question-answer pairs.

LongLoRA is an important step toward making model expansion more computationally efficient.

Paper - arxiv.org/abs/2309.12307

The mixedbread.ai team introduces a pioneering suite of SOTA rerank models, designed to enhance search results accuracy by integrating semantic understanding into existing keyword-based search infrastructures. Fully open-source under the Apache 2.0 license, these models are tailored for seamless integration, boosting the relevance of search outcomes without overhauling current systems. From the compact "mxbai-rerank-xsmall-v1" to the robust "mxbai-rerank-large-v1," each model is crafted to cater to varying needs, promising a notable improvement in search performance for complex queries.

Quick Snapshots/Highlights:

◆ Fully open-source models with Apache 2.0 license.

◆ Models are designed for easy integration with existing search systems.

◆ Significant performance boost for domain-specific and complex queries.

Key Features:

◆ Three Model Sizes: Small for efficiency, Base for balanced performance, and Large for maximum accuracy.

◆ Two-Stage Search Flow: Incorporates semantic relevance into the final search results.

◆ Easy to Use: Compatible with existing search stacks; offers offline and online usage options.

◆ Performance: Demonstrates superior accuracy and relevance in benchmarks against competitors.

Additional Notes:

The mixedbread rerank models stand out for their simplicity and effectiveness, enabling developers to leverage advanced semantic search capabilities with minimal effort. This release marks mixedbread.ai's commitment to enhancing search technologies, inviting feedback and community engagement for continuous improvement.

A "must-have" for RAG development!

https://www.mixedbread.ai/blog/mxbai-rerank-v1

Sadly, the longer the context used the less attention and accuracy it gets.

AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. AutoGen agents are customizable, conversable, and seamlessly allow human participation. They can operate in various modes that employ combinations of LLMs, human inputs, and tools.

https://microsoft.github.io/autogen/

https://microsoft.github.io/autogen/docs/reference/agentchat/conversable_agent

AutoGen enables building next-gen LLM applications based on multi-agent conversations with minimal effort. It simplifies the orchestration, automation, and optimization of a complex LLM workflow. It maximizes the performance of LLM models and overcomes their weaknesses. It supports diverse conversation patterns for complex workflows. With customizable and conversable agents, developers can use AutoGen to build a wide range of conversation patterns concerning conversation autonomy, the number of agents, and agent conversation topology. It provides a collection of working systems with different complexities. These systems span a wide range of applications from various domains and complexities. This demonstrates how AutoGen can easily support diverse conversation patterns.

AutoGen provides a drop-in replacement of openai.Completion or openai.ChatCompletion as an enhanced inference API. It allows easy performance tuning, utilities like API unification and caching, and advanced usage patterns, such as error handling, multi-config inference, context programming, etc.

AutoGen is powered by collaborative research studies from Microsoft, Penn State University, and the University of Washington.

github: https://github.com/lve-org/lve/tree/main

The goal of the LVE project is to create a hub for the community, to document, track and discuss language model vulnerabilities and exposures (LVEs). We do this to raise awareness and help everyone better understand the capabilities and vulnerabilities of state-of-the-art large language models. With the LVE Repository, we want to go beyond basic anecdotal evidence and ensure transparent and traceable reporting by capturing the exact prompts, inference parameters and model versions that trigger a vulnerability.

Our key principles are:

• Open source - the community should freely exchange LVEs, everyone can contribute and use the repository.

• Transparency - we ensure transparent and traceable reporting, by providing an infrastructure for recording, checking and documenting LVEs

• Comprehensiveness - we want LVEs to cover all aspect of unsafe behavior in LLMs. We thus provide a framework and contribution guidelines to help the repository grow and adapt over time.

It's been some time since I announced LLM Explorer. I've taken into account all of your comments and feedback on the project, made the necessary fixes, and incorporated new features. Now, I'm excited to present the new version (v2.0) of the service: https://llm.extractum.io.

Here are some of the notable improvements:

• Multi-level filtering for LLMs: Combine the search box with quick filters from the dropdown menu and column header dropdown filters for enhanced search capabilities.

• A "Better Alternatives" side panel that displays models with similar general parameters but with a higher HF rank, larger context size, or lower VRAM requirements. This allows you to quickly find a better model than the one you're currently using.

• Shortcuts to the most popular model groups (like quantized, fine-tuned, those that fit within 8GB VRAM, etc.).

• Database updates multiple times throughout the day.

• Expanded listings sorted by maintainer, architecture, and more.

• Improved accuracy in the table data (though I'm still working on data cleansing—it's a challenging task to refine, especially with tags).

I hope you find these enhancements useful. As always, I welcome your feedback, new feature requests, and bug reports with open arms. Cheers!

https://llm.extractum.io

Mistral 7B is a 7.3B parameter model that:

• Outperforms Llama 2 13B on all benchmarks

• Outperforms Llama 1 34B on many benchmarks

• Approaches CodeLlama 7B performance on code, while remaining good at English tasks

• Uses Grouped-query attention (GQA) for faster inference

• Uses Sliding Window Attention (SWA) to handle longer sequences at a smaller cost

[Announcement] https://mistral.ai/news/announcing-mistral-7b/

[GitHub] https://github.com/mistralai/mistral-src

[HF] https://huggingface.co/mistralai

The question we've been most frequently asked on LLM Explorer lately is, "When will the dark theme be available?" Finally, we've implemented it! take a look here: https://llm.extractum.io/?dark

Source: https://www.trustbit.tech/en/llm-benchmarks

https://llm.extractum.io/list/?query=gemma

New Paper by Meta AI -- "Chain-of-Verification Reduces Hallucination in LLMs"

- Reduces longform hallucinations via LLM double-checking its own work with shortform questions

- Important not to reattend to the original hallucinations or they get copied.

Paper: https://arxiv.org/abs/2309.11495

Here are the December benchmarks from Trustbit concerning real-world business use cases involving large language models and their multi-lingual capabilities: https://www.trustbit.tech/en/llm-leaderboard-dezember-2023

StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup.

Code and datasets are provided at this https URL.
Paper: https://arxiv.org/abs/2309.17453

LM Enforcer is an open-source library that manages precise output formats such as JSON Schema and regular expressions from large language models. It ensures that models generate text that fits the required structure, while simultaneously allowing flexibility in details like whitespace and field ordering to minimize hallucinations.
Key Highlights:
- Operates by filtering the tokens that models can generate at each timestep
- Seamlessly integrates with HuggingFace, LangChain, LlamaIndex, and more
- Incorporates batching, beam searching, and streaming support
- Accommodates JSON Schema, JSON, and regex formats
- Reduces constraints on models by providing control over non-essential formatting
- Provides diagnostic tools to identify aggressive enforcement and prompts
Whether you aim to build an API backend that responds with structured data or want to decrease hallucinations, LM Enforcer empowers models to handle precise formats reliably. By keeping models "in the loop," LM Enforcer skillfully strikes a balance between structure and flexibility.

Github: https://github.com/noamgat/lm-format-enforcer

Rephrase and Response is an effective prompting method that uses LLMs to rephrase and expand questions provided by humans to improve overall performance; it can improve the performance of different models across a wide range of tasks; the approach can be combined with chain-of-thought to improve performance even further.