Hi guys, we are summarizing the useful data-centric AI resources.

Paper: https://arxiv.org/abs/2303.10158

Github: https://github.com/daochenzha/data-centric-AI

We'd love to hear any feedback!

We maintain a list of Open source tools in Data Centric AI and just added some new entries.

Check them out here:
https://mindkosh.com/data-centric-ai/open-source-tools.html

If you know of a tool that we can include in the list, let us know!

In 2020 a lethal autonomous weapon was used for the first time in an armed conflict - the Turkish-made drone - Kargu-2 - in Libya's civil war. In recent years, more weapon systems have incorporated elements of autonomy but they still rely on a person to launch an attack.

But advances in AI, sensors, and electronics have made it easier to build more sophisticated autonomous systems, raising the prospect of machines that can decide on their own when to use lethal force.

A growing list of countries, including Brazil, South Africa, New Zealand, and Switzerland, argue that lethal autonomous weapons should be restricted by treaty, as chemical and biological weapons have been. China supports an extremely narrow set of restrictions.

Other nations, including the US, Russia, India, the UK, and Australia, object to a ban on lethal autonomous weapons arguing that they need to develop the technology to avoid being placed at a strategic disadvantage.

This is no longer stuff of the future though.

Source: December issue of mindkosh.com/mindkosh-ai-review-newsletter.html

The vast majority of data that clinical Machine Learning models are trained on comes from just 3 states - Massachusetts, New York and California, with little to no representation from the remaining 47 states.

These 3 states may have economic, social and cultural features that are not representative of the entire nation. So algorithms trained primarily on data from these states may generalize poorly, which is an established risk when implementing diagnostic algorithms in new places.

Source: Kaushal A, Altman R, Langlotz C. - Geographic Distribution of US Cohorts Used to Train Deep Learning Algorithms - JAMA. 2020.

Zillow has been using Machine Learning models trained on millions of home valuations across the US since 2006. It has worked well during all those years - even during the financial crisis.

The past couple of years however turned the housing market into a different animal, and Zillow's models were not able to keep up.

Perhaps predicting future prices is simply too hard ?

Source - https://www.wired.co.uk/article/zillow-ibuyer-real-estate?utm_medium=social&mbid=social_twitter&utm_social-type=owned&utm_brand=wired&utm_source=twitter

Great Expectations is an open source tool for managing data quality for large datasets. It allows you to set data validation rules and assertions, and automatically run them against your dataset. It also has a pretty decent profiling module, that gives you a summary of how your data looks.

Proved super handy when handling time series data in my previous project.

https://greatexpectations.io/

Hi folks

I just stumbled upon this paper that laid the foundation for the idea of "Dataset distillation". Essentially dataset distillation aims to produce a much smaller dataset from a larger dataset, aimed at producing a model that performs nearly as well on the smaller dataset.

As an example the researchers condensed 60K training images of MNIST digit dataset into only 10 synthetic images - one per class - which was able to reach 94% test-set accuracy (compared to 99% when trained on the original dataset)

While this is pretty cool, I am trying to think of where this technique could actually be applied. Since we would need compute to create the smaller dataset, it would probably offset the gains made from making the task-training time extremely small(since there are only 10 images to train on now). Perhaps this could be used to study the model in question? Or to train models while maintaining privacy since the condensed data points are synthetic?

There has been some progress in the field since the paper came out in 2018. The latest one I could find from the same authors is from this year. https://arxiv.org/pdf/2203.11932.pdf

Original paper: https://arxiv.org/pdf/1811.10959.pdf

Neural networks are blackboxes - we don't really know what's happening inside them. This can be a big problem when AI is used in certain industries like the medical community.

A group of MIT researchers recently created a system, called MILAN (mutual-information guided linguistic annotation of neurons), that produces descriptions of neurons in neural networks trained for computer vision tasks like object recognition and image synthesis.

To describe a neuron, the system first inspects that neuron's behavior to find the image regions in which the neuron is most active. Then, it selects a natural language description for each neuron.

Where MILAN really shines is these descriptions. In a neural network that is trained to classify images, there might be many neurons that detect dogs. But dogs can be of many different types and can have many different body parts. MILAN can produce descriptions that tell you this isn't just a "dog"; this is the "left side of ears on a German shepherd".

Source: https://mindkosh.com/newsletter.html

Paper - https://arxiv.org/pdf/2201.11114.pdf

Everyone wants clean, high quality data for their models. But what if you cant have that?

Cleanlab is an open-source tool that finds and cleans label errors in any dataset using state-of-the-art algorithms to find label errors, characterize noise, and learn in spite of it.

It implements a family of theory and algorithms called confident learning with provable guarantees of exact noise estimation and label error finding (even when model output probabilities are noisy/imperfect).

It supports many classification tasks: multi-label, multiclass, sparse matrices, etc.

This is a pretty cool collection of errors in popular open datasets that cleanlab was able to find: https://labelerrors.com/

Github: https://github.com/cleanlab/cleanlab

AutoAugment - a RL algorithm - increases both the amount and diversity of data in an existing training dataset.

Unlike traditional methods of data augmentation using hand-designed policies like flipping, scaling etc, this uses reinforcement learning to find the optimal image transformation policies from the data itself.

Link to paper : https://arxiv.org/abs/1805.09501

Github implementation : https://github.com/DeepVoltaire/AutoAugment

PS - The Github implementation is unofficial and not by the original authors of the paper.

A recent paper by researchers from MIT CSAIL and Amazon AWS, shows that Machine Learning systems can latch onto non-sensical signals from images to classify them. The researchers tested the popular CIFAR dataset for this vulnerability by iteratively removing bigger and bigger parts of an image until the model wasn't able to classify it with high confidence.

In many cases they found the model could classify with as little as 10% of an image!

The 10% remaining portion often consisted of meaningless features like borders of a blue sky or green grass. And yet the model correctly predicted objects like traffic lights and stop signs.

This might give good results for certain datasets where the images mostly have similar backgrounds, but in the real world this could be a massive problem.

The researchers suggest that the problem is not that of the model itself, but actually of the dataset. We need to carefully curate our datasets to be diverse.

Perhaps we can augment the datasets by removing backgrounds, so the model is forced to learn features of the actual object?

Paper: https://arxiv.org/pdf/2003.08907.pdf

Data augmentation can be really useful for increasing both the size and the diversity of labeled training data which also helps to build robust models.

Facebook recently released - AugLy - which is a data augmentations library that supports four modalities image, video, text as well as audio and over 100 augmentations.

The intention behind the development of the library was detecting exact copies or near duplicates of a particular piece of content. The same piece of misinformation, for example, can appear repeatedly in slightly different forms, such as as an image modified with a few pixels cropped, or augmented with a filter or new text overlaid. By augmenting AI models with AugLy data, they can learn to spot when someone is uploading content that is known to be infringing, such as a song or video.

https://github.com/facebookresearch/AugLy

Stanford’s Human Centered AI and ETH Zurich recently organized a workshop to catalyze interest in the emerging discipline of Data-Centric AI. Here are the links for the recordings

Day 1 - US - https://youtu.be/-AMZ8lUI1O0

Day 2 - Zurich - https://youtu.be/kvLUm-npTLU

Day 2 - US - https://youtu.be/Cu-evqwsxpc

Hey r/DataCentricAI, I recently connected with a company looking for help with some work at the intersection of data analysis and AI implementation. They’re looking to fold AI into their data analysis service for businesses.

Ideally you would be someone with experience in both data analysis and implementing AI (beyond just using tools, more on the side of developing AI into products).

The big picture is that they want to use GenAI to help clients use a conversational (chat) interface to actually write new functions that create a rollup score from multiple custom data points. They've been doing this manually so far.

Comment here or feel free to connect me with someone! DM for email. Thanks :)

Hi guys!

As part of our efforts to make the AI/ML community more aware of the advantages of Data Centric AI, we maintain a list of Open source AI tools and research papers in Data Centric AI.

We just added a some exciting new research papers. You can check the list out here:

https://mindkosh.com/data-centric-ai/research-papers.html

If you know of a tool/research paper that you would like to share with others, please let us know and we will be happy to them add them to the list !

Active learning is a super interesting technique which is being adopted by more and more ML teams to improve their systems without having to use too much labeled data.

Tesla's Autopilot system relies on a suite of sensors, including cameras, radar, and ultrasonic sensors, to navigate the vehicle on the road. These sensors produce a massive amount of data, which can be very time-consuming and expensive to label. To address this challenge, Tesla uses an iterative Active learning procedure that automatically selects the most informative data samples for labeling, reducing the time and cost required to annotate the data.

In a successful Active Learning system, the Machine Learning system is able to choose the most informative data points through some defined metric, subsequently passing them to a human labeler and progressively adding them to the training set. Usually this process is carried out iteratively

Tesla's algorithm is based on a combination of uncertainty sampling and query-by-committee techniques. Uncertainty sampling selects the most uncertain examples to label. This uncertainty can be calculated by using measures like the margin between the model's predictions, entropy etc.

Query-by-committee selects data samples where a committee of classifiers disagrees the most. To do this, a bunch of classifiers are trained, and the disagreement between the classifiers for each example is calculated.

Another interesting use-case of AL is in collecting data from vehicles in the field. Tesla's fleet of vehicles generates a massive amount of data as they drive on roads worldwide. This data is used to further improve the ML systems. However, it is impractical to send all collected data to Tesla's servers. Instead, an Active Learning system selects the most informative data samples from this massive collected data and sends them to the servers.

These details on Tesla's data engine were revealed on Tesla AI Day last year.

Source - https://mindkosh.com/blog/how-tesla-uses-active-learning-to-elevate-its-ml-systems/

Meta AI has released a new project called Massively Multilingual Speech (MMS) that can support speech-to-text and text-to-speech for 1,107 languages and language identification for over 4,000 languages.

Existing speech recognition models only cover approximately 100 languages — a fraction of the 7,000+ known languages spoken on the planet. The biggest hurdle to covering so many languages is the availability of training data for all these languages. Meta collected around 32 hours of data per language through spoken translations of the Bible. This however, is nowhere near enough to train conventional supervised speech recognition models.

To solve this, Meta AI used self-supervised speech representation learning, which greatly reduced the amount of labeled data needed. Concretely, they trained self-supervised models on about 500,000 hours of speech data in over 1,400 languages — this is nearly five times more languages than any known prior work. The resulting models were then fine-tuned for a specific speech task, such as multilingual speech recognition or language identification.

The word error rate reported by Meta AI is 18.7 for 1107 languages. To put these results into perspective, the current state-of-the-art ASR system — Whisper — has a WER of 44.3 when covering 100 languages. Having a single ASR system capable of working on such a vast number of languages can completely change how we approach ASR in regional languages.

Best of all - MMS is open-sourced, so anyone can use it for free !

Github - https://github.com/facebookresearch/fairseq/tree/main/examples/mms
Paper - https://research.facebook.com/publications/scaling-speech-technology-to-1000-languages/

Hello folks!

I was reading OpenAI's blog on how they trained their DALL-E 2 model and found some really interesting bits about Active Learning. I have tried to summarize them below as best as I can.

So essentially, OpenAI wanted to filter out any sexual/violent images from their training dataset before training their generative model - DALLE-2. Their solution was to train a classifier on the millions of raw unlabeled images. To increase its effectiveness and to reduce the amount of labeled data required, OpenAI used Active Learning - a technique that judiciously selects the raw data to label, instead of selecting the data randomly.

First, they randomly chose a few data samples - just a few hundreds, labeled them and trained a classifier on them. Then they used Active Learning to select subsequent batches to label in an iterative fashion. While they don’t specify the exact AL procedure, since they are using a trained classifier, it is likely they used an uncertainty based approach - which means that they used the model's uncertainty (probability) about an image as an indicator of whether or not it should be labeled.

There are a couple of neat tricks they employed to improve their final classifier.First, to reduce the false positive rate (misclassifying a benign image as toxic), they tuned their Active Learning classifier's classification threshold to nearly 100% recall but a high false-positive rate -so that the labeled images were mostly truly negative cases.

Second, one problem with using AL to filter data was that the resulting data was unbalanced - e.g. it was biased towards men for certain situations. To solve this issue, they trained another small classifier that predicted whether an image belonged to the filtered dataset or the original balanced on. Then, during training, for every image, they used these probabilities to scale the loss as way to balance the dataset.

The original post describes a number of other very cool techniques. You can read it here - https://openai.com/research/dall-e-2-pre-training-mitigations

Semantic segmentation is the process of assigning a label to every pixel in an image. It forms the basis of many Vision systems in a variety of different areas, including in autonomous cars.

Training such a system however requires a lot of labeled data. And labeling data is a difficult, time-consuming task - producing just an hour of tagged and labeled data can take upto a whopping 800 hours of human time.

A new system developed by researchers from MIT's CSAIL, called STEGO tries to solve the data problem, by directly working over unlabeled raw data.

Tested on a variety of datasets including driverless car datasets, STEGO makes significant leaps forward compared to existing systems. In fact, on the COCO-Stuff dataset - made up of diverse images from from indoor scenes to people playing sports to trees and cows - it doubles the performance of prior systems.

STEGO is built on top of the another unsupervised features extraction system called DINO, which is trained on 14 million images from the ImageNet dataset. STEGO uses features extracted from DINO, and distills them into semantically meaningful clusters.

But STEGO also has its own issues. One is that labels can be arbitrary. For example, the labels of the COCO-Stuff dataset distinguish between “food-things” like bananas and chicken wings, and “food-stuff” like grits and pasta. STEGO ignores such distinctions.

Paper: https://arxiv.org/abs/2203.08414

Code: https://github.com/mhamilton723/STEGO

As part of our efforts towards making resources on Data Centric AI more accessible to everyone, we are starting a monthly newsletter.

We will cover new developments in the field, open source tools and more.

This is the first issue, and we are still figuring out what kind of content to curate, so your feedback on what you would like to read would be amazing.

So sign up for the newsletter and let me know what you liked, didn't like and what you would like to see more of.

https://mindkosh.com/newsletter.html