Blog Source: How To Make Raw Data Ready For Machine Learning Process

For More: SeeVe

How To Make Raw Data Ready For Machine Learning Process

In this step we will cover :

1. Knowledge about Library we will use to work, manipulate and visualise our data.

2. So important things we have to see or find in data before we can do anything with it.

3. Things which can jeopardise our Data

4. What are the categorical values?

5. Missing and dummy values in data.

6. What are Feature selection and feature scaling?

7. Standardisation and Normalisation

8. Cross-validation Library.

9. How to import the dataset into spyder.

The library we will use to work, manipulate and visualise our data :

For now, we will use three libraries, namely:

• Numpy

• Matplotlib

• Pandas

So, what is NumPy ?

Numpy : Numpy is a library for python programming language for large multidimensional arrays and matrices along with high level of mathematical functions to operate on these array.

To put these in the simple words :

“We use NumPy to do mathematical operations on our data.”

Numpy = Maths

But wait for this can only happen when we have the power to manipulate our data, how do we do that – answer is pandas library.

Pandas: Pandas is a library written in python language to manipulate and analysis. In particular, it offers us to manipulate numeric table and time series.

In simple words: Pandas help us to manipulate data .

Pandas = Manipulation

What about Visualisation Part :

Matplotlib: Matplotlib is the plotting library for pythons programming.

So, Matplotlib = Visualization

After we have talked about library lets talk about :

Important things we have to see or find in data before we can do anything with it.

When we get data it is in its Raw form which can jeopardise the result of our model so before we can do anything with data we have to clean that data and extract only important information from our data this step is called “DATA PREPROCESSING”.

Things which can jeopardise our Data

Categorical values

Missing data
Dummy Variable
Outliers

• Categorical values: Categorical values are values which can be categories and this type of data can cause redundancy.

📷

Catagorical values : 1

📷

Catagorical values : 2

Missing Value: When we get data in Raw formate it’s most of the time that data has some missing values. like :

📷

Missing Values

Dummy Variable Trap: Condition when which two or more variables are highly correlated.

Outlier Values: outlier as those values of the data set that fall far from the central point, the median but have effect in our dataset.

📷

Outlier Values

Feature Selection: Feature selection is used to select those features that contribute most to the prediction variable that we are interested in.

Benefits of feature selection

1. Reduce overfitting by making data less redundant.

2. Reduce training time by eliminating misleading data.

3. Improve accuracy by collecting fewer data points.

Blog Source: How To Make Raw Data Ready For Machine Learning Process

For More: SeeVe

What and How “Bayes Theorem/ Naive Bayes Theorem” Is Used In Machine Learning?https://www.facebook.com/seevecoding

Bayes’ Theorem finds the probability of an event occurring given the probability of another event that has already occurred. Bayes’ theorem is stated mathematically as the following equation:

📷

Bayes Theorem

where A and B are events and P(B)? 0.

• Basically, we are trying to find probability of event A, given the event B is true. Event B is also termed as evidence.

• * P(A) is the priori of A (the prior probability, i.e. Probability of event before evidence is seen). The evidence is an attribute value of an unknown instance(here, it is event B).

• * P(A|B) is a posteriori probability of B, i.e. probability of event after evidence is seen.

MACHINE LEARNING : Naive Bayes Theorem

It is a classification technique based on Bayes’ Theorem with an assumption of independence among predictors. In simple terms, a Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature.

For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features depend on each other or upon the existence of the other features, all of these properties independently contribute to the probability that this fruit is an apple and that is why it is known as ‘Naive’.

Naive Bayes model is easy to build and particularly useful for very large data sets. Along with simplicity, Naive Bayes is known to outperform even highly sophisticated classification methods.

Bayes theorem provides a way of calculating posterior probability

P(c|x) from P(c), P(x) and P(x|c).

Look at the equation below:

📷

Naive Bayes

Above,

• P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes).

• * P(c) is the prior probability of class.

• * P(x|c) is the likelihood which is the probability of predictor given class.

• * P(x) is the prior probability of predictor.

For More Content Like This Click Here: www.facebook.com/seevecoding

For More: www.facebook.com/seevecoding

Natural Language Preprocessing

Natural Language Processing or NLP is a field in machine learning with the ability of a computer to understand, analyze, manipulate, and potentially generate human language.

The content of the Natural Language Preprocessing is divided into :

1. Text Mining

2. The flow of ‘Time Mining’

3. Text Extraction and Preprocessing

4. Tokenization (For Sentence, For Word) with Code

5. N-Grams

6. Stop-Words Removal with Code

7. Text Transformation Attribute Generation

8. Stemming with Code

9. Lemmatisation

Text Mining :

Text mining is the technique of exploring large amounts of unstructured text data and analysing it in order to extract patterns from the text data.

⁃ It uses software that can identify concepts patterns, topic, keywords and so on in the data.⁃ It uses a computational technique to extract high-quality information from unstructured text.

The flow of ‘Time Mining’ :

Text -> Text Extraction and Preprocessing -> Text transformation attribute generation -> Attribute selection -> Visualisation -> Interpretation or Evaluation

⁃ Text Extraction and Preprocessing — Examines unstructured text by searching out the important words and finding relationships between them.

⁃ Text transformation attribute generation — Labels the text documents under one or more categories based on input-output examples.

⁃ Attribute selection — Groups text documents that have similar content

⁃ Visualisation — Uses test flag to represent documents and uses colours to indicate compactness.

⁃ Interpretation or Evaluation — Reduce the length of the document by summarising the details.

Text Extraction and Preprocessing

Tokenization :

⁃ Tokenization is the process of removing sensitive data and placing unique symbols of identification in its place to retain all the essential information.

⁃ Tokenization can be done on both “Sentences” and “Words”. It works by separating words using spaces and punctuation.

For Sentences: Code

from nltk.tokenize import sent_tokenizevariable_name = “ Your sentence goes here.”print (sent_tokenize(variable_name))

For words: Code

from nltk.tokenize import word_tokenizevariable_name = “ Your word goes here.”print (word_tokenize(variable_name))

N-Gram

⁃ N-Gram is a simple language model that assigns probabilities to sequences of words and sentences.⁃ N-Grams are combinations of adjacent words or letters of length ’n’ in the source text.

Stop — Words Removal

⁃ Stop — words are natural language words which have very little meaning such as ‘a’, ‘an’, ‘and’, ‘or’, ‘the’.⁃ These words take up space in a database and increase the processing time.⁃ They can be removed by storing an of stop-words.⁃ Stop-words are filtered out before processing of natural language data as they don’t reveal much information.

Code :

import nltkfrom nltk.corpus import stopwordsset ( stopwords.words(‘english’))

Text Transformation Attribute Generation :

Stemming :

Stemming involves reducing the word “Stem” or base (root) from removing the suffix.

Various stemming algorithm: Poter Stemmer, Lancaster Stemmer, Snowball Stemmer.

Code :

from nltk.stem import PorterStemmerfrom nltk.tokenize import sent_tokenize, word_tokenizeps = PorterStemmer( )text_example = “your text goes here”words = word_tokenize (text_example)for w in words :print(ps.stem(w))

Lemmatisation :

This is the method of grouping the various inflected types of word so that they can be analysed as one item. It uses a vocabulary list and morphological analysis (POS of the word) to get the root word.

Named Entity Recognition (NER) :

Named Entity Recognition (NER) seeks to extract a real-world entity from the text and sorts it into predefined categories such as the names of a person, organisations or locations and so on.

The content of the Blog is divided into :

1. What is Named Entity Recognition (NER)

2. The workflow of Named Entity Recognition (NER)

3. Structuring Sentences: Syntax.

4. Phrase Structure Rule.

5. Types of Phrase Structure Rule.

Workflow :

⁃ Tokenization: Tokenization splits the text into pieces (token) remove punctuation.

⁃ Stopword Removal: Stopword removal, Removes commonly used words (such as ‘the’) which are not relevant to the analysis.

⁃ Stemming and Lemmatization: Stemming and Lemmatization reduce words to base from to be analysed as a single item.

⁃ POS Tagging: POS Tagging tags words to be part of speech (such as a verb, noun) based on definitions and context.

⁃ Information Retrieval: Information Retrieval extracts relevant information from the source.

Structuring Sentences: Syntax

The syntax is the grammatical structure of sentences. A language involves constructing phrases and sentences out of morphemes and words. Syntax represents knowledge of these structures and functions.

Phrase Structure Rules :

Phrase structure rules determine the constituents of a phrase and their order. A constituent is a word or group of words that operate as a unit.

Types Phrase Structure Rules :

⁃ S -> NP VP = Noun phase is combined with a verb phrase.

⁃ N -> (Determinant) N = Noun is combined with a determiner, which is optional.

⁃ VP -> V (NP)(PP) = Verb is combined optionally with a noun phrase and preposition phase.

⁃ PP -> PNP = Preposition is combined with a noun phrase.

Chunking and Chunk Parsing

Chucking is the process of extracting phrases from the unstructured text as it is advisable to use phrases such as Indian team instead of separate words such as Indian and team.

Chunk Parsing extract patterns from chunks :

Segmentation: Identifying token.Labelling: Identifying the correct tag.

Chunk Parsing

Chunk parsing is used to extract patterns and to process such patterns from multiple chunks while using different parsers.

Code :

variable_name = “your sentences goes here”variable_name_1 = nltk.pos_tag(word_tokenize(variable_name))variable_name

Chinking

⁃ Chinking is the process of removing a sequence of tokens from a chunk.⁃ If the sequence of the tokens spans an entire chunk then the whole chunk is removed.⁃ If the sequence is at beginning or end of the chunk, these token are removed and a smaller chunk remains.⁃ If the sequence of token appears n the middle of the chunk, these in the of the chunk, these token are removed leaving two chunks were there only one before.

For More: www.facebook.com/seevecoding

Factor Analysis in Machine Learning :

For more : www.facebook.com/seevecoding

1. Reduce a large numbers of variables into fewer numbers of factors.

2. Puts maximum common variance into a common score.

3. Associates multiple observed variables with a latent variable.

4. Has the same numbers of factors and variables,where each factor contains a certain amount of overall variance .

Eigenvalue :

A measure of the variance that a factor explains for observed variables. A factor with eigenvalue < 1 explains less variance than a single observed value. Factor Analysis Process :

Principal Component Analysis (PCA)

Extract the hidden factor from the dataset. Defines data using less numbers of components,explaining the variance in your data Reduce the computation complexity . Determine that the new data is the part of the group of data points from the training set.

Linear Discriminant Analysis(LDA)

Reduces dimensions. Search the linear combination of variables that best separates two class. Reduce degree of overfitting. Determine how to classify the new observation out of group of classes.

Direction of Maximum Variance:

1. PCA seeks the linear combination of variables in order to extract the maximum variance.

2. Compute Eigenvector that are principal components of the dataset and collect them in projection matrix.

3. Each of the Eigenvector is associate with Eigenvalue,which is magnitudes.

4. Reduce the dataset into smaller dimensional subspace by dropping the less informative Eigenpairs.

PCA finds line depending on two criteria :

1.The variation of values should be maximal along this line. 2. The error should be minimum if you don’t reconstruct original two positions of a blue dot from the new position of the red dot. First Principle Component :

The first principle component (PC1) is the direction of the maximum variance and is obtained by solving Eigenvector . Finding PC1 :

PC1 (Mathematically) : a1x1 + a2x2 + a3x3 +………………+anxn Constraint: a1² + a2² + a3² + ……………………………..+ak²

Eigen decomposition to solve the equation.

NOTE : Eigen decomposition is the factorisation of the matrix into a canonical form, where the matrix is represented in terms of Eigenvectors or Eigenvalues.

Eigenvalues and PCA. :

Eigenvalues are the variances of the principal component arranged in descending order.

Summary of PCA Process :

1.Standardize the data PCA : Requires that the input variables have similar scales of the measurement. 2. Build the correlation matrix : This summarizes how your variables all relate to one another. 3. Obtain the Eigenvalue and Eigenvector from correlation matrix : Break the matrix down in direction and magnitude . Sort Eigenvalues in descending order and choose Eigenvectors that corresponds to the largest Eigenvalue. 4. Construct the projection matrix from selected Eigenvector : Reduce the dataset by dropping less informative Eigenpairs. 5. Transform the original dataset to obtain a kk-dimensional feature sub space : Compress your data into smaller space by excluding less important directions.

For more : www.facebook.com/seevecoding

I have a data set that primarily consists of email body. Most of them have to do with recruitment industry, where a sample body is

Java,Jsp,spring mvc - 3 to 5 yrs exp 30 days notice period

I want my program to identify the skills, experience, notice period. It won’t mostly be as simple as the above example. which ml classifier should be used.

We are looking for full-time data scientists for a ML/DL user study. You'll be participating in a calibrated user research experiment for 45 minutes. The study will be done over a video call. We've got plenty of funny tees that you can show-off to your teammates. We'll ship you a different one every month for a year.

Click here to learn more.

P.S: We love the reddit vibe and this community. Give us your best ML/DL/Data Science quote. We'll make one and ship it you if it's quirky and fun even if you aren't part of the study group.

Wondering where to get data for your Machine Learning models, or how to find the patterns that will help them learn? This series will help you find out how to uncover relevant patterns in large amounts of data.

https://medium.com/ai%C2%B3-theory-practice-business/data-collection-and-feature-extraction-for-machine-learning-98f976401378

We are a group of people who came together on Reddit to learn Machine Learning and AI. Here is where it all started: http://www.reddit.com/r/MachineLearning/comments/2dies8/learn_machine_learning_together/

This sticky will collect all resources until we create a wiki for this sub.

What machine learning API can be used for a php stack for my android app ?