Predicting Tweet Sentiment

Predicting Tweet Sentiment

• 1.
  Predicting the Sentiment BehindMillions of Tweets Jaeduck Han Lucinda Linde Omesh Gadhave May 16, 2019 ALY 6110 Final Project Image credit ©Towards Data Science
• 2.
  Predicting Sentiment fromTweets Business Goals ● To help businesses understand customers’ feelings towards products/brands, response towards their advertising campaigns or product launches based on the sentiment (positive/negative) behind the tweets concerning them. Project Goals ● Build and assess different models to accurately determine whether a Tweet is positive or negative by training the models on a large split of data (training data) and testing their accuracy of predicting sentiment for the test data using python & its libraries. ● Gain experience using Hadoop, Spark and Pyspark.
• 3.
  Sentiment140 Dataset DataSet ● 1.6Mtweets from 2009, 800k positive and 800k negative. ● Tweets were labeled by interpreting emoticons as “distant supervised” data. ● Tweet labels: 0 = negative, 4 = positive. Attributes - 6 fields 1. target: the polarity of the tweet (0 = negative, 4 = positive) 2. ids: The id of the tweet ( 2087) 3. date: the date of the tweet (Sat May 16 23:58:44 UTC 2009) 4. flag: The query (lyx). If there is no query, then this value is NO_QUERY. 5. user: the user that tweeted (robotickilldozr) 6. text: the text of the tweet (Lyx is cool)
• 4.
  Data Examples target iddate flag user text 0 14678 10672 Mon Apr 06 22:19:49 PDT 2009 NO_QUERY scotthamilton is upset that he can't update his Facebook by texting it... and might cry as a result School today also. Blah! 4 14678 22272 Mon Apr 06 22:22:45 PDT 2009 NO_QUERY ersle I LOVE @Health4UandPets u guys r the best!!
• 5.
  Analysis Plan Cleanse andPreprocess data Explore the Data ● Word Clouds ● Bar graphs of Hashtags Assess different pros and cons of models Choose models Split Data into Train and Test Sets For each Model ● Choose parameters ● Make predictions ● Assess performance of model Discussion and Comparison Conclusion
• 6.
  Data Cleansing, Pre-processingand Modeling Using Python Clean, Pre-process and Explore ● Clean Tweets ● Tokenize, etc. ● EDA- Word Clouds ● EDA- Hashtag Graphs Models ● TF-IDF + Logistic Regression ● Random Forest ● fastText Evaluate Performance Using Pyspark Load Clean Tweets into Pyspark ● Remove Nulls (few thousand) Divide data set into train and test sets ● Build Models ● TF-IDF + Logistic Regression ● 2-Gram ● 3-Gram Compare Predictions to Labels Evaluate Performance
• 7.
  Data Cleansing andPre-processing Taming the Tweets ● Used NLTK for stemming- PorterStemmer ● Defined pattern find and replace for special character and stop word removal Tweets Remove special chars (not #) Remove words < 3 chars. Stem the words Stitch back together = Clean Tweets Tokenize words
• 8.
  Data Exploration: WordCloudsof Positive Tweets Positive Tweets ● Many positive words such as love, nice, well. ● Many neutral words like time, think, today, work. ● Positive and Negative Tweets are different, but not in an obvious way
• 9.
  WordClouds of NegativeTweets Negative Tweets ● Some negative words such as hate, sick, damn, ● Many neutral words like time, think, today, work. ● Difficult to say what actions should be taken
• 10.
  Hashtags- Positive -Many items reflect the time frame (4/09-6/09) ● #FollowFriday or #ff started in 2009 ● McFly British rock band, #1 album May 2009
• 11.
  Hashtags- Negative -What do we learn here? ● Iran Election took place in June 2009, then the arrests ● iPhone 3GS released June 9, 2009 (not great...)
• 12.
  Model selection-Why wechose what we chose We read several articles on sentiment analysis of tweets and of the sentiment140 data set. Among the successful models were: ● TF-IDF + N-Gram+ Logistic Regression ● Random Forest ● fastText
• 13.
  Pros and Consof Chosen Approaches Approach Pros Cons Logistic Regression Well-understood binary classification method Prone to over-fitting Random Forest Decorrelates trees reduced variance Not as easy to visually interpret fastText Fast, Conformed performance, Memory C++ language, Fixed data format, File I/O N-gram Captures “not good” or “not the best” instead of “good” or “best” Not “linguistically based” Lacks longer N dependencies
• 14.
  Assessing Model Performance AUCConfusion Matrix Area under Receiver Operator Curve How much more does the model predict above the presence in the population? Confusion Matrix, Recall and Precision Recall: What % of Actual Cancers are Detected? Precision: What % of Detected Cancers are Actual?
• 15.
  Performance Metrics toEvaluate Models ● Accuracy = (True positives + True negatives) / Total Examples ● Precision = True positives / (True positives + False Positives) ● Recall =True positives / (True positives + False Negatives) ● AUC = Area Under the Receiver Operator Curve ● F1 score = 2 * (Precision * Recall) / (Precision + Recall)
• 16.
  Model 1: RandomForest Pre-processing # of words Accuracy Precision Recall AUC F1-score BoW 6000 0.6856 0.6856 0.6856 0.6856 0.6856 Bag of Words : 300 words 1 tree : 14s Results: 0.61~0.62 Bag of Words : 300 words 100 tree : 1114s Results: 0.61~0.63 Bag of Words : 6000 words 1 tree : 266s Bag of Words: 6000 words 100 tree : more than 12 hours
• 17.
  What is fastText? fastTextis a library for efficient learning of word representations and sentence classification. Requirements fastText builds on modern Mac OS and Linux distributions. Since it uses C++11 features, it requires a compiler with good C++11 support. These include : ●(gcc-4.6.3 or newer) or (clang-3.3 or newer) Compilation is carried out using a Makefile, so you will need to have a working make. For the word-similarity evaluation script you will need: ●python 2.6 or newer ●numpy & scipy (fastText)
• 19.
  Model 2: fastText Model# of epoch Precision Recall fastText 1 0.7497 0.7497 fastText 30 0.7444 0.7444 fastText 50 0.7398 0.7398 The number of epochs is a hyperparameter that defines the number times that the learning algorithm will work through the entire training dataset.
• 20.
  Model 3 N-GramsGroups of N Consecutive Words 2-Gram Process Hadoop and Apache Spark are both big-data frameworks 3-Gram Process Hadoop and Apache Spark are both big-data frameworks that
• 21.
  Pipeline for 1,2 and 3 gram (N-Gram, TF/IDF, LR) TF = Term Frequency (# times term-t appears in doc)/ (Total # terms in doc) Clean_Tweets Tidy_Tweet Target (0,4) Tokenizer Input: Tidy_Tweet Output: “words” Tokenize the words of clean tweets Hashing TF+IDF Input: “words” Output: tf-grams, idf-grams Create Inverse Document Frequency Logistic Regression Input: target, tf-grams, idf- grams Output: Predictions Train Model and Predict Test data N-Gram Input: “words” Output: NGrams Create Term Frequencies IDF = Inverse Document Frequency log_e(Total number of documents / Number of documents with term t in it).
• 22.
  2-Gram Approach YieldsBest Results Pre-processing Model Accuracy Precision Recall AUC F1-score 1-Gram Logistic Regression 0.7566 0.7495 0.7755 0.8274 0.7623 2-Gram Logistic Regression 0.7702 0.7581 0.7980 0.8457 0.7768 3-Gram Logistic Regression 0.7696 0.7578 0.7867 0.8458 0.7768 Packages used: pyspark.ml.feature; NGram and VectorAssembler
• 23.
  Table Comparing ModelResults Pre-processing Model Accuracy Precision Recall AUC F1-score BoW Random forest 0.6856 0.6856 0.6856 0.6856 0.6856 fastText fastText - 0.7497 0.7497 - - 1-Gram Logistic Regression 0.7566 0.7495 0.7755 0.8274 0.7623 2-Gram Logistic Regression 0.7702 0.7581 0.7980 0.8457 0.7768 3-Gram Logistic Regression 0.7696 0.7578 0.7867 0.8458 0.7768
• 24.
  Discussion and Conclusions ●It was hard to beat 0.76 accuracy ● Many more combinations are possible to pursue ○ Preprocessing (BOW, TF/IDF, stemming, lexicons) ○ Algorithms (Naive-Bayes, Support Vector Machines, Logistic Regression, etc.) ● For each preprocessing and model, there are also many parameters to adjust ● Sentiment mining is still at a very early stage of development ● Text analysis doesn’t handle sarcasm (meaning the opposite). ● Even humans have trouble assessing the sentiment of Tweets
• 25.
  Future Work ● Trydifferent combinations of pre-processing and machine learning techniques ● Test how practical these approaches are for 10x or 100x the number of rows. ● What changes are needed for much larger volumes (e.g.sampling strategy) ● Analyze topics over longer time frames to characterize sentiment waves ● Attribute causes to changes in tweet sentiment, volume etc.
• 26.
  Image Credit: https://englishalcalans.wordpress.com/the-questions-suggestions-corner/
• 27.
  References Bourguignat, C. (2015,July 19). 6 Differences Between Pandas And Spark DataFrames. Retrieved from https://medium.com/@chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2 FastText. (n.d.). FastText. Retrieved from https://fasttext.cc/ Giachanou, Anastasia and Crestani, Fabio. (2016, June). Like it or not: A survey of Twitter sentiment analysis methods. ACM Comput. Surv. 49, 2, Article 28 Retrieved from: https://www.researchgate.net/publication/304916478_Like_It_or_Not_A_Survey_of_Twitter_Sentiment_Analysis_Methods Go, A., Bhayani, R., & Huang, L. (2009). Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, 1(12), 2009. Joshi, P. (2018, July 30). Comprehensive Hands on Guide to Twitter Sentiment Analysis with dataset & code. Retrieved from https://www.analyticsvidhya.com/blog/2018/07/hands-on-sentiment-analysis-dataset-python/ Kim, R. (2018, March 13). Sentiment Analysis with PySpark. Retrieved from https://towardsdatascience.com/sentiment-analysis- with-pyspark-bc8e83f80c35 Kim, R. (2018, January 13). Another Twitter sentiment analysis with Python - Part 5 (Tfidf vectorizer, model comparison... Retrieved from https://towardsdatascience.com/another-twitter-sentiment-analysis-with-python-part-5-50b4e87d9bdd
• 28.
  Data source ● http://www.sentiment140.com/ ENDOF PRESENTATION