FNB Namibia Twitter Sentiment analysis using Vader and RoBERTa

Introduction

Sentiment analysis is a field in Natural Language Processing that identifies and extracts the emotion expressed in a text and then labels it as either positive, negative, or neutral. It is commonly used to analyze customers’ online engagement with an organization or product, as well as product reviews and feedback. Importantly, it is a valuable tool in helping businesses figure out what consumers think of them and their products, without having to run large-scale surveys that are usually costly and time-consuming.

A key component of sentiment analysis is polarity classification. Polarity is the overall sentiment conveyed by a text, phrase, or word. This polarity can be expressed as a numerical rating known as a “sentiment score”.

Problem Statement

In this project, we implement an FNB Twitter sentiment analysis Model that helps classify tweets that mention FNB as either positive, negative, or neutral. The aim of this project is to identify the proportion of negative, positive, and neutral tweets respectively, which helps us identify the overall sentiments Namibian Twitter users have about FNB.

In this model, we will specifically implement the Valence aware dictionary for sentiment reasoning (VADER) sentiment analyzer, and the pre-trained RoBERTa model by hugging face. The latter has the advantage of understanding the sequence of words better and thus should be better at sentiment classification. We will be comparing these two models. The data used in this project is scraped from Twitter using the snscrape python library.

Import Necessary Dependencies

Specify the city and the 400km radius to avoid getting tweets from outside Namibia. This effectively means that only tweets from users in a 500km radius of Windhoek were included in this analysis.

Here is the data we are working with. We have a date, and content column. As well as a few other columns such as like count and reply count.

We were able to scrape 1117 tweets with 27 columns.

Data Cleaning and Preprocessing

We dropped the above columns because they will not be necessary for this analysis. We also dropped the current tweet Id column as it does not help us with identifying the different tweets as we would like. In the next lines of code, we create another Id column that starts at 1 and ends at 1117.

Notice that our date column has the +00:00 at the end of each date. This is a timezone that is not helpful in our analysis, so we should remove it.

While we are at it, we will also create a column that identifies what the day of the week was on that date.

Above we can see the newly added Month and Weekday column.

Now we preprocess the contents of the tweets by removing emojis, and URLs in the tweets. We will not be removing stopwords because research has shown that doing so reduces the ability to properly classify the text. We will add these preprocessed tweets in a column called PreprocessedTweets.

We are interested in knowing the Twitter users that tweet most about FNB. For that, we need to clean the user variable column. Remove all pieces of information so that we are left only with the user name.

We need to get rid of all information before and after ‘Eboy945’. We are going to apply string operation to get rid of such information, so we need to change the data type to string instead of object and then remove all the unnecessary information.

Viewing the user column, all the unnecessary information has been removed, and we are now only left with the Twitter usernames.

Exploratory Data Analysis

Let's take a look at the top 10 people tweeting about FNB

The users ItsJenn_fer and Jossy_G_ have all tweeted about FNB over 30 times, with the rest having done so fewer than 20 times.

We would like to visualize how the tweets are distributed per day and per month.

Tweets about FNB appear to mostly happen on Wednesday. with the other weekdays just being slightly behind. The Weekend, however, happens to have the least tweets about the bank. In terms of Month, September appears to be an outlier, however, that can be explained by the fact that the tweets have been scraped in September, and the scraper prioritizes recency.

finally, we would like to see the most common word used in these tweets by plotting a wordcloud

The WordCloud plot shows the most words in every word with frequency organized by word size. Words like Charge, Queue, and Ewallet seem to feature in many of the tweets about FNB. Unsurprisingly, many tweets also mention other banks like Standard Bank and Nedbank. This is because perhaps many tweets are comparative in nature.

From the descriptive analysis of the numerical variables, we find that the average number of likes a tweet about FNB receives is 4.15, while they receive on average 1.3 Retweets. The maximum number of likes a tweet has received is 239, and the maximum number of retweets they received is 148 retweets

Get Polarity Scores using Vader

In this section, we analyzed the sentiment of the tweet by calculating each tweets’s polarity, and then, based on the polarity, analysing if said tweet has a negative, positive or neutral sentiment.

Vader is a sentiment analysis tool with a lexicon and rules that is atuned to the sentiments expressed on social media.

Vader assigns 3 probability scores to a text; the probability that a text is negative, positive, or neutral. It then takes the assigned probabilities and computes a compound score. If the tweet is negative, it computes a negative score, if it’s positive, it computes a positive score, and it computes a 0.000 if the tweet is neutral.

The average compound score is 0.042. This suggests that on average, tweets about FNB have a positive sentiment.

The above lines of code merge the polarity score DataFrame with the main DataFrame. and then proceeds to create a column that labels a tweet as either positive (if compound score>0), negative (compound score<0), or neutral (compound score=0).

According to the Vader analysis, 31.6% of the tweets about FNB are positive, 25.5% are Negative, and 42.9% are Neutral.

After printing out the tweets labeled positive, negative, and neutral respectively, it becomes very obvious that Vader does not really do a good job of classifying the tweets in the proper sentiment. for example the tweet “I'm still shaken after being shouted at by someone who was supposed to help me !! Completed that after service survey but wow, they all ignored me!!! Thanks for the nudge...this is definitely an abusive relationship" clearly has a negative sentiment about FNB, however, it is classified as positive.

The above example has a high positive compound score, even when the tweet is clearly negative.

This is because some of the disadvantages of Vader is that it does not account for the relationship between words or sarcasm. It takes the words in a text and assigns them a positive, negative or neutral score and then combines those scores to give you a sentiment score. In the above sentence words like “wow”, “completed”, ‘thanks, “nudge” and “relationship” may have gotten scored positively because they are generally good sentiment words, however, in this context, the sequence of the words makes them an overall negative sentiment.

This is where RoBERTa comes in.

RoBERTa Model

RoBERTa is a transformer-based deep learning model that is able to pick up the context in which the words in the text are being used, and the relationships they have with each other. In this project, we are going to use a model that Huggingface has already trained. This makes training our model less computationally tax, but even more accurate because the Huggingface model has been trained on millions of tweets already.

Now that we have imported the RoBERTa model, we will run the same example we run with the Vader model on the RoBERTa model to see if it does a better job of classifying the tweet than Vader.

As can be seen, the RoBERTa model gives the tweet a 95% chance that it is negative, a 4% chance that it is neutral, and 0.5% chance that it is positive. This is a far more accurate classification of the tweet in the example.

Now we can run this model through our entire data set.

above is what the new DataFrame looks like.

RoBERTa does not compute a compound score like the Vader model. So we compare the 3 probabilities (Negative, Neutral, and Positive) and the highest one becomes the label of that tweet. From the earlier example, the tweet had a high negative score of 0.9598, therefore that tweet will be labeled as negative.

According to the RoBerta analysis, 12.0% of the tweets about FNB are positive, 30.1% are Negative, and 57.9% are positive.

Comparing the Vader and the RoBERTa model, there is a 62% decrease in the number of tweets classified as positive. As can be seen from the two graphs below, RoBERTa classified fewer tweets as positive, compared to Vader.

The graphs below indicate how the sentiment in the tweets is distributed between the days of the week and the months in a year. There is nothing notable about how the sentiment is distributed in the week. The more tweets are in the week of the day, the more likely each sentiment will occur. Same can be said about the distribution in the months of the year.

The line graph indicates a peak in the positive scores on Mondays. This is followed by a dip on Tuesdays, which is the day with the highest neutral scores. The highest negative scores are awarded to tweets posted between Thursday and Friday.

The graphs below show how the sentiment scores are distributed for each RoBERTa sentiment. The hues indicate how those tweets had been classified by the model in the end.

We conclude, this project by getting the positive, negative, and neutral tweets respectively, as classified by RoBERTa.

Conclusion

In this project, we scraped Twitter data from users in and around Windhoek who mentioned FNB in their tweets. The aim of this project was to see the sentiment behind tweets about FNB and classify them as either positive, negative, or neutral. Furthermore, we wanted to see how these tweets are distributed by day of the week and month. Importantly, we wanted to know the Twitter users that are mostly tweeting about FNB.

Based on the RoBERTa model, tweets about FNB are 12.0% positive, 30.1% Negative, and 57.9% Neutral.

The secondary purpose of this project was to demonstrate how the two models covered in this models perform on real-life data, and how they compare with each other.