For Deceptive Reviews, these are terms with high specificity

reviews.tfidf.df <- left_join(reviews.df, reviews.tfidf)
deceptive.tfidf <- reviews.tfidf.df %>%
    filter(truthfulness=="deceptive") %>%
    filter(pos!='PROPN')
deceptive.tfidf <- distinct(deceptive.tfidf, word, .keep_all = TRUE)
deceptive.tfidf <- deceptive.tfidf[order(-deceptive.tfidf$tf_idf),] 
kable(deceptive.tfidf[1:19,c(2,3,4,5,6,9,10,11)], format = "markdown")

Deceptive writers use the word deceive in deceptive reviews.

For Truthful Reviews, these are terms with high specificity

truthful.tfidf <- reviews.tfidf.df %>%
    filter(truthfulness=="truthful") %>%
    filter(pos!='PROPN') 
truthful.tfidf <- distinct(truthful.tfidf, word, .keep_all = TRUE)
truthful.tfidf <- truthful.tfidf[order(-truthful.tfidf$tf_idf),] 
kable(truthful.tfidf[2:19,c(2,3,4,5,6,9,10,11)], format = "markdown")

Truthful people don’t say smelly, they say stinky.

Alright!, Most of the heavy database lifting is done. Let’s start teaching the machine.

In a classic machine learning process we randomly divide the data into a training set and a testing set - here 80% of the 1600 files are randomly selected for training, and the other 20% will be used to test the trained machine’s accuracy. 80% of a small 1600 member training set leaves an even smaller training set of 1280 files. You don’t go to war with the army you wished for, you go to war with the army you have. Poor justification, I know LOL - it’s stinky.

Naive Setup

This is my first pass at training. Hopefully we will get better than average human performance (an average accuracy of no better than random at 50%). Let’s start with a naive approach, by using the filtered, stemmed words that we’ve processed so far, adding a label for parts of speech.

First let’s build a document/text database consisting of the existing filtered text. The we build a document term matrix, where each row is a document in the training set, and each term is the Tf°Idf of that term in the training document set.

# input_data_X is the input training data
# output_data_Y is the output prediction (i.e. the classification result)
gather_review_string <- function(aDocID) { paste(reviews.tfidf.df$word[reviews.tfidf.df$doc_id == aDocID], collapse = " ") }
reviews.docs <-  distinct(reviews.tfidf.df, doc_id, .keep_all = TRUE) 
reviews.docs$review <- sapply(reviews.docs$doc_id, gather_review_string)
reviews.docs <-  select(reviews.docs, doc_id, truthfulness, review)
write_csv(reviews.docs, './files/filtered_reviews.csv')

full_range <- 1:length(reviews.docs$doc_id)
training_range <- sample(full_range, round(.8*length(full_range)))
test_range <- setdiff(full_range,training_range)

# Weighting by Tf°Idf drops %accuracy by 10!
doc_term_matrix = create_matrix(reviews.docs[training_range,]$review)#, weighting=weightTfIdf)
input_data_X <- doc_term_matrix 
output_data_Y <- reviews.docs[training_range,]$truthfulness

Naive SVM Training:

We then create the SVM model and ask it to train itself (time for a drink of coffee)

# We can run prediction on the model to determine execution time

# Set up an SVM model
# Configure the training data
container <- create_container(input_data_X, output_data_Y, trainSize=1:length(training_range), virgin=FALSE)
 
# train a SVM Model
model <- train_model(container, "SVM", kernel="linear", cost=1)

Naive SVM Performance:

Once the model is trained, we pull out the test set of data, and use the model to predict the truth. We were going to use the machine to produce King Solomon’s sword. Typically the first part of analyzing and improving performance is to produce a “confusion matrix” which tells us how well (or poorly) we are doing on a 2x2 binary classification problem.

## If you error out:
# trace("create_matrix",edit=T)
# edit it and on line 42 will have a misspelling of the word "acronym". 
# Change the "A" to an "a" and hit "Save" - it should work fine after that.
predictionData <- reviews.docs[test_range,]$review
# create a prediction document term matrix 
predMatrix <- create_matrix(predictionData, originalMatrix=doc_term_matrix) 

# create the corresponding container
predSize = length(predictionData);
predictionContainer <- create_container(predMatrix, labels=rep(0,predSize), testSize=1:predSize, virgin=FALSE) 

# predict
results <- classify_model(predictionContainer, model)
predicted_results <- as.character(results$SVM_LABEL)
actual_results <- reviews.docs[test_range,]$truthfulness

confusion.df <- as.tibble(data.frame(predicted_results, actual_results, stringsAsFactors = FALSE))
Prediction.Incorrect <- 100* sum(confusion.df$predicted_results != confusion.df$actual_results)/length(test_range)
Prediction.Correct <- 100* sum(confusion.df$predicted_results == confusion.df$actual_results)/length(test_range)

classification.truth_truth <- sum((confusion.df$predicted_results == 'truthful') &
                                  (confusion.df$actual_results == 'truthful'))
classification.truth_false <- sum((confusion.df$predicted_results == 'truthful') &
                                  (confusion.df$actual_results == 'deceptive'))
classification.false_truth <- sum((confusion.df$predicted_results == 'deceptive') &
                                  (confusion.df$actual_results == 'truthful'))
classification.false_false <- sum((confusion.df$predicted_results == 'deceptive') &
                                  (confusion.df$actual_results == 'deceptive'))
Actual.type <- c('Deceptive', 'Truthful')
Prediction.Deceptive <- c(classification.false_false, classification.false_truth)
Prediction.Truthful <- c(classification.truth_false, classification.truth_truth)
conf.mat <- data.frame(Actual.type, Prediction.Deceptive, Prediction.Truthful)

#print(paste("Accuracy is", Prediction.Correct, "%"))
kable(conf.mat, format = "markdown")

Ok. Well 85% accuracy won’t get me a job on Wall St. But it’s a naive Bag-of-words/Parts-of-speech approach. And with some additional feature engineering I know I can get to 95%

Oh, by the way - even the naive machine did MUCH BETTER than human intelligence.

Let’s improve and tune the model. King Solomon’s sword needs to be sharper. The authors of the original paper noted that bigram’s worked better than unigrams. Also what about a different type of learning model? Although our training model size is small, as is our dictionary size, I’d like to try a neural net to see what happens with overtrained models…