Our methodologies for review summarization, scoring and ranking

Ranking system

To show the best products and services that you are looking for, we make sure to keep our data sources legitimate, up-to-date, and useful. We gather online reviews from reliable websites, weed out suspicious reviews and apply state-of-the-art machine learning algorithms to extract positive and negative sentiments of each service. Each sentiment is utilized in generating representative scores and ranks for all services relating to the keywords you search for.

ranking system picture
Scrape
ranking system picture
Filter
ranking system picture
Summarize
ranking system picture
Score & Rank
ranking system picture
Recommend
picture of blue circle
Scrape: Review Gathering Methodology
  • For every available search category at Consumerchoice.com, we research and gather a comprehensive list of vendors providing similar services.

  • We collect all possible reviews about each service vendor on the list by scraping data from genuine online sources using platform-specific web scrapers.

  • Our database is consistently updated with the latest reviews to keep you best informed.

🔬 A walkthrough scenario:

One of the search categories we offer is “Top VPN Services”. Under this category, we include as many VPN service vendors as possible, such as SaferVPN, LibertyShield, Le VPN, and more. For each service vendor on our list, we scrape its related user reviews from legitimate websites, such as the Apple and Google app stores. The data scraping process is repeated consistently over time to obtain the most updated reviews.

picture of blue circle
Filter: Suspicious Review Detection Methodology
  • A rule-based algorithm is implemented to analyze text features of every scraped review. There are 63 checkpoints in place to examine if a text sentence might read like a fake review.

  • The algorithm compares the check result of each text sentence with the result of existing text sentences that are flagged as normal reviews. It is to determine the probability of incoming sentences being an “outlier” from legitimate reviews.

  • Reviews that are flagged as anomalies/outliers are excluded from our algorithm training for review summarization.

🔬 A walkthrough scenario:

Each review text from the scraped data is feed into the review filtering algorithm. The algorithm analyses the text by calculating features based on 63 checking criteria. The associated outcome is compared with the outcome distribution of normal reviews. If the probability of the review is high being an anomaly, it is flagged as a potentially fake review and is filtered out from further usage.

picture of blue circle
The 63 checking criteria of our review filtering algorithm:
  • Grammatical usage: To count the total occurrence of nouns (singular, singular proper, plural, plural proper), pronouns (personal, possessive), verbs (base form, past tense, past participle), third persons (singular present, participles, possessive ending), adjectives, comparative adjectives, adverbs, prepositions, superlatives, determiners, pre-determiners, modals, and coordinating conjunctions.

  • Sentence formulation: To count the total occurrence of full stops, commas, cardinal numbers, upper-case letters, stop words, existential there, interjections, comparatives, negative words, interrogative words, foreign words, difficult words, power words, casual words, tentative words, and emotion words.

  • Inclusion of selected characters: To count the total occurrence of “Wh” (pronouns, determiners, adverbs), quotes (double and single), brackets (left and right), colons, symbols, “to”, “$”, “RBS”, and “#”.

  • Readability scores: To compile and compare the outcome of Flesch reading-ease score, Flesch Kincaid grade, smog index, automated readability index, Dale Chall readability score, Linsear write formula, Gunning fog, and text standards.

picture of blue circle
Summarize: Review Clustering Methodology
  • For topic clustering and summarization, we implement a state-of-the-art machine learning model named BART [1], a transformer-based natural language processing algorithm with robust performance.

  • We fine-tuned and retrained the BART model to produce customized clusters of topics. Reviews related to similar contexts are clustered within the same group, forming a topic/context-based summary.

  • For each topic cluster, the reviews within it are further processed to determine their sentiments towards the context. Each review is tagged as either a positive or a negative review concerning the topic.

  • The tagged reviews are re-compiled under each service category, forming a catalog of pros and cons for each vendor.

  • We currently process reviews only in the English language, with the possibility to expand into six more languages in the future.

🔬 A walkthrough scenario:

A review that passed the filter test is assigned a topic group by the BART summarization model, as a representation of its main context. If the review describes how the user is satisfied with the reliability of the SaferVPN performance, it will be assigned to the topic cluster of “reliability”, and then tagged with a positive sentiment. After recompilation, the review is assigned to the service category of SaferVPN, with the topic of “reliability”, of positive sentiment in the pros and cons summary.

picture of blue circle
Score & Rank: Review Processing and Recommendation Methodology
  • We implement another state-of-the-art machine learning algorithm, named RoBERTa [2], to generate a score per review based on its positive or negative sentiment around a topic.

  • The customized and retrained RoBERTa model outputs a rating for every review with a range of 1 to 5. A rating of 5 being the highest score and a rating of 1 being the lowest.

  • All ratings concerning a service are aggregated and averaged to generate a final score that represents the current reputation of each service vendor.

  • All service vendors belonging to the same search category are ranked accordingly based on their final score. Intuitively, services with higher ratings are positioned near the top of the search result.

  • To generate a fair and reliable ranking for both established and new service vendors (new = contain less than 50 reviews), we place greater importance on the Google search rank among these services than their pros and cons ratings.

🔬 A walkthrough scenario:

All reviews concerning a service vendor, for example, SaferVPN, are given a rating from 1 to 5 by the RoBERTa algorithm. Taking the same example – a review that describes how the user is satisfied with the reliability of the SaferVPN performance, it might get a high rating of 5. Assume that the SaferVPN has 50 reviews in total, its final score is the average rating of all the 50 reviews. All services under the search category of “Top VPN Services” are ranked according to their final score. But because SaferVPN has only 50 reviews (considered new), we take its Google rank order as its rank position on our website.