Can We Trust App Store Reviews?
Getting feedback from customers is extremely important no matter what industry you work in. If you aren’t geared towards understanding your customers wants and needs and enacting required changes, well then you probably won’t last too long. Technology has made it so easy for customers to both give their feedback and to read reviews to make informed decisions. But in a world flooded with so much information how do we know what makes a source of information reliable?
A Feefo report of 2,000 UK consumers found that 94% of consumers will check the online reviews of a product before deciding whether or not to purchase it and 84% said that reviews of existing users are the most reliable source of information. However, only approximately only 1.5% of consumers actually leave an online review with a majority of these not representing the “average consumer”.
The Journal of Consumer Research published a study in 2016 that found very little correlation between objective quality and online reviews. From this, we can assume that people who leave online reviews tend to be individuals whos expectations are slightly ajar to the main product or service definition and therefore are either ecstatic that the product was able to adjust to their requirements or annoyed that it couldn’t.
So why is the public perception that online reviews are a reliable source of information?
How App Store Reviews Work
The Apple App Store and Google Play Store are the two easiest places to find online reviews for apps since when someone downloads an app onto their device they are given the opportunity to give it a rating. But knowing what we know now are the app store reviews a reliable source of information?
The app store reviews are very straightforward, asking for a star rating out of 5 and an optional written feedback box.
Each rating out of 5 is averaged into an overall rating for the app.
This is then shown on the app page in the app store and helps to rank search results to prioritise the highest ranked apps.
One major downside to the app stores rating system is on the user interface is there is no apparent correlation between the rating and the app version number. That means a rating from Version 1 of the app can still be present (and affecting the overall rating) even if it is 5 years later and the app has gone through several major developments. Thus, showing a rating that might not accurately describe the experience a new user would face.
The Apple App store gives the developer the option to remove previous ratings from their overall rating when they deploy a new version of their app. While this helps make the overall rating more representative of the current product, it removes the credibility of that rating because it includes fewer reviews and discounts the reviews of long time users. It also doesn’t show improvement over time.
Example
Joe downloaded an app (Version 1.1) and made an input error during sign up which affected his onboarding experience. So he left an angry review of 1 star on the app store. A few days later he realised his mistake and went on to use the app very happily for the next 6 months but never went back to change his rating. This rating will now sit there affecting the overall rating of the app but not actually represent the experience Joe is having.
Feedback For Software-As-A-Service
As we know the emergence of Software-as-a-Service has dramatically changed the game of technology. Now instead of standalone software products, we have software services that are continually being improved and updated. In addition, these updates are pushed out to all existing users of the platform or sold as additional add-on products available through the platform.
Having a static review/rating system isn’t fit for purpose.
When software is so dynamic, always changing and improving, a static rating only misinforms the consumer. Feedback needs to be able to be given at any point in time and asked multiple times throughout a users lifetime with the product. This provides context to the review because it shows exactly what product and version of the product, the user was providing feedback on. It always gives an insight into improvements.
I think improvement is one of the most important metrics to showcase. Being able to see how a company has listened to feedback and improved its software to match the needs of its users provides a lot more confidence to new users than something that has always been highly rated. Because it shows that as time goes on the company prioritises the end-users experience and will do what it takes to keep its users happy.
The Logmate Story
Anyone looking at the Logmate App would be lead to believe that it isn’t very good. We currently have a 2.6 out of 5-star rating on the Google Play Store and a 1.8 out of 5-star rating on the Apple App Store. Definitely doesn’t paint a pretty picture or inspire a lot of confidence for new users. But this isn’t the full story.
In the past 6 months, we have sent out our user feedback survey to 10,332 of our past and current users, with 5,352 of the users being active during that same time.
We found that 58% of those active users read the survey email, 9% clicked on the survey link and 4.9% completed the full survey form. That gives a total number of 262 completed surveys in the past 6 months.
Using some simple statistics we can determine that to have a margin of error of +/-5% we needed a sample size of 258 respondents to be 90% confident of our results.
Through our feedback survey, we have received an overall experience rating of 4.1 out of 5 stars and a 4.1 out of 5 stars specifically for the Logmate App.
We have a schedule in place to send out feedback surveys at specific times along a users journey with us. This gives us the ability to see the particular users experience over time but also gives us the ability to look at all the feedback received at a certain point in time, say after a new version is released or a new product is introduced.
So I think we can conclude that the App Store reviews for Logmate are not actually representative of the true experience of your average active user. It would be interesting to find out if other SaaS companies have the same experience.