What Can be Concluded from User Feedback? - An Empirical Study

Crowd-based Requirements Engineering supports capturing high amounts of user feedback in order to understand what users think about a software and which changes they are interested in. While much progress has been made in automatically classifying the feedback, it is less clear which conclusions can be drawn from feedback. Studies on utilization of feedback in industry report that companies are afraid that user feedback might be biased, because little is known about the users. In this paper, we report a preliminary empirical study with 100 participants where we asked users to give feedback on an app and also describe their comprehension of this app as an approximation of their opinion about the app as whole. We compare feedback and comprehension description by looking at the frequencies of mentioned features. We find that the feedback of frequent users (46 out of 100) differs only a little from the overall feedback. We confirm that feedback is biased in that users individually and as a group know more about the app (mentioned in their comprehension) than they tell in their feedback. However, the feedback does represent which features the users find important in their comprehension.