Status : Verified
| Personal Name | Capuno, Charmagne Faith F. |
|---|---|
| Resource Title | Generating predictive model for user satisfaction of human-to-machine goal-oriented dialogues using a restaurant booking case |
| Date Issued | 17 January 2023 |
| Abstract | In recent year, studies for goal-oriented (GO) chatbots (which execute tasks in lieu of humans) have heavily focused on user intent detection and response generation with standardized evaluation and performance in the research community. For human evaluation, soliciting human ratings for features of concern remains as the most common practice. As this approach is time-consuming, costly and expensive, a means to minimize (if not eliminate) post-interaction user evaluation must be performed. The study explored classification and regression approaches using a restaurant booking dataset to predict satisfaction/dissatisfaction of chatbot user. Regression techniques did not generate relevant predictive models while Classification techniques did at > 80% accuracy, not outperforming baseline human evaluation by much but good enough as substitute. Word sequence models prove to perform better than feature-based regression while word-overlap metrics show no correlation to human ratings. However, the best performing model for the dataset used still does not generalize well for unseen data from other datasets. It is recommended that human evaluation chatbot studies focus on standardizing user rating satisfaction to binary instead of using different scales and developing a standard means of detecting repeat responses, inquiry count, and error count as main features to predict user satisfaction for task-oriented dialogues. |
| Degree Course | MS Industrial Engineering |
| Language | English |
| Keyword | human-to-machine datasets; goal-oriented dialog systems; task-oriented dialog systems; user satisfaction; human evaluation |
| Material Type | Thesis/Dissertation |
Preliminary Pages
128.05 Kb
Category : I - Has patentable or registrable invention of creation.
Access Permission : Limited Access