Final Report

The final report, Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges, (Maxine Eskenazi and Tiancheng Zhao) can be found here.


The topic of the Workshop concerns how our field of research can change its perspective from being often oriented toward the qualities of the agent to fulfilling the needs, judgements and perspectives of the user. The goal of the Workshop is to identify the challenges for future research in dialog and intelligent agents from a user-oriented perspective and to inform the federal funding community. It will explore the present state of the art and envisage the research path of the future for intelligent agents and dialog. Specifically, it will define large scale collaborative projects and community-wide challenges. It will also identify priorities in research and the best directions in which to invest research funding. The workshop will produce a report that integrates input from all participants and conclusions from the workshop, which can be used by funding agencies to shape the future direction of research in this area. To achieve this goal, the USER workshop will gather experts from fields related to intelligent agents and dialog as well as representatives of funding agencies for a discussion of how our research can be better directed toward the user, going from “how may I help you?” to “have I helped you?”

Many intelligent agents (we use this term to also encompass conversational agents and multimodal agents) have appeared as products for general consumption in the past decade. The public has welcomed them with open arms. Public use has brought interesting and novel challenges for the intelligent agent and dialog research community. The community is now in both an enviable position and a fragile one. On the one hand, the acceptance of these agents enables research to push its boundaries, challenged by a flow of data and a plethora of new applications, user interactions and assessments. On the other hand, in order to maintain this golden opportunity, the users’ expectations must be met. Much of the research on intelligent agents has centered on the agent itself: giving it human-like qualities and concentrating on advanced methods for the representation of dialog. While this has shown interesting results, it has not taken the user into account: the user’s assessment of system performance, real user data, the behavior of the user and its evolution during the course of a dialog. If users’ behavior and expectations continue to be disregarded, intelligent agents may be abandoned.

The issue of orienting an agent to serve the user touches on expertise in many domains within the area of Artificial Intelligence such as: advanced dialog systems and machine learning, commercially available agents, dialog assessment, datasets, entrainment and adaptation, ethics. These domains need to cooperate and change dramatically to better consider the user. Focusing on the user challenges the architecture of neural representations of dialog. Care must be taken in the choice and implementation of features of the intelligent agents that are available for purchase. Much assessment at present deems that a system utterance is correct based on past context, but not on the reaction of the user. There are many datasets available for research, but most have synthetic or paid users and lack annotation concerning the final effect of system actions on the user. There has been a body of work on entrainment and adaptation to the user that has often been ignored. Yet this work is central in attending to the user. And the ethical treatment of the user is a concern that must be addressed going forward. More information on the motivation behind this Workshop can be found at