Submitted to IROS 2003 A Simple and Scalable Approach for HumanSystem Interaction

A Simple and Scalable Approach for Human-System InteractionAshley D. Tews, Gaurav S. Sukhatme, Maja J. MatarićUSC Robotics Research Lab, Center for Robotics and Embedded Systems,Computer Science Department, University of Southern CaliforniaLos Angeles, USA{atews|gaurav|mataric}@Abstract— While there is much research concerning one-to-one human and physical system interaction, there is little on the larger scale of many-to-many. We propose an interaction infrastructure for many human operators to utilize a large number of robots and systems. The primary criterion for scalable interaction is to reduce the communication required for interaction while providing a simple but functional interface. To achieve this, we assume that each system and robot has some degree of autonomy allowing the humans to have a supervisory role in the interaction. Our approach is analyzed in detail on a large-scale experiment involving several systems offering many tasks and services to many users. The scalability criteria are analyzed in the context of this experiment and demonstrate the infrastructure to be a valid interaction mechanism for many systems and many users.1IntroductionThe interaction between a physical system and a human can occur at many levels. There is much research on one-to-one (e.g., single robot and single human) interaction but these approaches would not be applicable for a larger number of interactions. This is mainly due to the large and dedicated communication paths required for interaction between entities, especially if they require tightly coupled interaction.In this paper, we investigate many-to-many interaction: what is required to facilitate it. We intend our method to scale to the order of hundreds of systems and users, where a system is defined as a robot or embedded device. It is anticipated that multitudes of these types of systems will soon be used to achieve goals beyond their current small-scale applications. These goals may be building large structures on Earth, in space, or on other planets. They may also be human-coordinated de-mining or search and rescue efforts, where a few human supervisors can coordinate many robots. It is assumed that many of these applications would use systems that have some degree of autonomy, such that they are supervised by humans rather than operated by them.In its simplest form, human-system interaction consists of the human, the system, and the interaction mechanism. The system and human interact through a communication medium, whether it has an explicit form such as spoken language or message passing, or implicit such as a robot observing a human's deliberate actions. To extend the numbers of entities (humans and systems) capable of interacting, the communication bandwidth required between them must be kept low. Additionally, the interaction mechanism should be efficient and allow for inter-system collaboration to occur if possible. These requirements are the inspiration for the design of our interaction infrastructure.First, the infrastructure requires a low communication bandwidth between itself, users and systems. The assumption that the systems are relatively autonomous contributes to this. Second, the infrastructure is efficient in the sense that the latencies in processing user requests are small. Third, it provides functionality beyond connecting users to systems by providing task queues for systems and allowing services to be combined.To determine how the infrastructure conforms to these requirements, two experiments are presented in this paper. The first demonstrates the task queuing ability with many users accessing a service offered by a single system. The second is a larger scale experiment involving five robotic and embedded systems and hundreds of users. The experiments allow the efficiency and communications aspects of the infrastructure’s operation to be examined.2Related WorkIn [1], a taxonomy is presented categorizing human and intelligent system interaction. It defines an intelligent system as a system that allows user interaction (e.g., tele-robotics[2]). The categories in the taxonomy include the application of the research, researchSubmitted to IROS 2003approach, system autonomy, interaction distance, and interaction media, with each category divided into subcategories. Numerous references are grouped under these categories, illustrating the vastness of the field. A key axis omitted from this classification is the scalability of the interaction methods.In this section of the paper, we classify human-system architectures in terms of their scalability, assuming a system consists of a set of nodes that provide a set of services to users. These classifications are shown in Table 1 and are discussed in the following sections, separating them into single (non-shaded table entries) and multi-user architectures (shaded table entries).TABLE 1U SER/SYSTEM SCALABILITY CLASSIFICATIONUsers Systems1One One2One Multiple3Multiple One4Multiple MultipleSingle-user System ArchitecturesThere are many examples relating to the entries in the table above. The issues concerning scalability are highlighted when examining them. An important issue that affects the scalability of an approach is how tightly coupled the users are with the systems. This concerns both access to system resources and levels of control over system services. A system that allows interaction with only a single user can allow the user full access to system resources and control of its services. To scale this to more users can be difficult due to the service attributes (such as communication bandwidth) having to be shared. Tele-operation of a single system node is an example where this is highlighted [2][3].Multi-robot research also provides single system examples where multiple robots can be controlled individually or as a group by a single user [4][5][6][7]. In each of these systems, the nodes have a certain degree of autonomy and the user has either high-level group control, or low-level individual control. This indicates that the higher the number of users an interaction architecture has to deal with, the looser the coupling between users and systems.The concept of a single user for multiple systems indicates a hierarchical command structure where there is a top-level entity that controls the systems below its level.Multi-user System ArchitecturesMulti-user systems are forced to address issues of scalability, system resource allocation, and control. A key concern is how to control the information and communication pathways for multiple users. Common approaches involve using multi-agent architectures [8][9][10][11] and the Internet.In the context of human-system interaction, multi-agent architectures typically offer services to users with suitable access to the systems they serve. Examples include interactive workspaces [12][13] which consist of dedicated environments with embedded devices providing services for users. The users generally must be in the environment to access the services.JINI [14] is an example of a multiagent architecture that allows systems’ services to be offered to users. There is little restriction on what constitutes a user or a system since a user can offer services as a system would and a system can be a user of services. Each component connected to a JINI architecture is required to run the Java Virtual Machine.Multi-user/multi-robot architectures are demonstrated by [15][16]. These systems use the Internet as a communication medium for commanding robots from remote locations.Many of the examples presented in this section are scalable only in their local operating environment or application domain. Furthermore, they generally have high overhead for implementation and communication. The infrastructure we present in the next section is intended to operate beyond these limits by providing a simple and efficient interaction mechanism.3The InfrastructureFigure 1. A simplified view of the infrastructure for connecting multiple users to multiple system services. Many-to-many interaction is shown as broken lines and potential one-to-one as solid.The infrastructure is based on a server/client architecture as shown in Figure 1. Each system is typically a robot, robot group, or embedded system (e.g., a security camera network) that offers its capabilities as services for the users. Each entity (system or user) has a client that interfaces with the server and manages the low-level signal and message passing betweentheinfrastructure components. It is assumed that the server's contact information is known to all systems and users wishing to use the infrastructure. The server is the centralized hub of operations that manages the allocation of systems' services to users’ requests. The data flow throughout the infrastructure is by UDP communication. This is chosen over TCP primarily to reduce the communication overhead.Each system that connects to the infrastructure sends its service list to be added to the server’s database. The systems may optionally request task queues to be set up for them in the server to facilitate multiple user requests. Each user that connects to the infrastructure is presented with the global service list from the server’s database. The users select whichever services they require and are connected to the corresponding systems if they are not put on a queue. The appropriate level (coupling) of interaction is then established between the user and the selected service, and is system- and service-dependent. The communication between the user and the service is decoupled from the server to maximize bandwidth for interaction and not burden the server from carrying out its higher level operations. However, the bandwidth may be reduced if the user is currently interacting with multiple services. The interaction between a user and a service continues until either terminates it or the service completes. In the latter case, the user and server are notified so the service is ready for another request. A more detailed description of the infrastructure’s operation is provided in [17].4ExperimentsOur experiments were carried out in the Stage simulator using Player devices [18]. Stage has been demonstrated on dozens of projects to provide a valid representation of real robot operations. Player provides a suite of software clients that represent the handlers for the devices on the robots, such as sonar, SICK laser range-finders, and Sony PTZ color cameras. The combination of Player and Stage allows programs developed and tested in Stage to transfer to the real robots.Our first experiment consists of several users requesting a service from a single system. It allows an analysis of the efficiency of the infrastructure while demonstrating the benefit of using task queuing. The second experiment involves more users and systems, to demonstrate the infrastructure’s operation in a higher scale, more realistic application.4.1Experiment 1This experiment shows the operation of the infrastructure using a single system, 25 users, and task queuing. The system provides a random-wander service represented by a robot that randomly wanders about the environment and provides information about its current location. The task lasts for 20 seconds, then terminates and becomes available for the next request. The system, users, and server are run on separate machines.The parameters for the experiment are shown in Table 2. An automated user is generated upon system start-up and every 15 seconds thereafter, until 25 have been generated. Once a user is on line, they immediately select the service if it is available. If it is not, they do not make another attempt. The number of tasks that can be completed ranges from 12 to 25, depending on the size of the task queue. With no task queue, the maximum number of requests that can be fulfilled is determined by the user generation rate versus the task running time. This equates to every other user’s task being fulfilled which is 12 or 13 depending on the efficiency of the infrastructure. When task queues are used, the minimum size of the queue required to manage all requests for this experiment is derived from the difference between task request rate (0.067/s) and task completion rate (0.05/s) for the length of time that tasks are generated (360s). This determines a theoretical queue size of 6.12, which can manage all requests assuming negligible latencies in the infrastructure and system.T ABLE 2P ARAMETERS FOR EXPERIMENT 2Number of users generated25User generation frequency15 secondsExperiment running time500 secondsTask running time20 secondsMinimum achievable requests12 or 13Maximum achievable requests25Queue size0, 3, 5, or 7For the buffered approaches, the server’s response time to a user’s request is generally between 60 and 80ms, regardless of the buffer size. The approach with no buffering is indifferent to whether the user’s requests are fulfilled or denied and the response times are between 50 and 60ms. The statistics for task fulfillment are shown in Table 3.T ABLE 3U SERS’ REQUESTS FULFILLEDBuffer Size Requests Fulfilled013 (52%)321 (84%)523 (92%)725 (100%)These results demonstrate that a buffer size of seven allows all users’ requests to be fulfilled within theconstraints of the experiment parameters. It also demonstrates that the time for the server to manage these requests is negligible, since the theoretical maximum number of requests that could be fulfilled within the time limit was achieved. This would not be possible if the server was inefficient with processing the queued requests.The times taken for the system to respond to requests are shown in Figure 2. These times are measured from when the user submitted a request to when they were notified their request was being processed by the system.Unsuccessful attempts are recorded as a response time of 0 to identify the corresponding requests.Figure 2. User client service request response times (solid lines corresponding to the left y axis) compared to the number of requests in the queue (broken lines corresponding to the right y axis).For the system with no task queuing (not shown in Figure 2), the response times consistently averaged 75ms.For the buffered approaches, the response times varied depending on the size of the task queue. In all buffered approaches, the first 10 users had their requests fulfilled.Once each buffer is full, the next generated request is denied. Since the task length is 20 seconds and the first user in the queue is currently connected to the task, each approach has a maximum theoretical response time for the requests of:(buffer_size – 1) x task_length (1)This gives 40, 80 and 120 seconds for the three, five and seven-buffer approaches, respectively. These upper limits are apparent in the graph.4.2 Experiment 2This experiment presents an indication of the scalability aspects of the infrastructure, using five systems located in simulated office space and hundreds of users over four hours of operation. Each system consists of robots or embedded devices and provides varying numbers and types of service. The services are classified as discrete or continuous. Discrete services provide onlythe immediate status of a sensor while continuous services provide streaming updates of their task status.Each system is briefly described next.1) Murdoch – continuous servicesA task-allocation system, Murdoch [19][20] is a form of a Contract Net Protocol [21] used to control three robots in this experiment. Each robot is capable of carrying out three tasks: random walk, target tracking,and delivery. Random walking requires a robot to wander randomly and provides location information to the user.For target tracking, the robot acquires and follows a colored mobile object. The object’s position is provided to the user. The delivery service requires the user to input the destination coordinates for the robot to navigate to.Once a robot is tasked, it does not carry out another task until the current one is terminated. The Murdoch-controlled robots each have a task buffer set up for them on the server.2) Tracking – continuous servicesEight robots are positioned to provide almost complete coverage of the corridors using a tracking system from [22]. Each is equipped with a color camera and reports the global positions of simulated humans. This system provides three services concurrently and the system client is configured to manage up to ten simultaneous requests.3) Room Cleaning/Trashing – continuous servicesThis system involves a robot carrying out room cleaning and object scattering tasks. The cleaning task is similar to vacuuming or trash-collecting where a robot moves objects to a certain area in a room. Trashing involves dispersing these objects. The users can request either of these services and the system manages one request at a time.4 & 5) Room Lights and Room Occupied - discrete servicesThirty-three rooms in the simulated building have sensors that determine whether the room is occupied and the status of its lights. A room is considered occupied if it contains a simulated person. For the Room Lights system, the lights are automated to switch on and off at random times.4.2.1 SetupStage is used to simulate the robots and the environment which is the second floor of the USC Computer Science building (Figure 3). This presents a realistic office space environment with corridors and rooms for the robots to carry out their tasks, and allows the systems’ parameter settings to be used for testing in the real world [17]. Apart from the service robots, there are also three simulated humans (implemented as robots)1113151719212325ClientR e s p o n s e T i m e (m s )N u m b e r o f Q u e u e d T a s k sthat wander randomly through rooms and corridors. Each “human” has a differently colored marker for identification.Figure 3. The simulated environment showing robots in their initial positions.The statistics for the experiment configuration are shown in Table 4. In the table, a ‘node’ is defined as either a robot or embedded device. There are 80 services in total. Note that the tracking system has eight nodes capable of three services each, but the system only provides three services in total. This is due to each robot operating in parallel with its peers, so they form a global, rather than a local sensor network.T ABLE 4E XPERIMENT ENTITIES AND SERVICESSystem No. ofNodes Servicesper nodeTotalSystemServicesTotalUsersperSystemMurdoch339 3 (+6) Tracking83310 Cleanup1221RoomLights33133N/aRoomOccupied33133N/a T OTAL7880In Table 4, the ‘Total Users per System’ column indicates how many users can utilize the system simultaneously. Each Murdoch system allocates one robot per request and allows two requests to be buffered in the task queue, hence the ‘+6’. The discrete services do not have a limit to the number of users they can manage since requests are handled with a single response. All systems except Cleanup remain connected to the server while managing requests.The users are real and simulated. The simulated users are generated randomly on several different computers and can make random service selections. The maximum number of services they can be interacting with is arbitrarily chosen to be two. The user clients, system clients and server connect to each other over a 10Mb/slocal area network and the experiment is conducted over four continuous hours of operation.4.2.2ResultsThe overall statistics of the experiment are shown in Table 5. Over 200 users per hour connected with a maximum of 32 being simultaneously on line. Not all users requested services and of those that did, approximately 80% of the requests were fulfilled. Requests were rejected for several reasons depending on the system:Murdoch - the request queues are full;Tracking- the maximum number of user requests is being managed;Cleanup - a request is made from an outdated service list. Other rejections occurred if a user selected a service and logged out before notification was received from the system that the request is being processed.T ABLE 5S TATISTICS FOR EXPERIMENT 2Total users985Maximum users on line32Total requested services916Total fulfilled services730Total requested service lists2022The high number of service list requests results from the users initially being sent the global list of services from the server upon connection. Users also generally refreshed their list prior to selecting a service to ensure they were not making selections from an outdated list. The system usage statistics are shown in Table 6. The continuous service systems were connected to the infrastructure for the longest period of time but had fewerrequests due to the large number of discrete services, and the more lengthy running times of the continuous services.T ABLE 6S YSTEM USAGE STATISTICSSystemNo. of Services Running Time (%)No.Requests Request Ratio (%)Murdoch 910015116.4Tracking 310014215.5Cleaner 277.924 2.6Room occ.3365.834737.9Room lights3353.325327.6The times for the systems to respond to user requests is shown in Figure 4. This is measured as the time the users had to wait before getting acknowledgement from the system that their request is being fulfilled. Generally, the times are low except for the Murdoch systems which had task queuing. Since there were no time limits imposed for service usage, a queued request took an arbitrary amount of time to be fulfilled.Figure 4. Example response times for users requesting services.Server StatisticsApart from the number of systems, services, and users the infrastructure managed during the experiment,another important analysis of scalability is the communication throughput for the infrastructure’s operations. Since the server is the centralized hub of operations, the following analysis is based on its communication with the clients.The most time-consuming and data intensive transaction for the server is processing requests for the global service list. Figure 5 shows the throughput of information for these requests during the experiment.The size of the requests are shown in gray (bytes received) with the service list size (bytes sent) in black.The steps in the service list size indicate when a new system was added to the server’s database and its services’ descriptions are added to the global list. Thesmall fluctuations indicate when services were unavailable. This occurred often for the Cleanup system.The other requests sent to the server are service requests and service termination messages. A service termination message can be sent from either the system or a user connected to the service. The user sends this message when finished with the service. The system sends this message to indicate the service is to be removed from the global service list. These messages are typically 100 bytes or less. The average time for the server to process both types of message is less than a millisecond with the maximum time less than 40ms.Figure 5. The number of bytes sent and received by the server in response to a service list request.Figure 6 illustrates how the rate of requests affects the server's response times for all messages it received. The maximum request rate was 11 per second. Upon analyzing the requests during the high burst time, most were found to be service list requests, which require the most bandwidth. As can be seen from the response times,the server’s efficiency was not affected by this rate. The size of the service list, however, did have an affect that is apparent by correlating the steps in response times in Figure 6 to the steps in ‘Bytes Sent’ in Figure 5.Figure 6. Number of responses/second versus response time for the server.02040608010012014055527263886548870378303939510912123051354014715Event Time (s)554238042066033785996851151113338Time (sec)N u m b e r o f B y t e s S e n t20000400006000080000100000120000140000Cleaning Murdoch Light Status Room Status Tracking5SummaryThe primary issues identified to allow large numbers of users to interact with large numbers of systems include reducing the amount of communication required for interaction, providing a highly efficient interaction mechanism as a substrate for potential interactions, and providing additional functionality for managing multiple systems and users. Our interaction infrastructure has addressed each of these issues. By assuming that the systems are relatively autonomous, much of the communication overhead is reduced since simple commands can be issued to task a system. Moreover, the communication required for the general operation of the infrastructure is minimal. This increases the infrastructure’s efficiency in managing requests. Finally, the extended functionality is provided by allowing task queuing and service linking.We have demonstrated each of these issues in the experiments presented here and in our previous work with real robots [17]. The next stage of evaluation is to determine the scalability limits of the infrastructure in terms of the number of systems it can manage simultaneously, the maximum frequency of requests, and message throughput. Further development on the infrastructure includes distributing the server to provide redundancy and increased scalability.Bibliography[1] A. Agah, "Human Interactions with Intelligent Systems: ResearchTaxonomy", Computers and Electrical Engineering, 2001, no. 27, pp. 71-107.[2]R. Murphy, and E. Rogers, "Cooperative Assistance for RemoteRobot Supervision", Presence, Special Issue, vol. 5, no. 2, pp. 224-240, 1996.[3]K. Kawamura, R. Peters, C. Johnson, P. Nilas, P. and S. Thongchai,"Supervisory Control of Mobile Robots Using Sensory Egosphere", IEEE Symposium on Computational Intelligence in Robotics and Automation, Jul. 29-Aug. 01, Banff, Canada, 2001.[4]K. Ali, and R. Arkin, "Integration of Reactive and TeleroboticControl in Multi-agent Robotics Systems", From Animals to Animats 3: SAB 1994, pp. 473-478.[5]R. Arkin, T. Collins, and Y. Endo, "Tactical Mobile Robot MissionSpecification and Execution", Mobile Robots XIV,Boston, MA, September, 1999, pp. 150-163.[6]H. Jones, and M. Snyder, M. "Supervisory Control of MultipleRobots Based on a Real-time Strategy Game Interaction Paradigm", Proceedings of the 2001 IEEE International Conference on Systems, Man, and Cybernetics, Tucson, AZ, October 2001, pp.383-388.[7]P. Rybski, I. Burt, T. Dahlin, M. Gini, D. Hougen, D. Krantz, F.Nageotte, N. Papanikolopoulos, and S. Stoeter, "System Architecture for Versatile Autonomous and Teleoperated Control of Multiple Miniature Robots", Proceedings of the 2001 IEEE International Conference on Robotics and Automation, Seoul, Korea, 2001. [8]T. Finin, R. Fritzson, D. McKay, and R. McEntire, "KQML as anAgent Communication Language", Proceedings of the Third International Conference on Information and Knowledge Management, ACM Press, 1994, pp. 456-463.[9]S. Goldsmith, S. Spires, and L. Phillips, "Object Frameworks forAgent System Development", Sandia National Laboratories, Tech.Rep. SAND98-0742A, 1998.[10]L. Phillips, S. Goldsmith, and S. Spires, "CHI: A General AgentCommunication Framework", Sandia National Laboratories, Tech.Rep. SAND98-2825C, 1998.[11]S. Ponnekanti, B. Lee, A. Fox, P. Hanrahan, and T. Winograd,"ICrafter: A Service Framework for Ubiquitous Computing Environments", Proceedings of the Ubiquitous Computing Conference. Lecture Notes in Computer Science", vol. 2201, 2001, pp. 56-75.[12] A. Fox, B. Johanson, P. Hanrahan, and T. Winograd, "IntegratingInformation Appliances into an Interactive Workspace", IEEE Computer Graphics and Applications, May/June, 2000, pp. 54-65.[13]J. Lee, and H. Hashimoto, "Intelligent Space - Its concept andcontents", Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Takamatsu, Japan, 2000, vol. 2. pp. 1358-1363.[14]Sun Microsystems. Jini Network Technology,[Online], Availablefrom: /software/jini [13 March 2003].[15]I. Elhajj, J. Tan, N. Xi, W. Fung, Y. Liu, T. Kaga, Y. Hasegawa, andT. Fukuda, "Multi-site Internet-based Cooperative Control of Robotic Operations", Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems, Takamatsu, Japan, 2000, pp.826-831.[16]H. Surmann, and M. Theißinger, "ROBODIS: A Dispatching Systemfor Multiple Autonomous Service Robots", Proceedings of the FSR'99 Robotics Applications for the Next Millenium, Pittsburgh, PA, 1999, pp. 168-173.[17] A. Tews, M. Matarić, and G. Sukhatme, “A Scalable Approach toHuman-Robot Interaction”, accepted, IEEE International Conference on Robotics and Automation, Taipei, Taiwan, 2003. [18] B. Gerkey, R. Vaughan, K. Støy, A. Howard, G. Sukhatme, and M.Matarić, "Most Valuable Player: A Robot Device Server for Distributed Control", Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2001), Wailea, Hawaii, Oct. 29-Nov. 3, 2001, pp. 1226-1231.[19] B. Gerkey, and M. Matarić, "Principled Communication forDynamic Multi-Robot Task Allocation", Experimental Robotics VII, LCNIS 271, Springer-Verlag, 2001, pp. 353-362.[20] B. Gerkey, and M. Matarić, "Sold!: Auction Methods for Multi-Robot Coordination", in IEEE Transactions on Robotics and Automation Special Issue on Multi-robot Systems, vol. 18, no. 5, Tamio Arai, Enrico Pagello and Lynne Parker Eds., 2002, pp. 758-768.[21]R. Davis and R. Smith, “Negotiation as a Metaphor for DistributedProblem Solving, Artificial Intelligence, vol. 20, no. 1, 1983, pp. 63-109.[22] B. Jung, and G. Sukhatme, "A Region-based Approach forCooperative Multi-Target Tracking in a Structured Environment,"accepted, IEEE/RSJ International Conference on Intelligent Robots and Systems, Switzerland, Sep. 30-Oct. 04, 2002, pp. 2764-2769.。