U.S. Office of Naval Research Project
Natural-Language based multi-modal interfaces for wearable maintenance documentation systems
Presentation for the first critique (.pdf)
Presentation for the second critique (.pdf)
The Navy, together with other branches of the military, has been working diligently to convert paper versions of technical documentation to electronic forms of documentation known as Interactive Electronic Technical Manuals (IETMs). The reasons for this effort are well motivated from a practical perspective, and include issues of portability, timeliness of information, and scalability (the amount of paper documentation for one commercial aircraft can more than fill its entire passenger cabin).
Specific expected benefits of performance-aiding systems with advanced IETMs include:
- Access to accurate and updated technical information
- Being able to get help (contact others easily)
- Controlling sign-on/offs
- Flexibility to start/stop task and know current and future status
- Hands/eyes-free access to information
- Interfacing to legacy systems, e.g., for parts ordering
- Maintaining privacy about own training/work performance
- Reliability and durability of hardware/computers
- Seeing evidence that on-line access to documents is more efficient than paper
IETMs are categorized from Class I to Class V, with Class I containing the least amount of interactivity and being little more than a direct scan of documentation. Class IV and V IETMs contain the most opportunities for interaction, as well as the most potential for solving difficulties that have plagued IETM interface designs.
Technical maintenance documentation usually contains information that depends heavily on procedural steps and sequences, cross-referencing and procedures embedded within procedures, checklists, branching, etc. In addition, the information includes notes, cautions, and warnings that escalate in importance, with warnings being extremely important to heed to avoid potential calamity. Finally, diagrams and schematics are integral to the content and have been increasing in complexity in direct proportion to technological advances in military equipment.
- Current IETM designs are insufficient in many aspects of their design, despite the technological potential of Class IV and V IETMs.
- The interfaces are not yet adaptive for maximum usability across technicians’ experience levels.
- MIL Standards that specify how to design the interfaces neglect many core tenets of usability.
- The user’s learning curve is steep for tasks that are simple when using paper, for example, search query and answer and indexing.
- The computer systems themselves are unstable or difficult to use in the field, even as technology continues to evolve.
- There are continuing legacy issues.
It is almost impossible to divorce the effects of an overall computer system from IETM design and usability issues.
For more than a decade, researchers at Carnegie Mellon University and their sponsors have been investigating how to improve performance for wearable computer systems and IETMs. The most recent efforts involve a system known as LARRI (Language Assisted Retrieval of Repair Information), which is a persistent language-based electronic partner that assists humans in efficiently performing complex problem-solving tasks in a maintenance domain. Once developed and tested in a field setting, it is expected that LARRI will provide a means for:
- Intelligent access to technical manuals through natural language querying and navigation
- Integrated equipment, procedure and personal history for effective coaching during performance
- Active assistance in diagnostic and repair procedures
- Support for synchronous and asynchronous work-group collaboration through dialog-based problem reporting and solution capture
The expected impacts include:
- Allowing maintainers to become self-sufficient earlier in their training
- Making maintainers’ core skill transferable over a wide range of specific equipment
- Improving the speed and quality of maintenance work
Introducing multiple modalities, however, requires careful attention to the interaction design and user experience, particularly when integrating multiple systems as well as the means for accessing and using them in work contexts.
Interaction and Interface Design
The potential of Class IV and V IETMs has yet to be realized in terms of interaction and interface design. Nevertheless known principles exist, that can be taken into consideration when authoring and designing such systems. Even so, many issues remain that require additional empirical work, particularly in the field, with actual users.
Previous work at CMU has produced an artifact, the LARRI system, a study examining user interaction with voice-supported on-line documentation and reviews of applicable HCI and human factors research.
The goal of this project is to design, develop and evaluate interfaces for IETM documents. Since this is an ongoing project some of the implementation details have been determined. In particular, the documentation domain is that of SLQ-32 radar systems. These are used on a variety of Navy ships and consist of several components, including consoles below deck and sensor units attached to a mast. Maintenance and repair activities require access to documentation (currently on paper) consisting of ledger-sized sheets bound into volumes. Key maintenance activities require, on occasion that two sailors participate simultaneously, one at the console and another on the mast.
Our goal is to improve the maintenance process by providing improved access to manuals and support mobility. The program has identified three different scenarios, each with specific characteristics that will be used to evaluate the utility of new approaches. We furthermore believe that each type of scenario may require a different interaction style. It is already clear that different types of documentation (e.g., text, procedures, diagrams, tables) require different interaction styles. We will be exploring these issues over the course of this project.
The current LARRI system (or, more correctly, the IMP system) makes use of maintenance documents in XML format, combined with SALT markup that supports voice-based interaction, as well as pointer interaction (mouse or touch). The system includes the Carnegie Mellon OpenSALT browser, Sphinx recognition systems and Festival synthesizer. The system also includes external components developed by other sites, including a simulator from USC and an LSA-based search engine from Pearson/KAT. The system can be hosted on either a wearable computer with a head-mounted display or a more conventional tablet or notebook computer. The hardware and core software components of the system have been selected and will not need to be modified (others in the project will be able to deal with this if it comes up). Class participants will however need to make themselves familiar with these tools and learn to use them to create interfaces tailored for the work at hand.
The specific objectives of this project are to develop instantiations of the LARRI system for each of the target scenarios, implement these using the tools provided and to evaluate the characteristics and usability of the resulting artifacts, first in a laboratory context and later in the field with the targeted user population (USN sailors). We envision an iterative development process, with early prototypes being evolved into robust systems with increasingly sophisticated capabilities, informed by ongoing usability testing. Over the course of the project, we may as well identify scientific or design questions of particular interest that will in turn suggest empirical investigation that we can pursue.
The ultimate goal of the project is to produce robust, working prototypes that both satisfy the needs of the ultimate user and that introduce powerful new concepts into the design of interactive wearable systems.