Don’t go in there! using the APEX framework in the design of ambient assisted living systems

An approach to design Ambient Assisted Living systems is presented, which is based on APEX, a framework for prototyping ubiquitous environments. The approach is illustrated through the design of a smart environment within a care home for older people. Prototypes allow participants in the design process to experience the proposed design and enable developers to explore design alternatives rapidly. APEX provides the means to explore alternative environment designs virtually. The prototypes developed with APEX offered a mediating representation, allowing users to be involved in the design process. A group of residents in a city-based care home were involved in the design. The paper describes the design process as well as lessons learned for the future design of AAL systems.

Participation of users in the development of AAL systems is crucial to their success and according to Brereton and Buur (2008) participatory design faces new challenges in the context of ubiquitous systems. The participatory design process should not simply be a means of developing acceptable solutions but should also be a basis for exploring and experiencing the ubiquitous system. New challenges to participation arise because a ubiquitous system is, by its nature, immersive. Exploring a design with an individual necessarily takes them out of the world that the environment is designed to create. "Calm technology engages both the center and the periphery of our attention, and in fact moves back and forth between the two" (Weiser and Brown 1996). The problem for the designer is to explore the requirements for such a system with its potential users without intruding on these fundamental issues of attention. These requirements are not just about ease of use, they are about the experience that the user obtains from being immersed in the proposed system. Any early development of design concepts therefore must enable the users of the proposed design to reflect accurately on what their experience will be of the final design without their reflection being compromised by the limitations of the prototyping medium. This is particularly relevant if the user group is likely to be hostile to proposed designs, if they are outside their experience, if they involve novel technologies and techniques.
Rapid prototyping can help to explore a user's experience of a candidate design early in the development process. In principle this can be achieved at minimal cost while at the same time reducing disruption to the target environment. Different types of prototyping can be useful, for example focusing on device, hardware or software, or considering the environment as a whole in terms of workflows or the aggregated physical features of the environment. APEX is designed to address aggregated whole environment behavior (Silva et al. 2012. It provides a framework that combines modeling of the control logic of the devices in a proposed environment with a virtual reality (VR) simulation of the target environment. Based on our experience with APEX, in this paper we propose a design process for AAL systems. The process is illustrated with an example.
The particular design example described in this paper involved older occupants of a residential home in a city location in Portugal. Lindsay et al. (2012), when describing their OASIS technique, note that older people are a very diverse group. They observe that attention spans relating to systems are commonly short because of a lack of interest in the technology. Older people often have little enthusiasm for envisaging the role that technology can play and to propose new or alternative designs in a given scenario. Providing aids to visualisations and relevant scenarios, in which the groups can engage, helped overcome these problems.
The APEX prototyping approach was used therefore as the medium of communication in a participatory design process for a proposed AAL in a care home for the elderly. The focus was a "concern for the user's point of view" (Halskov and Hansen 2015). A virtual environment, with connected physical devices, was used to enable participants to explore design ideas, to explore their needs and to contribute suggestions for redesign. The prototype provided a vivid and appropriate experience for participants. They were sufficiently immersed in the proposed environment that it was as if they were there. For example, one participant expressed concern about her privacy when other participants began to enter her room in the virtual environment while exploring a scenario. The environment was not threatening. It simply extended the kind of experience they were already used to while watching television.
The paper describes the proposed participatory design process and its instantiation in the care home scenario. The prototyping environment, the prototype that was built and its evaluation are discussed, together with potential alternative designs and design suggestions that resulted from the process. This study is compared with related work. We conclude the paper by discussing lessons learned and proposing a roadmap for future work in the area. The paper makes four main contributions.
--It demonstrates a participatory design process, based on the use of virtual reality prototypes, for the design of AAL systems. --It illustrates how the APEX environment enables rapid development of alternative designs, making design ideas more concrete for participants. --It gives examples of how a mixed reality environment enables older participants to engage more effectively with the design concepts and to provide constructive feedback about design proposals. --It proposes a roadmap for developing mixed reality prototypes for ubicomp prototyping.
An initial version of this paper was published in Campos et al. (2015). The current paper extends the previous paper by articulating the design approach for AAL more thoroughly. A more thorough description of the prototype is presented with more details about what was learnt from the analysis as an illustration of applying the approach. Related work has also been extended, and a roadmap for future work is now discussed. The remainder of the paper is structured as follows. Sections 2 and 3 introduce the APEX framework and the proposed design approach, respectively. Section 4 describes the care home and the initial stages of applying the approach. Sections 5 and 6 present the APEX prototype used and its evaluation in detail. Section 7 discusses related work, and Sect. 8 the validity of the work and lessons learnt. Section 9 proposes a road map for future work, and Sect. 10 ends the paper with conclusions.

APEX prototyping
The APEX framework integrates an existing 3D Application Server (OpenSimulator 1 ) with a modeling tool (CPN Tools 2 ) and physical devices (e.g. smartphones). APEXbased prototypes enable users to navigate and interact with a virtual world simulation as well as some of the physical devices of the envisaged ubiquitous environment. A design process is envisaged in which the environment can be gradually made more concrete by substituting actual physical devices for virtual devices. Users can experience many of the features of the proposed design. In other words, the user can experience an element of the system virtually and then subsequently use an implementation of the device connected to the virtual environment. The three distinctive features of APEX are that it: --Allows rapid and multi-layered prototyping of ubicomp environments; --Provides a 3D virtual environment as a basis for representing the system to be developed in a way that can be explored by users through an immersive experience; --Enables the connection of actual devices, as intended for the envisaged ubicomp environment, providing a more immersive user experience.
3D application servers, such as SecondLife 3 or OpenSimulator provide a fast means of developing virtual worlds. OpenSimulator, which is the server used in APEX, has the advantage of being open source. This has made it possible to extend and configure the tools more effectively. A multi-layered prototyping approach enables APEX to use scripts associated with the 3D objects and/or a formal notation, Coloured Petri Nets (CPN) (Jensen et al. 2007), to describe the behavior of the virtual environment. If a behavioral model, specified in CPN, is created to drive the virtual environment, it provides support for exhaustive and systematic analysis of the environments' behavior. The communication between the CPN model and OpenSimulator is achieved using a specially designed APEX component. The prototype uses a combination of purpose built 1 http://opensimulator.org (last accessed: 26 July 2016). 2 http://cpntools.org/ (last accessed: 26 July 2016). 3 http://secondlife.com (last accessed: 3 August 2016). components , object warehouses, and an appropriate off-the-shelf viewer (e.g. Cool VL viewer 4 ).
APEX supports the construction of interactive and rich environments, providing users with an experience close to that of being in the real environment. Several users can establish connections simultaneously, using different points of view in the OpenSimulator server. Users experience the proposed solution, as avatars, by navigating and interacting with the simulation and with each other (e.g. by chat, movement, sound, etc.). The avatar can be controlled by mouse/keyboard, Wiimote or smartphone.
Several ubicomp prototypes, mostly based on existing physical spaces, have been developed as part of the framework's design and development. Examples include a smart library (Abade et al. , 2015, an AAL system aimed at children who are asthma sufferers , as well as the system used as illustration in this paper .

The design approach
The described design approach results from experience gained by developing and testing several ubicomp prototypes using APEX. The approach adopts the conventions of many user centred design practices. It involves four steps of an iterative process: establishing and refining requirements (R); producing or refining a prototype (P); evaluating the current design with the stakeholder group (E); summarising the conclusions from the evaluations and returning to refine the design (I).

Establishing requirements
At this stage a set of requirements for a new system is developed with a group of stakeholders. The goal is to define current stakeholders' needs and define a set of initial requirements. This first stage should provide the basic success criteria against which the project and the designs are to be measured. It will also help designers gain initial insights about how AAL technologies might be used to address the identified needs.

Creating the prototype
The design proposals, established as a result of the early discussions above, are used to drive the design of an initial prototype. Concrete ideas about how the requirements might be addressed by AAL technology are captured in this prototype. The technical feasibility of different solutions must be considered and, when relevant, alternatives explored.
The prototype will typically be developed using the APEX tools, although lower fidelity options might be considered in the initial stage of the process. The purpose of the prototype is to enable the targeted user group to experience the design and to elicit their experience. It will be the medium of communication for the participatory design.

Evaluating the design
A group of stakeholders is brought together in a meeting, to experience the prototype design and to feed back comments. This might be the same group as in the previous requirements gathering phase, or it might include further stakeholders (e.g. end users' involvement might be deferred until this stage). While ideally end users should be involved as early as possible, this will be dependent on the specificities of each particular design context. It is intended that this is an exploratory session with the stakeholders group.
Possible alternative design ideas are developed and experienced during the meeting. APEX supports different degrees of immersiveness, from using a full immersion setup to presenting the envisaged environment, such as a CAVE, to using a large screen. Multiple users can also access this system simultaneously from different machines. In previous conditions we have had up to 25 simultaneous users connected to the framework. The level of immersion used will depend on the nature of the target group and technical feasibility.
Typical viewers combine viewing and editing facilities. They are used by developers to create the prototype, and by stakeholders to experience it. This makes it possible to update the prototype at runtime, during evaluation. When providing direct access to the prototype to multiple users, care must be taken to control the changes that are being made to the environment. The ideal solution is to centralise changes (by disabling authoring rights to participants) so that changes made by one participant will not disrupt the experience of others.

Iterating the design
As a result of the evaluation meeting, comments and design alternatives are collected and these are used to inform the next step of the iteration. With this information a new refinement of the design is created and the prototype updated. In some cases this may involve substituting physical implementations for virtual implementations. For example, a virtual device for calling for help may be replaced by a physical implementation. This physical implementation, perhaps using a smart phone as its platform, is then used in the virtual environment. In other cases it might imply rethinking the scope of what is considered the target environment for the AAL system.

The illustrative example: a care home
The effectiveness of the APEX framework in the context of the proposed design approach is now illustrated through an example.

The house
'Casa do Professor' is a private non-profit social-welfare association aimed at teachers. The organization initially provided cultural and leisure services to its members. The association has gradually extended its scope and today offers a range of services, including continuous professional development and a residential home targeted towards retired teachers.
The care home is set in a house in a city centre location in Braga, Portugal (see Fig. 1). This historic building has been extensively adapted for the purpose. It is also used as the headquarters for the association as well as providing conferencing and administrative facilities. The need to adapt an existing building for these multiple uses has placed restrictions on the building's interior. As a result the organization of the rooms and their connection via corridors is complex, making navigation difficult. The ground floor contains a living room, a dining room and a bar. It also contains offices and a reception area. The basement contains an auditorium and other rooms for meetings and workshops. There are also services for the residents in the basement, for example, hairdressing and some medical care. The residents' private rooms are found on the first and second floors.
At the time of the study, the house accommodated more than twenty residents. Support services, such as medical care, are provided twenty-four hours a day. Several services are provided specifically for residents, who may also participate in other activities that are aimed at all members of the association. The house mixes public and private spaces and public and private activities. This requires a degree of openness that can hinder activities designed to ensure the safety of residents. It is not possible for example to log those who enter or leave the building.
The aim of the project was to design an AAL system that could be used to help manage the space and provide relevant services to its users. The system aims to cater for the diverse needs of residents, their carers and management. It was required that the designed facilities should not be disruptive to the residents' everyday lives. The first stage in designing the environment involved meeting with the institutional stakeholders to obtain their views about what facilities would be useful in the house. These meetings provided the material for an initial design that was later further discussed and developed. The second stage involved a participative design session with a group of residents using the design and possible variants as developed in the first stage, as a basis for exploration. Ideally we would have liked to engage with residents earlier in the process, however management were concerned to keep disruption to the residents' daily routine to a minimum. Additionally we were asked to postpone engagement with residents until such time as a concrete design proposal was available to be discussed.

Defining initial requirements (R)
Four meetings with institutional stakeholders provided material for the basic requirements upon which the initial designs were based. The association's director established, at the first meeting, an understanding of why the association wished to introduce relevant technologies. The meeting established the conditions for viability of the project, providing the common ground needed for the project to move forward. The head of the care home was assigned to be the association's contact point.
The next three meetings involved the head of the care home who had been unable to attend the first meeting. The first of these (the second in the process) began with a brief overview of the discussion and ground rules established at the first meeting. The meeting continued by exploring how services could be facilitated using a ubiquitous computing environment. This discussion led to concrete ideas based on the head's view of how the environment might satisfy care-home requirements. The requirements' focus was on what the home needed rather than the technology.
The following requirements were identified: --Knowing the whereabouts of care home residents -the complexities and intricacies of the internal architecture of the house made this issue particularly difficult. A resident could be located anywhere on different floors. Moving locations might involve different stairs and different paths. Inevitably, individual security was discussed as an important issue. The open nature of the house and the need for individual freedom made this a complex issue. --Being aware of whether tenants are in their rooms or not -this was a requirement triggered by the previous discussion. Carers, as they patrol the corridors, are concerned to know whether residents are in their rooms without having to knock on doors. --Providing the means for tenants to call for assistance, ensuring a distinction between urgent and non-urgent situations-carers and inhabitants should be aware of any potentially dangerous situations, therefore a system was required to help residents communicate with their carers. A call button is already available in rooms, but the fact that it does not differentiate between urgent and non urgent calls means it is not an ideal solution.

The first prototype (P) and its evaluation (E)
The above requirements provided the basis for defining a ubicomp environment that offered the desired services. A "paper prototype" that represented a sketch design based on ideas that had been developed in the earlier meeting was developed. This prototype was used in a third meeting, as a basis for discussion of how the required services should be delivered to carers and inhabitants. One of the issues discussed related to the fact that the use of GPS was not possible inside the house. It was proposed that alternatively WiFi could be used to provide detection but at a coarser level. The head of the home agreed that providing the relevant equipment was technically feasible in the context of existing facilities. No details about types of device were considered at this stage. A light by each resident's door was proposed as a means of indicating whether the room was occupied. Buttons by the door were proposed as a means of enabling a resident to call for help. In discussion it was felt that two buttons should distinguish between urgent and non-urgent calls. Their placement was discussed and a particular location by the door agreed upon. All suggestions were accepted as useful by the head of the care home.

The second prototype (P) and its evaluation (E)
A system prototype was then developed to explore these design ideas with the target users. The residents did not have a high degree of computer literacy. Paper prototyping would have been too low fidelity to allow stakeholders a feel for what it would be like to be in the ubicomp environment. Building and deploying an initial version of the system could in principle have been a feasible option, using recent embedded technologies such as Arduino, but it would have been too disruptive to the house's operation, and to the residents' own daily routine. The APEX environment was used to produce the design and enabled a degree of immersion so that users were able to experience the system (see Sect. 2).
The developed prototype of the proposed system was presented to the head of the care-home at a fourth meeting. The aim of the meeting was to check that the requirements established in these early meetings had been met. Feedback was positive. An issue that arose as a result of the meeting concerned exploring the possibility of using a floor located light to guide people to the toilet at night.
The prototype, and its modifications in the light of the fourth meeting, provided the basis for participatory design with the residents. The participatory design process, and what was learned from the meeting that took place, will be discussed after describing more details of the prototype and how it was developed using the APEX framework.

The APEX prototype (P)
A prototype was developed to encapsulate and explore the discussed design ideas. This section presents the second iteration of the APEX prototype. It was important that the prototype should support sufficient features of the proposed design and sufficient texture of the environment to enable the residents to experience the systems as if they were there. Additionally, the way in which the prototype was presented to users should not be detrimental to their experience of an implemented system based on the prototype. The prototype AAL system proposed for Casa do Professor's residential home included a "virtual home" and an Android application that used the phone's motion sensor. A virtual world was developed to recreate the care-home.

The virtual world
A virtual world was created to represent one of the floors exclusively dedicated to residential use. The floor is composed of ten bedrooms connected by corridors laid out around a central stairwell and elevator shaft. Two of the rooms are accessed through a bridge over the main stairs on one side of the building (see blueprint in Fig. 2).
Creating the virtual world involved two steps. First a skeleton of the 3D virtual environment was created. This was done by importing the blueprint in Figure 2 into SweetHome3D 5 (a free interior design application), drawing the walls, windows and other architectural details over it, and then using the tool to generate the corresponding 3D environment (see Fig. 3). This skeleton was exported as an OBJ file and then transformed to a COLLADA file using Blender 6 for the second step of the process. In that second step the environment was enriched with furniture taken from Google's 3D Datawarehouse using a viewer, and pictures of the home's interior and surroundings. Pictures of the interior were used as memory aids, to guarantee the virtual world was as faithful as possible to the actual house. Pictures of the surrounding area, as seen though the windows of the building, were used to make the environment more realistic. Figure 4 shows, side by side, an actual room from the house and its virtual counterpart. The virtual environment required two person days of effort once the blueprint and photographs were available.
The APEX multi-layered prototyping approach was used to produce the prototype for evaluation. The simulation layer was used to provide early experience for the users, simulating in the virtual environment the behavior of the envisaged AAL system. System behavior was specified using scripts associated with 3D objects. The modeling layer was not used in this phase, as analysis of the behavior of the environment was not required until later in the development process. This layer can be added easily at a later stage. The physical layer was used both to provide a more immersive user experience and to augment the environment with a physical device (i.e. smartphone) designed to behave as it would in the target system, as well as a fall detection system using the phone's motion sensor.

Simulated technology
Each simulated bedroom was equipped with two buttons placed by the door, and a presence light placed outside the room over the door. The buttons, one red (emergency) and one yellow (normal call), generated notifications for carers. The presence light, initially green, was programmed to turn red whenever the avatar entered the room. Adding these features to the model took two to three hours. This included the time to add each object to the world and to program its behavior.
A mobile Android app was developed. It was designed for use by the staff in the house, so that it was possible to receive notifications from the system. The app features a map of the house (see Fig. 5), and whenever an alarm situation is detected a notification is generated on the phone, and the location of the alarm indicated on the map (see red dot in the figure). Two additional features were implemented. One, featured in this specific prototype, enables resetting the presence lights in the rooms. The other, more general purpose, enables using the phone's accelerometer to navigate the world. Communication between the APEX server and the mobile phones is done over the WiFi network. Alarms can be generated by both virtual devices (the buttons in the rooms) and physical devices (see below).
A specific alarm situation considered was fall detection and notification. Hence, a further development, this time requested by the residents, used the mobile phone's integrated accelerometer as a fall detector initiating the alert mechanism. The phone was programmed such that whenever a sudden movement was detected, a predefined command was sent to the APEX server. The server was itself programmed to act on that command by calculating the position of the avatar associated with the device and generating an alert. This alert was then communicated to the app described above to inform staff when a resident had suffered a fall.
As stated, the location of the alarm is obtained from the position, in the virtual world, of the avatar associated with the device generating the alarm. While an exact location can be obtained from the virtual world, in practice this location would be approximate, as the proposal was to use the WiFi signal for estimating a resident's location. At this stage the focus was in understanding the users' reaction to the functionality, rather that exploring its implementation.
Developing the app took two weeks. This component of the prototype took longer to implement than other parts of the prototype but it should be noted that it is an actual Android application. This development can be seen as a step in the process of evolutionary prototyping, leading towards the final application. Furthermore the developer was learning the technology while developing the app.
Finally, implementing the floor lights to guide people to the toilet at night was done in one hour. This last-minute addition was produced in the time between the final meeting with the head of the house and when the focus group was run.

The evaluation (E)
The prototype just described was used as the medium of communication for participatory design. At this stage it had to be decided how the prototype would be presented to the residents in the house. As already discussed, a prototype of this kind could be used by test participants either through personal computers or in a CAVE environment if available. The evaluation environment was to be a room within the care home rather than a laboratory, to minimize disruption to the target audience. This invalidated the option of using the CAVE environment. Using personal computers had the advantage of facilitating the simultaneous exploration of the prototype by multiple users. However, such an approach would have been intimidating for the user group and any results from exploration would have been affected by the simple problem of using the personal computer. To facilitate the presentation of the prototype and, at the same time, encourage discussion between the residents, a focus group format was chosen where the prototype was used to illustrate different scenarios to the residents. The environment was displayed on a screen visible to all members of the group.
A laptop computer, running the prototype, was connected to a large screen TV, and the participants were seated on sofas around the TV. The APEX team members stood by the TV in front of the group. The system was controlled by one of the team members while a second member recorded the discussion. This provided a setting similar to that of watching a play or movie on TV, something the members of the focus group would be accustomed to. The idea was that the presentation of a group of scenarios would be used as a baseline with the possibility of changing the trajectory of any of the scenarios in response to user feedback. The team member who was controlling the prototype asked questions and promoted discussion within the focus group.

The focus group
The focus group consisted of eleven people involved in the care home: nine residents, a psychologist who already met with this group of residents weekly, and the head of the care home. The style and the format was to be as consistent as possible with the meeting that the psychologist already led. Prior to the focus group session, a meeting was held with the psychologist. The purpose of the focus group session was explained and the psychologist agreed to allow it to go ahead. At a later meeting she proposed the session to the group, explaining its context and its goal. The members of the group all agreed to participate in the focus group. The session was then scheduled in a meeting room in the house. Group members were all in the 70+ age group. They expressed no knowledge or understanding of smart houses or ambient assisted living. When the virtual environment was presented it was possible to observe that residents identified the environment with the house. They were able to identify which rooms belonged to whom. The fact that the group were immersed in the environment in a visceral way was particularly evident when one of the residents became anxious when the demonstrator used the avatar to enter her room. When the avatar started walking towards the room, the resident first commented that the room belonged to her.
When the demonstrator did not understand what was happening the psychologist suggested that he entered the next room instead: "Don't go in there, why don't you go to the next room?" This situation was repeated at a later stage, and at that time the psychologist made a signal for the room not to be entered.

The design exploration
Five scenarios were illustrated. The APEX team member in charge of presentation would illustrate the scenario by using the prototype to enact a situation where the ubicomp technologies present in the environment would be used and useful. The participants' views in relation to the scenario as well as the role of the technology presented were then recorded. Depending on the participants' reaction, questions or further alternatives were put forward.
Initially the idea of adding guiding lights to the bathroom was illustrated using the prototype (see Fig. 6). The residents recognized that going to the bathroom in the dark was potentially difficult. They did not feel the need for lights however because each room had its own en-suite bathroom. Lighting switches were placed by each bed for convenience.
A presence light outside each room, indicating whether residents were in or not (see Fig. 7), did not arouse any interest. Our impression was that this feature would be of more interest to staff than to residents. In previous meetings the head of the care home had indeed proposed the idea, identifying the need to be aware of the movement and presence of people inside the house as an important issue (for example, to satisfy health and safety regulations). The open nature of the house made this requirement more important. The continual coming and going of people on the lower, more public, floors made it hard to identify who was in the building.
Having the two buttons to call for help in emergency and normal conditions was illustrated using the prototype. Reaction to this feature by the participants tended to be negative. The proposed solution was seen to be too confusing. Some residents were concerned that they would press the wrong button. Residents also pointed out that rooms already have a calling button, but not of the type (nor in the position-its place by the bed) presented in the prototype.
One interesting aspect here was evidence of a tension between the staff (in particular the head of the house) and the residents. It was desired that staff should be able to differentiate real distress calls from more trivial ones. This feature was not available in the current system and was identified as an issue during the earlier meetings. This was contrasted with the residents' desire both for a simple system as well as to be assured of their independence.
A further criticism of the presented solution related to the positioning of the buttons. One of the participants mentioned that in an emergency situation the button, positioned by the door, might not be easily reachable. However this conflicted with the more general view that residents were self sufficient and did not need help in their rooms.
To foster discussion, we adjusted the position of the buttons in the room, for example, moving buttons to the WC or closer to the bed (see Fig. 8). In this context, the person presenting the scenarios using the prototype also acted as an editor of the environment. Discussion within the group revolved around whether the button by the bed was enough, or whether it should be complemented by another, and where that one should be placed. From the discussion it emerged that, although residents were reluctant to admit it explicitly, providing assistance in the bedrooms was indeed a desirable service. While discussing the best positioning for the buttons, one resident explained in detail how she had fallen from the bed and had a very difficult time trying to climb back to reach for the calling button.
Following this productive discussion, the motion sensor to detect falls was then illustrated, using a smart phone connected to the prototype. The virtual environment, as presented by the desktop display, was augmented with the actual smart phone application and accelerometer. The scenario illustrated both how a sensor would be able to detect sudden movements and how the system would then notify carers through the smart phone application. This was considered by the group as a very useful possibility. Some residents, however, expressed concerns about how the sensor device would be worn. If, for instance, the feature was implemented through the smart phone worn on the clothes, then falling from bed at night would not be detected. The proposal for something to wear like a bracelet was well received. The possibility of having a panic button on the device was also discussed and generated positive, if not enthusiastic, feedback. There were also concerns, expressed mainly by the psychologist, about false positives and what type of movements would trigger the device.
Finally, the idea of the device serving as a localization device inside the house was explored. Residents were shown how staff members would be able to see their location on a map. Surprisingly (to us), none considered this feature to be an invasion of privacy. This might be attributable to how the residents saw this feature's applicability, see the discussion below. The general opinion, however, was that the device would not be very useful in the common areas of the house where typically there are other people present. Someone suggested, with general agreement, that this feature would be most useful outside the house. Residents felt that when they were out in the street they were most vulnerable. Residents agreed that the location service should indicate where residents were, whether outside the house or in their room.
Again, here, it was possible to identify a divergence between the administrative view, and the residents' view of what the system's requirements should be. The administrators were mostly interested in being able to discover quickly where residents were, both to contact them when necessary, and to monitor their well being. The residents, however, thought more about their personal safety and of feeling uncomfortable when left alone. For them the usefulness of the system was related to guaranteeing they would not get lost, and that they would be in contact with the people they knew.

Updated requirements (I)
The initial requirements were updated based on the focus group feedback. The new requirements represent a mix between the original requirements as discussed with the house administrators, and the views of the house's residents as identified during the focus group.
The guiding lights to indicate the path to the bathroom were removed. The requirement for a system to guide residents to the bathroom during the night related mostly to the residents needs. During the focus group this was found to be redundant as the residents felt the current room lighting provides enough help. Consequently this requirement was removed.
Triggering automatic calls for assistance as a result of a fall using a bracelet was a preferred option. In addition a panic button on the bracelet to call for assistance was also introduced as a new requirement. This replaced the two buttons solution initially conceived. The original button was left for non-urgent calls.
The requirement to know the whereabouts of care home residents, and whether they were in their rooms, was maintained. However a new requirement for locating tenants outside the care home was introduced. The original requirement was relevant to the administrative staff in the house but not seen as relevant by the residents.
APEX enabled rapid updates in the light of feedback and new requirements as they were established. When prototyping the outside location feature, two alternatives were considered. The first was to track actual physical devices connected to the virtual environment. This would imply that subjects would have to be outside in the city streets to test the system. The second alternative was to recreate part of the city where the house is located and simulate the location feature. The prototype was to be used in a focus group setting, and therefore requiring participants to be outside in Fig. 8 Discussing the buttons' location the city is not feasible. A further iteration of the participatory design, still to be completed, using the updated prototype will include not only one floor of the care home but also part of the city. This step could not be completed interactively during the meeting.

Related work
A number of approaches to participatory design have been previously developed to assist the exploration of ubiquitous environments. Sanders et al. (2010) have developed a framework to help the organization of these practices into form, purpose and application. Examples of practices considered include stories and storyboarding, diaries, game boards, 2-D collages, 2-D mapping, 3-D mock-ups using foam, clay, lego™ and velcro™-modeling.

Video
Video has been used as a prompt in participatory design (Lindsay et al. 2012), showing scenarios in which the envisaged technology can be used. Researchers have developed facilities for editing documentary film so that participants can understand and respond to possible design proposals (Hook et al. 2011;Raijmakers et al. 2006).
While video might provide a more realistic rendering of the environment, one problem with this approach is a lack of flexibility to support quick reaction to the users' attitudes towards the prototype and input. Using video only it would not be possible to adjust movement in the house in response to specific resident's reactions, or experiment with different locations for the buttons.

Theater
Live theater has been used in participatory design (Newell et al. 2006). Drama has been used to give texture to scenarios in which a proposed design is intended to be used. An interactive scenario method, including improvisation, and the engagement of participants as actors in scenarios (Strömberg et al. 2004), has also been used. In our case the scenery was important and varied significantly as users moved around the house or outside the house. APEX provided this flexibility to move around the space of the house and even change some of its features.
However this would be an interesting approach if a virtual environment such as the one we describe could be used as a backdrop to enacted scenarios. In this particular case, while one of the demonstrators enacted some scenarios through the avatar, and by manipulating the mobile phone, no other participants were directly involved as actors. However, the combination of virtual reality prototypes with the enactment of specific scenarios by users raises an interesting prospect to be explored in the future.

Paper prototyping
Paper prototyping has been used as a Rapid Participatory Design Technique (Osman et al. 2009) that enables speedy redrafting and change of design ideas. This approach is less immersive than the other types of media already mentioned but it provides a mechanism for sketching alternatives rapidly.
Our main concern with paper prototyping arose from the lack of computer literacy of the residents. Our impression is that paper prototyping is too low fidelity to allow stakeholders to imagine what it would be like to be in the ubicomp environment.

Laboratory-based high fidelity prototype
Part of a proposed environment (in this case an ambient kitchen) has been used to provide a physical context in which design ideas relating to the kitchen, and more broadly to other aspects of the environment under design, can be considered (Olivier et al. 2009).
The Aware Home (Kientz et al. 2008) at Georgia Institute of Technology (GaTech) contains two identical floors with nine rooms, each designed to explore emerging technologies and services in the home. The Aware Home team is also exploring the use of a suite in a senior living residence. Their concern is to overcome mobility limitations relating to older adults who might be unable to travel to do their usual tasks.
The issue here is the cost of these approaches, something we address through the use of virtual reality representations of the actual environments.

In situ high fidelity prototypes
Building and deploying an initial version of the system could in principle be a feasible option, using recent embedded technologies such as Arduino, but would be too disruptive to the house's operation, and to the residents' own daily routine. It would also mean that exploration of the prototyped system would imply moving about in the house. While this might at first seem to be a better approach, the logistics, and potential for disruption, of such an approach made it less attractive.

A dolls' house
A dolls' house has been used as the physical context for considering design issues in an AAL (Urnes et al. 2002;Kanis et al. 2011). While the house is a rigid design it provides a graphic reminder of the context as design discussion is conducted.
It is possible to think of the APEX developed prototype as a virtual reality based version of a dolls' house, with the advantage of being more dynamic since elements in the house can exhibit behavior. For example, presence lights turn on when the resident's avatar moves into the room.

Designing AAL services
Another perspective that also uses participatory design is to see the design problem as a service design problem. See, for example, work by Menschner et al. (2011) and by Pantsar-Syväniemi et al. (2014).
The interesting feature of this type of work, from the point of view of APEX, is that the framework encourages a view of ubiquitous environments as delivering implicitly a set of services. These services can be characterized in APEX using CPN. The CPN provides a specification that an off-the-shelf service should satisfy or provides a precise characterisation of how an existing service should be modified.

Virtual and mixed reality
The advantage of using virtual environments in participatory design is the flexibility that it affords. Video, theater and physical dolls' houses all provide barriers to flexibility. However a possible disadvantage of VR is the validity of the feedback obtained when compared with these other approaches. Work exploring the use of VR to assess user experience has indicated that VR is indeed a viable alternative that enables appropriate user experience, see Rebelo et al. (2012), when framed using appropriate methodologies to compensate for lack of texture.
Others have explored the use of virtual reality in participatory design. Davies (2004) describes the adaptation of a VR-based tool for this purpose. He concludes that while the tool could be successfully used by small groups, for larger groups the task of editing the prototypes should be assigned to experts. He concludes such kind of tool should be seen as part of the toolset used by design experts. Mobach (2008) explored the topic in the context of architectural and organizational space design. He analyzed the cost-benefit of using VR, concluding that the approach is a valuable and affordable alternative to co-create better spaces. Note that these conclusions were obtained with an approach that was less flexible than APEX. For example, the approach did not support the connection of physical devices of the programming of 3D objects. The benefits of VR simulations for architectural design have been further explored by Koutsabasis et al. (2012). Hong et al. (2016) focussed on the use of avatars to support remote collaboration. APEX supports this style of collaboration, although this was not the focus in the present context.
Using mixed reality in participatory design has the advantage that it improves user immersion, enabling participants to interact both with physical and digital objects in real time, potentially enhancing attention span. Bruno and Muzzupappa (2010) use VR with physical devices to involve participants in product interface design rather than ubiquitous environment design. They have demonstrated the efficacy of focus groups in the analysis of virtual products and demonstrated how users can be codesigners using VR prototyping. These results accord with Reich et al. (1996) who claim that ideal participation involves customers as co-designers. However, some limitations were identified: (a) observing users outside of their daily context may lead to a variation of the modes of interaction with the product; (b) haptic devices cannot be used when interacting with a virtual environment.
The APEX framework makes it possible to adapt configurations and designs in real time and to explore mixed reality environments using the (physical) smart elements that are part of the design as they become available. All these elements can be changed and re-presented relatively quickly. We know of no other work, using such a multilayered approach, that includes the combination of virtual and physical devices in participatory design for AAL systems. While there are other prototyping tools that are based on the development of 3D virtual reality environments (O'Neill et al. 2009;Nazari Shirehjini and Klar 2005;Irawati et al. 2008;Papka and Stevens 1996), none of them focus on the experience the users will have of the design. See Silva et al. (2014) for a detailed comparison.
The use of a mixed reality approach was, however, hindered by the fact that interaction with the virtual world is currently restricted to the use of viewers. This limited the possibilities of exploring the interplay between the physical and the virtual elements. This will be further discussed next.

Discussion
Our discussion will be concerned with two issues. First the validity of the work reported. The more general issue of the validity of using virtual and mixed reality in the design of AAL systems has already been addressed in the previous section, by considering possible alternatives to the use of APEX. Here we will discuss the validity of the particular case put forward in the paper. Second, we will discuss lessons learnt.

Threats to validity
The fact that participants did not use the prototype directly limits the sense of immersion. Discussion by participants revolved around what the house could become, and not about being in the new house. Even so, relevant insights were gained from the exercise.
The fact that participants did not interact with the prototype in person might also raise questions as to whether this type of prototype would be effective when used directly by participants. Issues relating to type of engagement and to how to collect data and feedback are important to understanding the value of this participatory design. Previous experience using the framework directly has shown that engagement is easy to achieve Gomes et al. 2014). This experience ranges from prototyping existing or envisaged ubicomp systems (e.g. a library or a bar at a theatre), to a serious game aimed a primary school children. Deployment has been mostly desktop/laptop based, but the use of a CAVE environment has also been tested successfully.
The environment can also be instrumented to collect information about the behavior of users . In situations where the number of test subjects is high, making direct observation impractical, questionnaires designed to be used after participation have been used. Video recording has also been considered, but so far not used without the appropriate ethical clearances.
A further point to consider is that only the managers and the psychologist were consulted and not other staff in the home. This will inevitably have biased the initial requirements. This omission was a result of the home's internal policy. It did not however hinder our goal of studying the applicability of the approach, and does not invalidate the conclusion that the approach was indeed useful. This is illustrated by the fact that it was possible to redefine some of the initial requirements through interaction with the prototype.
Presenting residents with AAL technologies, using the adopted format, risked appearing patronizing, thereby generating negative responses that did not fairly reflect the value of the technology. Our approach was not to present solutions to their problems, but rather to ask for their advice and opinion on the envisaged technologies. By empowering the group in this way the risk of offending or patronizing its members was diminished.

Lessons learnt
Our experience shows that the use of virtual environments can be a viable and low cost option for the participatory design of AAL systems. The development of the prototype took 3 person-weeks. No additional costs were incurred as all tools where free to use, and the laptop and mobile phone used for the study were already available to the project. We do not include here the costs of travelling to the site and meeting with stakeholders as these would have been incurred whatever the approach taken to design the system. While we do not have an estimate for the cost of creating a physical prototype, such costs would be clearly greater. The cost benefit, however, is only one of the advantages of using VR in participatory design. Others include such aspects as increased creativity, user commitment and satisfaction (Mobach 2008).
Nevertheless, as stated in the previous section, being restricted to the use of viewers to interact with the virtual world placed some restrictions on the case study, both in terms of how the prototype could be presented to users and what type of elements could be moved from the virtual to the physical domains. In practice, this was only feasible for wearable devices as the users were physically located in the room where the focus group was happening, even if virtually they could be somewhere else. A scenario that is relevant here is where a virtual fall detection sensor is activated in a (virtual) bedroom and the system wants to notify a carer by announcing the event using a public screen somewhere. The options that were available were to simulate the screen on the virtual world, which meant the carer would have to be physically present in front of the screen, or it would involve installing a physical screen in the house. We have found that this need to be physically present in front of the screen makes it harder to simulate ecologically valid situations. While for the residents it was reasonably valid to interact with the virtual world to enact some scenario, for the carer it meant moving away from other duties and focusing attention to a situation that was supposed to be unexpected. The above limitation also poses constraints when considering the final system design, as we would like to explore problems related to the users moving around in the city as, for example, GPS signal quality, which are only possible in the physical domain.

A roadmap
This section sets out a roadmap for the development of a mixed reality system in support of participatory design. The roadmap is defined in relation to APEX but can as well be used when considering other approaches. The vision is one where immersive prototypes capture a combination of existing applications and communication systems, with simulated technology not yet developed. These prototypes will evolve from the virtual world simulations, currently possible in frameworks such as APEX, to more physically grounded prototypes explored in-situ through Augmented Reality (AR). This might be achieved, for example, using augmented reality glasses.
By providing a mixed prototyping approach involving physical and virtual elements the platform will provide more ecologically valid scenarios and thus better support the identification of potential user problems in early phases of development. Ultimately the evolutionary nature of the prototypes will enable the agile development of ubicomp environments.
Supporting such prototypes requires extending APEX in order to: --Create AR prototypes-we will explore how to bridge from the worlds in the virtual reality prototypes to the augmentations of the actual physical spaces represented in those worlds. Hence, a public screen displayed in the VR simulation of the common room will be rendered in the appropriate position in the actual room through appropriate hardware. We envisage exploring different levels of immersion from AR glasses down to a tablet pointed at the wall. --Interconnecting the two types of prototypes so that the a mixed scenario is possible where the VR prototypes are augmented with information from the physical sensors and systems available, and the AR prototype is able to reflect the events happening in the VR prototype.
Such an extended platform must address two aspects: 1. Augmented Reality (AR), where physical inhabitants will sense the presence and interactions of distributed avatars populating the same space; 2. Augmented Virtuality (AV) where the virtual space is populated with: (a) avatars representing both physical users present in the physical space and distributed users connected virtually (b) virtual elements representing virtual and physical devices/sensors that will respond both to virtual and physical interactions.
An additional important aspect for a successful use of the platform is related with the user immersion provided. All users and developers remotely connected or physically present should be able to interact with the platform and develop an impression of what it would be like to be in the ubicomp environment beside of the distribution of the elements. Adequate interaction techniques within the platform will be developed to improve user immersion. This interconnected platform will provide a richer user experience of ubicomp environments, even when the element that compose the space are distributed.

Conclusions
Participatory design should enable participants as codesigners to explore a system. As a result, a design solution will be produced that will enhance their experience of the system and avoid usability pitfalls. Prototypes enable the exploration of the system from the early phases of design.
A participatory design process for ubiquitous systems has been presented which makes use of an immersive prototype. Speedy iteration of the design was made possible by using APEX, a framework for prototyping ubiquitous computing environments. APEX also enables the development of virtual environments in combination with physical devices. The process was demonstrated by considering an AAL system.
The design process, which encompassed the development of a mixed reality environment, enabled older participants to engage with the design concepts and to provide constructive feedback about design proposals. APEX enabled the rapid creation and iteration of the design based on the users' feedback. The process also made it possible to better evaluate and establish requirements for the technology enhanced environment.
Lessons learnt relate to both the design process and the design itself. In terms of process, it became clear during the participatory design that virtual environments provided good support for evaluation at a very reduced cost. This supports claims found elsewhere in the literature, e.g. Mobach (2008) and Rebelo et al. (2012). The prototypes, that had been developed using APEX, enabled developers to ground discussion about the proposed design as well as to explore alternative designs in real-time. Although developing specific behaviors could take a little longer, using off the shelf components was almost immediate.
The way the virtual environment was presented (using a large screen) helped the residents to understand the scenarios and the proposed design solutions for the AAL system and enabled them to visualise what the environment would be like. The approach worked effectively with this older target population.
In what concerns the particular systems being designed, when exploring the design differences between the perceptions and opinions of the different stakeholders (director versus occupants) were identified. These related to the perceived utility and desirability of the features of the design, as well as to their interpretation by the stakeholders. As a result of the exercise we were able to produce an updated set of requirements to better reflect the interests of all involved. possible. We also thank the reviewers for their helpful comments and suggestions on previous versions of this paper.