Inverse Presence

Inverse Presence

People use virtual reality for a purpose - like any 

kind of media the purpose of experiencing it is

to effect some change. This change might be simply 

at the level of enjoying something (entertainment), or viewing something (for the sake of understanding, design or development), or to learn something new, or to have some new kind of experience that is not available or difficult to achieve in everyday reality.

In some sense there is always a task to be realised - whether it is "enjoy yourself"or something more concrete than that such as to realise a specific set of actions. 

Now for many years there has been research on factors that contribute to 'presence'. Here by 'presence' I mean that aspect where people tend to respond to situations and events in the virtual world as if they were real: an avatar smiles at you and you smile back, or an avatar comes close to you and you feel uncomfortable and possibly step backwards, since this breaks the norms of proxemics. 

Even if only implicitly most applications of VR rely on presence occurring. For example, using VR for psychotherapy would be useless unless the patients, to some extent, respond realistically to what is depicted in the VR. So VR therapy, for example, for fear of heights would not be useful if patients did not feel some anxiety around their experience of precipices in VR.

Lydia Reeves Timmins and Matthew Lombard used the term 'inverse presence' to describe situations where something happens in reality that is perceived as if it were not real - real events (especially horrific ones) may be experienced as 'simulated' - i.e., we have all experienced moments in which we think "this is not really happening". Here I want to give a different meaning to the term "inverse presence" - to mean that we assume that presence in a VR will happen, and therefore we exploit this to get participants to achieve some particular tasks that they had never explicitly been told to do. For example, suppose the task is "get this person to smile" - then elements of the virtual environment (such as avatars) must learn to carry out actions that evoke this response. Probably here it would be quite easy - since from 'presence theory' we know that if an avatar smiles at the participant they are very likely to smile back - and so introduction of smiling avatars would probably do the trick.

Most experiment studies on presence vary factors that are thought to contribute to presence, and then see when and if presence occurs. 'Inverse presence' means - we know that presence will occur and so certain behaviours are likely to follow from this, and let's utilise it to get people to do certain specified actions.

In the recently published paper (PDF) of Jason Kastanis and myself in ACM TAP we described a quite simple example of this approach. When you interact with a virtual human character in immersive VR you tend to respond realistically. In particular several other works have shown that the rules of 'proxemics' operate - that is, if the avatar approaches too closely to you, you step backwards (the implicit rules of social, personal and intimate space seem to apply in your interactions with avatars too). Our goal was for an avatar to learn how to get the participant to go to a particular place within the virtual environment - a place some metres behind where they were initially standing. The avatar was programmed with a number of actions it could take - like move forward or back, do nothing or wave to the participant saying "come here". At first the avatar chose these actions at random, but over time it converged on the right behaviours - get the person close to the avatar and then move forward towards the person so that the person backed away. The avatar was controlled by a Reinforcement Learning agent, that received a reward when the person moved towards the target, and a loss when the person didn't do that. The RL algorithm is designed to maximise long term reward. What we found is that in circumstances when the avatar was allowed to move to intimate distance to the person, it learned how to drive them to the pre-specified place within 7 minutes. It took much longer if they could only move to personal distance, and didn't work at all if it just selected random actions (move forward or back or wave).

The purpose of VR is to get people to 'do' things (doing includes experiencing). Here we let the VR system learn how to get the person to do things by relying on their likely responses to events, as predicted by presence theory. The RL worked efficiently, but of course this was a very simple 1D problem. Nevertheless I think that the paradigm is worth pursuing with more complex scenarios.

The Presence of Shy Men

We recently had a paper published in PLoS ONE:

Socially Anxious and Confident Men Interact with a Forward Virtual Woman: An Experimental Study

Xueni Pan, Marco Gillies, Chris Barker, David M. Clark,Mel Slater

A man goes alone to a party. There are not many people there, a few sitting around, and one lone woman who starts looking at him. Very soon, the lone woman walks towards him and starts a conversation. At first the conversation is mundane and she maintains a normal social distance from the man. But eventually she tells him that she thinks he looks very nice, moves much closer to him breaking normal social boundaries, and then asks him whether he is involved with anyone at this time.

How does the man react? Well for the experiment we recruited men who were either were quite socially confident in their relationships with women, or quite socially anxious. Our expectation was that those who were socially confident would simply enjoy the encounter, whereas the anxiety level of those who were socially anxious would go through the roof.

Oh - don't forget that all this happened in virtual reality, in the Cave system at UCL, London.

So what happened in fact? On the whole the men in both groups were more anxious than previously, when the woman first approached. Then for the confident group the anxiety levels returned to what they had been beforehand, but, and to our surprise, the anxiety levels of the anxious group actually seemed to go even lower than they had been beforehand.

I believe that what happened is that men who are socially anxious with women tend to avoid such encounters (because it makes them anxious). In particular they do not initiate encounters, and avoid situations where they might be required to approach the opposite sex - such as dances, parties etc (except perhaps unless they are fortified by alcohol). But here there was a 'woman' who was doing all the work, making all the small talk, and also interspersing a more mundane conversation (about living in London etc) with the more challenging topic of relationships. This allowed the anxious men to relax more than usual, certainly more than they would have expected than when the woman first approached.

I think that this points an interesting way forward in therapy for such social phobic conditions. In this case the virtual woman initiates the conversation, and reduces the 'threat' level, by keeping it at a mundane level for a while. Then more problematic issues can be raised. This interspersing of mundane and more difficult topics in the conversation, could, I think, be a key to therapy in this area.

We carried out this study quite some time ago, but only recently wrote and submitted the paper. Now we can do avatars and scenarios of much higher quality than we used then. Moreover, we now routinely animate virtual characters through motion capture, so overall the experience for any participant has a much greater level of realism. But the interesting point is that in spite of the low level of realism in this study, the men still tended to respond as if it were real. I'm not sure how much additional realism would actually change things.

Finally, social phobia is a hidden illness, with very severe consequences for the sufferers. Many years ago we were doing a case study with Prof. David M. Clark and the patient had a particular form of social phobia - fear of public speaking. This had affected his career - choosing a career where he could avoid speaking in public was a critical consideration for him. He told us he was worried about making a speech at his daughter's wedding. His daughter was, at the time, 3 years old.

Even Though You Know it is an Illusion...

Today was Andrea Brogni's PhD exam at Universitat Politècnica de Catalunya. He started his PhD with me some time in 2006 or 2007. His thesis is entitled Haptic Illusion in Virtual Environments. The idea is that instead of using a haptic interface to make the experience in a virtual environment a physical one, we use the brain. First he showed that if you let people 'touch' objects in VE without using a haptic interface, then they will experience sharp objects as sharp and smooth objects as smooth. Some evidence suggested that people have a physiological response to sharp objects that they do not have to smooth objects. Moreover, the experimental design ruled out the possibility that this was purely suggested by vision. Next he showed that it is possible to give the illusion of physicality when moving objects by forcing people to use their muscles. In other words if you try to move a 'heavy' object, nothing happens unless you exert sufficient muscular effort (as measured by EMG). This is an idea I've had floating around since the early 1990s, and indeed Martin Usoh and I wrote an EPSRC grant to further this, but it was rejected. I'm happy to see that finally it was realized and investigated by Andrea and done very well. We will carry on with this in the future. As he says though, it is not an attempt to 'replace' haptic interfaces, but an alternative method to experience physicality in virtual environments.

Actually why I'm writing today is because Andrea said something that reminded me that there is an issue I've had on my mind to write about for some time. When I talk about 'presence' (place illusion or plausibility) I always put a rider 'in spite of knowing that this is an illusion'. I.e., place illusion is 'the illusion or sensation of being in the virtual place in spite of the sure knowledge that you are not there' or plausibility is 'the illusion or sensation that what is happening is real, even though you know that there is nothing real happening'. Why 'even though you know that it is an illusion'? This was pointed out to me by someone as being redundant, or even wrong. But I had the feeling that it was a necessary part of the definition. This is why ….

It is because the feeling that is associated with the knowledge that the virtual experience is an illusion is part of the overall sensation associated with the illusion (e.g. of PI or Psi). Let's imagine that someone did not have knowledge that it was an illusion. Then for them there is no question of 'place illusion' since as far as they are concerned that are in that place, just as they would be in any place in everyday life. Hence there is no feeling associated with an illusion, since subjectively there is no illusion. Now it is quite a different issue to engineer a situation where participants do not know that it is an illusion. This is equivalent to subjectively 'reproducing reality' for that person. From a scientific and engineering point of view this is a more profound objective, and currently unachievable. Even in the famed 'holodeck' of Startrek people know that what they are experiencing is an illusion, unless it were possible to engineer in them the 'forgetting' of the fact that they entered from the real world into the holodeck, and the suppression of their knowledge about how the holodeck works. So in a sense the more modest goal is to get people acting in a VE in a realistic manner *even though they know that it isn´t real* which is different from the goal of getting them to falsely 'know' that it is real (and therefore the question of trying to get them to act realistically wouldn't arise, since by definition if they didn't know it wasn't real they would act realistically). At one level this seems like 'hair splitting' and only philosophical, but I think it actually reflects different engineering goals, as well as scientific ones. An analogy would be that to remove that part of the definition is equivalent to saying that we could make virtual reality operate like a dream. While the dream is happening you do not know that it isn't real - it is only afterwards when you wake up that you realise this. I think that at the moment this objective isn't realisable - except perhaps by the use of drugs, hypnotism, or neural implants.

Presence at SIGGRAPH 2010

One of the problems with ‘presence’ has always been that of measurement – since ‘place illusion’ (the sensation of being in the place depicted by the VE displays) and ‘plausibility’ (the sensation that what is happening is really happening) are both qualia, they are feelings that cannot be directly measured.

Some years ago I put forward an analogy with colour science - in the now defunct online journal Presence Connect – see [1].

For example, the sensation of seeing the colour ‘red’ is also a qualia, much beloved of scientists and philosophers who study consciousness. In colour science there is a physical function, the wavelength distribution, that describes the energy distribution of light emitted or reflected by a surface patch. Yet what we see is not simply some simple function of this physical energy distribution but a complex, and not completely understood interplay between the physics of colour and our perceptual systems. Moving on from there to the sensation of ‘red’ and our consciousness of seeing red is another far leap into the science of consciousness.

With respect to ‘presence’ the physical basis is the type of immersive system used, its properties and capabilities. This physical basis then becomes transformed into our perception and action within an alternate reality. Again how this transformation occurs is a problem for significant study.

There is a successful quantitative and predictive theory of colour science, that shows how an ‘average observer’ is likely to respond to patches that emit light with specific energy distributions. This success is partly built on the psychophysics of colour matching experiments. We applied an analogy of this idea to ‘presence’ (Place Illusion and Plausibility) and carried out an experiment to show how this idea could work. This has just been published at SIGGRAPH 2010 [2].

This shows how to create functions that predict how the ‘average participant’ would respond to particular system configurations in terms of presence. The paper provides a methodological counterpart to the earlier theoretical paper [3].

There is an associated youtube video, and a version of the presented slides is given below.

1. Slater, M., A note on presence terminology. Presence connect, 2003. 3(3).

2. Slater, M., B. Spanlang, and D. Corominas, Simulating Virtual Environments within Virtual Environments as the Basis for a Psychophysics of Presence. ACM Transactions on Graphics (SIGGRAPH) (TOG), 2010. 29(3): p. Paper: 92.

3. Slater, M., Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments. Philos Trans R Soc Lond, 2009. 364(1535 ): p. 3549-3557.

Presence in a Another Body

We recently published a paper on how we used virtual reality to give people the illusion that their body was temporarily that of another (First Person Experience of Body Transfer in Virtual Reality). In particular we were able to show that to some extent some men can be given the (temporary!) illusion that their body is that of a girl.

The image shows an overview of the scenario. The experimental participants were located in or near the seated girl's body, via wearing a head-tracked wide field-of-view head-mounted display. In one condition of the experiment, when they looked down towards their body they would see the girl's body instead, as if their eyes were located in the same position as hers. If they turned to look to their left, they would see the body of the girl in the virtual mirror, with its head movements matching their own. When the standing woman stroked the arm of the seated girl, they would feel this on their own arm.

This paper seems to have generated a lot of interest judging from press reports and the PLoS ONE metrics.

Some of the comments about the paper are quite funny, and it seems to have awakened kinky fantasies in some men. Other comments are off the mark believing (spurred on by some journalistic takes on the article) that we were transforming men into women. Indeed one angry comment on the paper notes that to give men the experience of being female we would have to also simulate all of the social conditions that women are subject to, and also physiological processes such as menstruation, etc..

I agree very much with this. We were not giving men the experience of what it is like to be a woman! We were generating an illusion that is very hard to describe or understand unless you have experienced it. It is an illusion that the different body that you see when you look down at ‘yourself’ in virtual reality, or when you see ‘yourself’ in a virtual mirror, is somehow your body (even though you know it isn’t).

The brain seems to be quite liberal in deciding what is part of your body, and such illusions have shown that it is not difficult at all to trick the brain into believing that something is part of your body when it is not. It is important to realise that this ‘trick’ does not happen at the cognitive level, that is you never believe that the fake body or body part is really part of you. Rather it is at some lower perceptual and proprioceptive level that you don’t have much conscious control over that this happens. So it is like ‘presence’ in virtual reality - you know for sure that this is not your body, but nevertheless it feels like it is.

These illusions stem from an initial insight by Botvinick and Cohen known as the ‘rubber hand illusion’, and see also the New Scientist video with Olaf Blanke. Here people experience the illusion that a rubber hand is their hand. However, no one would then go on to say that people who experience the rubber hand illusion know what it is like to have a rubber hand! Similarly we do not claim that a male who has the illusion that his body looks like a female one knows what it is like to have female body. These are different things at quite different logical levels.

I hope you find the article interesting, and on the PLoS ONE web page you can see a video which gives some idea of how it looks to the experimental participant. However, the only way to really know how the illusion feels would be to experience it. Welcome to Barcelona.

The Illusion of Violence

If you see some people fighting in the street, how do you respond? Some people may intervene and try to stop the fight, others may try to get help, others may just watch, join in one side or the other, or do nothing but get away as quickly as possible. How people respond to violence is an important topic in social psychology, and the issue goes back to a case that happened in the US in 1964, the violent murder of a young girl while apparently 38 bystanders stood by and did nothing. There is controversy over what actually happened, about whether there really were these bystanders, but anyway a whole area of research was opened up in social psychology to try to understand this ‘bystander effect’. One of the problems in studying this is that, of course, it is impossible to carry out experimental studies of how people respond to violence that they might encounter by chance in a public place. The social psychologist Dr Mark Levine of the University of Lancaster has come up with some ingenious ways to try to study this problem, and actually his view of how crowds behave is not quite so negative as the popular view that crowds are bad and that they encourage or don’t prevent violence. A few years ago Mark and I talked about the possibility of using virtual reality to study this issue, since in VR one can set up apparently contingent violent confrontations between virtual people and then study how real people respond to this.

We know from research into presence that people do, in VR, tend to respond as if situations and events were real. (On this topic, my paper published in the Proceedings of the Royal Society, mentioned in earlier posts, is freely available online until the end of February).

So if we place people in a virtual reality where there are virtual characters who start arguing and fighting with one another, then this might prove an interesting way forward for the experimental study of the topic of the bystander effect.

Actually one reason why we did the virtual reprise of the Stanley Milgram obedience experiment was that we needed to convince sceptical EPSRC project reviewers that it is the case that people do tend to respond realistically. (On this point too another paper has been published recently using brain imaging to try to understand what happens when people do experience that Milgram paradigm). The EPSRC were convinced, and Mark Levine, Prof. Jian J Zhang and I obtained project funding, with this as the major application.

We carried out a first pilot study on the issue of people's responses to violence in virtual reality which has now been published. Actually this did not portray violence but aconfrontation between two football fans that would eventually lead up to violence. The results were very encouraging, and we are now carrying out a full study.

Other News

On the topic of ‘response as if real’ there are some other recently published papers from our group that might be of interest. The first examines whether illumination realism makes any difference when people experience standing over a virtual pit. Does it help to have dynamic shadows and reflections? A pre-publication version is available online.

The second shows that when people walk across a narrow beam in virtual reality they tense their back muscles to avoid falling. This is in press with IEEE Transactions on Visualization and Computer Graphics, and is available as a pre-publication version.

The third shows that how people scan their eyes over a scene in virtual reality follows a pattern that is similar to what might occur in reality.

The fourth shows that people exhibit behaviour consistent with proxemics theory when virtual characters break into their virtual space, which is in press with ACM Transactions on Applied perception

Research Posts Available

I have several jobs available in the group in Barcelona. Take a look at www.event-lab.org over the next few days and you will see the adverts appear.

Illusion is Part of the Definition

In July there was a conference in Benasque, Spain, Art and Science: Exploring the Limits of Human Perception. We (Mavi Sanchez-Vives and I) teamed up with Nonny de la Peña and Peggy Weil from California, to create a piece based on the idea of immersive journalism. You can read about what it is like to be a prisoner at Guantánamo, but can you get first hand knowledge of what the experience is really like?

We created a scenario in immersive virtual reality, relying on research on body ownership, to give people the illusion that they were in a cell standing in a stress position, while hearing an interrogation going on in the cell next door. There is an article about it in El Pais: Presos de un Guantánamo virtual, and a youtube video: Towards Immersive Journalism: The IPSRESS Experience.

My new paper attempting to give my current understanding of 'presence' has been formally accepted for the Philosophical Transactions of the Royal Society, Series B. This is about deconstructing the concept of presence into two orthogonal concepts: 'place illusion' - the original idea of having a strong illusion of being in the virtual place, and 'plausibility', the illusion that what is happening is really happening. Of course these are illusions, no one really believes that they are in the virtual place or that what is happening is real - the fact that these are illusions is part of the very definition. Also the paper discusses what I think is a good way to understand 'immersion'. Altogether there are four concepts: Place Illusion (PI, based on sensorimotor contingencies), Plausibility (Psi, based ultimately on various correlations between actions and events), immersion (forming a simulation hierarchy) and the virtual body which is the intersection between PI and Psi. I think that it is relatively clear to me now, ideas that I've been trying to find for many years. 'Presence' in the sense of place illusion is not and never really was the main problem - it can be relatively easily achieved most of the time with the right display and especially tracking systems. The real problem is Plausibility. The pre-publication version of the paper can be downloaded from my publication website.