By Jo Liberman
Virtual reality is useful because it allows us to create environments that would be difficult, expensive, or impossible to control in the real world. In physical environments, a wall is usually both visually present and physically solid. In VR, however, those two things can be separated. A wall can appear solid, transparent, passable, or inconsistent with the behavior of other objects around it. This makes VR especially useful for studying object affordance, or how people perceive what an environment allows them to do.
For this project, I wanted to test whether an open-source markerless motion-capture system could measure affordance-related behavior during a VR walking task. The concept of affordance comes from the field of psychology, where scholar J.J. Gibson theorized that when we perceive something like a door, we also perceive its usability to us (that it is something that we can open). Conversely, something like a solid wall would be perceived as an obstacle, and would therefore have less affordance or use to us.
Research Design in Virtual Reality
For my experiment, I created a VR environment where participants walked between two circles on the virtual floor. A wall was placed directly between the start and end points, and the wall changed across trials. Sometimes the wall was transparent. Sometimes it was solid. In some trials, objects fell through the wall, which suggested that the wall was less solid and had more bodily affordance. In other trials, objects stayed on top of the wall, which suggested that the wall was more solid and less passable.
The basic idea was that participants would move differently depending on how the wall appeared and how the surrounding objects behaved. I expected transparent walls to increase bodily affordance because participants would be more likely to treat them as something they could walk through. I also expected that when objects fell through the wall, participants would interpret the wall as less physically constraining. As a result, I expected less hesitation and higher approach speed in these conditions. In contrast, when the wall appeared solid, or when objects landed on top of it, I expected more hesitation, slower speed, and less direct movement.
Each participant completed eight trials. The instructions were given through an automated trial manager, which helped keep each trial consistent. Participants were told to walk from the start point to the endpoint and were also given a score based on how direct their path was. A perfect walk was a straight line from one point to the other after the start prompt appeared. This was included to encourage participants to pay attention to the task and the wall configuration rather than walking randomly through the environment.
Open-Source Motion Capture in VR
To record the movement, I used six Arducam Mono Global Shutter USB cameras placed around the room. The cameras were positioned in the four corners, with two additional cameras on the North and East sides of the room. They were angled so that at least two cameras could capture the participant fully at all times. All six camera views were recorded at the same time through OBS Studio, and the videos were later processed through the affordance algorithm.
The affordance algorithm was built in Python and was designed to work with open-source tools. It used markerless pose tracking, camera calibration, AprilTags, and triangulation to reconstruct the participant’s movement in 3D space. After the trajectory was reconstructed, the system analyzed hesitation, speed, curvature, forward lean, step width, and trajectory. These measures were then used to estimate whether the participant’s behavior suggested that the wall felt passable, uncertain, or obstructed.
One of the most important parts of the project is that the system was designed to measure affordance-related behavior directly. It was not just collecting movement data and then trying to figure out later what might be interesting. Instead, the pipeline was built around the specific behaviors that matter for affordance.
Results of Affordance-Related Behavioral Study
The results showed that the open-source markerless motion-capture pipeline was able to capture and quantify affordance-related behavior using hesitation and speed metrics. The system produced consistent outputs across trials. These outputs changed in structured ways across the different wall configurations, which suggests that the system was detecting meaningful differences in how participants responded to the virtual wall.
Broadly, the project shows that affordance can be studied through movement itself. If a participant slows down near the wall, pauses, curves their path, or changes their walking pattern, that behavior can reveal how they are interpreting the environment. Traditional motion-capture systems can be expensive and difficult to set up, which limits who can study these kinds of questions. An open-source markerless system creates a more accessible way to measure embodied behavior in VR. It also allows researchers to design motion-capture systems around the specific behavioral metrics they care about, instead of relying only on general movement tracking and doing most of the meaningful analysis afterward.
Overall, this project suggests that virtual reality and open-source motion capture can be combined to study affordance in a more direct and flexible way. VR makes it possible to control the relationship between visual appearance and physical interpretation, while the motion-capture pipeline makes it possible to measure how those interpretations appear in the body. The main point is that affordance is not just about what an object is, but about how it is perceived as available for action. In this sense, hesitation, speed, and trajectory become more than simple movement variables. They become evidence of how a participant understands the environment in real time.
References
Gibson, J. J. The Ecological Approach to Visual Perception (1979).
Chemero, A. “An Outline of a Theory of Affordances.” Ecological Psychology 15, no. 2 (2003).
Ishikawa, T., and D. R. Montello. “Spatial knowledge acquisition from direct experience in the environment.” Cognitive Psychology 52, no. 2 (2006): 93–129.
Baskaya, A., C. Wilson, and Y. Z. Özcan. “Wayfinding in an Unfamiliar Environment.” Environment and Behavior 36, no. 6 (2004): 839–867.
Dong, W. et al. “Wayfinding Behavior and Spatial Knowledge Acquisition: Are They the Same in Virtual Reality and in Real-World Environments?” Annals of the American Association of Geographers (2021).
Bhargava, A. et al. “Revisiting affordance perception in contemporary virtual reality” (2020).
Siegel, A.W.; White, S.H. The development of spatial representations of large scale environments. In Advances in Child Development and Behavior; Reese, W.H., Ed.; Academic Press: New York, NY, USA, 1975.
Lepecq, Jean-Claude & Bringoux, Lionel & Pergandi, Jean-Marie & Coyle, Thelma & Mestre, Daniel. (2009). Afforded Actions As a Behavioral Assessment of Physical Presence in Virtual Environments. Virtual Reality. 13. 141-151. 10.1007/s10055-009-0118-1.
Cmentowski, Sebastian & Krüger, Jens. (2021). Effects of Task Type and Wall Appearance on Collision Behavior in Virtual Environments. 1-8. 10.1109/CoG52621.2021.9619039.
Sohn, S, DeStefani, S, Schwartz, M, Feldman, J, Kapadia, M, & Stromswold, K. On the Separability of Social Navigational Behaviors in Virtual Reality.