By Jiayu Yang
Introducation
Can you imagine becoming a worm in a beehive, in a jar, or on a disk in a laboratory? How would a worm perceive man-made structures? Kafka’s The Metamorphosis (1915) follows the transformation of the protagonist, Gregor, into a giant bug, offering a metaphor to look back at the alienating and dehumanizing aspects of modern human society. Today, when insects are increasingly integrated into human society, what could be seen if viewed through the senses of a parasite or a worm in a lab? More than postulating metaphors, my project at the LCDSS aims to model artificial habitats from a speculative worm’s view in order to reveal the unstable captivity of nonhuman creatures.
To recap, I am a documentary filmmaker and digital media artist working with wax worms (Galleria Mellonella), which were first found in nature as parasites of beehives, and are now proteins for pets, model organisms for immunology, and sources of plastic-degrading enzymes. In an era of species endangerment, wax worms are a unique case of thriving insects that are making habitats across the urban landscape. In my previous blog post, I detailed the emergence of various artificial habitats for wax worms across apiculture, worm farms, and laboratories, and how I want to transport these habitats into VR environments through Photogrammetry and Unity. In this blog post, I will share details about my workflow and how the 3D models will be displayed and interacted with.
Becoming an Insect Through Visual Simulation
To simulate a worm’s perception, which is haptic and non-visual, I find the format of point clouds to be of great potential. Point clouds are collections of three-dimensional information, with each dot representing the x, y, and z coordinates on the perceivable surface of a physical space or an object. Primarily generated for computer processing, point clouds are used to tell photogrammetry programs where a surface is located; their density indicates how much information is gathered and how confident the computer should be in reconstructing that surface. They are an intermediary translation of the physical world into the digital.
Breaking solid space into contingent, porous clouds, point clouds reflect a nonhuman vision that alienates familiar physical spaces from the human eye. Today, point clouds have become a new aesthetic in visual art, with many visual artists working with them to simulate animal vision and represent the world in an alienated form. Examples that have inspired me to work with point clouds are Marshmellow Fest’s In the Eyes of the Animals (2015), Reality Design Lab’s Echovision (2024), and Refik Anadol’s Echos of the Earth: the Living Archive (2024).
Methods
To transform a physical space into metamorphosing particles, there are a few steps I take:
Step 1: Photogrammetry
Step 2: Import to Unity VFX Graph
In the following sections, I will walk through these steps in more detail.
Photogrammetry
Photogrammetry is a method of 3D scanning that interprets multiple photos of a physical space or an object and constructs 3D models based on the spatial information (x,y,z) in them. In my case, I used an iPhone camera, a Sony a7s2 camera, and an Insta360 camera to take the photos. Usually, I take around 70 to 150 photos from all angles, depending on the size and details of my object. With cameras that have higher resolution and higher shutter speed, such as the Sony full-frame mirrorless camera and the 360 camera, I often record videos first, then extract the frames from the recording as photos for photogrammetry. After all, videos are photos playing at 24 frames per second.
After the photos were ready, I used Agisoft Metashape to interpret the spatial information. Metashape would first “align photos” and then build a “tie point” (figure 2) out of them. At the stage of “tie point,” the 3D structure is sparse and lacks details. I then asked Metashape to build a “point cloud” by locating more points in the 3D space. With a point cloud, the model becomes more legible, while retaining a surreal look (figure 3). Then, simple edits such as “select” and “delete” can be done with the software. After some quick edits, I exported the point cloud as .ply files for creative use later.
Figure 2. The tie points of the beehives.
Figure 3. Point clouds created by the tie points.
Import to Unity VFX Graph
Using Unity as the primary platform for viewing and animating the point cloud, I have learned to import point clouds as Unity assets with a point cloud importer/renderer, since .ply is not natively supported in Unity. To import point clouds into Unity, I found the PCX package shared by Keijiro Takahashi (a renowned visual artist) on GitHub helpful. By installing this package in Unity, I was able to import the .ply file and render it as a position map and a color map. These maps are then used to tell VFX Graph how to position and color the particles.
Unity’s VFX graph is an updated particle system, allowing faster, more efficient rendering of large numbers of particles. It uses a node-based structure to program the particles. To create a VFX graph, I first installed it via the Unity package manager. Then it became an asset I could create and add to my scene.
After creating a preset looping VFX graph, I went into the “initial particle” block to add two more blocks: “set position from map” and “set color from map.” These blocks allow me to link the position and color maps to the VFX graph (figure 4). After that, I adjusted the spawn rate of the VFX graph to match the point count of my point cloud, so the particles would look exactly like the point clouds I generated.
Once translated into a VFX graph, the point clouds become clouds of particles that can be given lifetimes, sizes over time, velocity, or attraction. I can set them to flow with turbulence or slowly reconfigure into other shapes. The potential to manipulate and animate is infinite.
A Work in Progress
So far, I have built a scene of an infested artificial beehive in Unity. This is the first scene I built because these hives are where the worms first enter civilization as parasites. When set to morph with turbulence, the point clouds of the boxes of the hive become floating particles in this scene, as if they are bees or pollen (figure 5).
I am building more scenes and working on the design of haptic interactions to simulate elusive visions or unattainable memories. My goal for the near future is to translate viewers’ hand movements into forces that disrupt and reshape the point clouds of a habitat. This way, a viewer can see themselves dissipate the human narrative and reimagine these emerging organisms as something beyond the scientific captures.