Featured Work

Interactive Holiday Installations

Primal Screen was hired for a massive project - produce all the graphics, projection mapped content for a huge holiday themed experience. As Technical Director and head of the engineering team, I handled development of the three interactive experiences that were included in the project.

Hardware

All three were based around the Realsense D435 depth sensing camera, and developing the image processing pipelines around that - hardware we had not used before, but was critical to all of the experiences.

Unity Software

The software side involved writing compute shaders, building calibration systems, writing software for segmentation, trajectory estimation, compute shaders for image processing, 3D translations, and having everything run reliably, every day.

I chose Unity for a balance of graphics capabilities and SDK integration with realsense. Artists could produce work in familiar software for them and ensure it looked great on screen, and I had the tools needed to build the technical side.

There were three main installations, all of which involved projection mapped screens and realsense cameras taking in environment data. These were broken into the Aurora, Slide, and Snowball experiences.

Aurora Wall

The aurora utilized Unity's VFX graph to mirror guests onto a wall as a collection of bright swirling particles. It also responded to gestures, taking vertical movements and emitting bursts of particles into the sky in response.

This required a few technical solves. First was calibration. For depth cameras mounted over 12 feet high, I had to appropriately detect people in front of the screen while masking out the ground. Since this was an experience for families, cameras had to reliably detect kids who were close to the ground, without detecting the ground itself, and reject high amounts of environmental noise that came from being in an otherwise dark environment at the extreme ranges of the camera.

Compute shaders and OpenCV became critical here for blob detection and filtering, along with an in-software calibration mode which would translate the point clouds from the high angle of the camera mounting point, to an accurate height and shape in front of guests. The camera processing also accounted for motion, so upward motion between frames above a certain threshold could trigger the upward particle trails. Unity's VFX graph allowed for custom defined particle systems and shaders that handled millions of particles.

Interactive Slide

The slide was a physical slide, with projected graphics that tracked and reacted to guests as they slid down. The same challenges of processing noisy depth data were present, with extra difficulties due to the smooth reflective surface of the slide. In this case, blobs were processed and converted to game space, triggering both a trail to follow guests down the slide, and snowflakes to burst any time they slid past.

Snowball Wall

The snowball throwing game was a 30 foot stretch of projected winter landscape, where guests could throw physical plastic balls at the screen to demolish snowmen and holiday decorations.

The main challenge here was detection. While object tracking by color is a very simple problem, this game involved throwing white plastic balls at a white snowy landscape. Thus I quickly rejected color-based object tracking in favor of depth. This still had to account for flexible fabric screens and noisy depth data however, meaning I could not simply detect the exact point of impact.

Roughly, each cameras depth feed is segmented into high contrast blobs for anything within a given range. The depth data at the center of each blob is sampled to get a position, which is matched to any related, previous frame positions, to define a trajectory. This then lets us predict the final impact position and time, which is then translated to the game space to trigger hit effects. This allowed for accurate positions without having to detect the exact point of impact.

The development of the game itself went extremely smoothly. A couple of initial conversations with artists about asset delivery formats and the general gameplay loop were all that were necessary. In the words of the creative director, "We just game him a pile of assets, and then we had a full game."

Challenging Work

Testing and development space was limited early on, which presented its own challenge with lack of options for testing camera placements and testing at final scale, so many adjustments had to be made on-site in the short time leading up to opening night. There are many ways I would approach a project like this differently in the future, but in the end, the project delivered on time and the client was happy. As a project that involved many technical challenges and high pressure, it ended up being one of the most fun and memorable to work on.