Deep in VR – Pistol Whip fan trailer capturing tests

I’ve for long been curious about VR games and how one would go about capturing trailers for them. A week ago I finally took the plunge and got an original HTC Vive set to do some testing myself.

It seems to be a consensus that first person VR gameplay with bobbing camera and a controller flashing into view every now and then, doesn’t do a great job delivering the experience to the viewer. 3rd person gameplay where you can see the whole player reacting to the surroundings in the game, does it much better, whether it’s a 3d character following the movement of the player’s body, or an actual player in mixed reality.

Since I don’t have a studio with green screens and expensive video equipment (nor any idea how to use them), I decided to try out what I could do with a pad camera and a rigged character model pictured in 3rd person. A mixed reality capturing software named Liv came up as the most on point solution for this. It allows you to shoot a player with a webcam attached to a motion controller, but also using a gamepad as a camera.

Although mixed reality trailers can be awesome, I believe that depending on the game, displaying a 3d character model as the player can have its upsides as well. Having a real person in front of a game landscape presents a clash of visual styles that may not work for the immersion. A camera freely movable in the virtual space also provides more freedom, having only watch out for clipping with surrounding 3d models, as opposed to studio surroundings where the size of the room and placement of green screen etc. pose some restrictions.

Trying out camera controls of Liv

I chose Pistol Whip as the target of this test because it supports Liv and seemed fun to play, which it was too. It certainly makes you feel like an expert gunman, its generous auto-aim allowing you to hit enemies in a fast sequence to the beat of the music, even though I had to keep it silent most of the time to get a clean capture of the audio effects.

In Liv, you can choose between different types of cameras with adjustable FOV, offsets and smoothing of movement in case of first person camera. You can also use an “avatar”, a specially prepared 3d character model, in case you want to show show a character model in place of a live player. With its 3rd person cameras, Liv captures the gameplay as a separate foreground and background layer, and automatically composites webcam footage of the player, or the avatar, between the layers. As a smallish tradeoff, anti-aliasing doesn’t work on any of the camera types.

After my first sessions playing Pistol Whip, I launched it with Liv to try out its pad camera, and see what the gameplay looks like from different angles.

Skeleton

Even though the foot IK and pelvis position of the avatar act surprisingly well based on the tracked movement of just the helmet and two controllers, there’s still the shortcoming of having to shoot the character only from the waist up, because the world is coming towards the player and the player character should be walking forward at an even speed.

To solve the issue with the legs, I decided to try using a “skeleton” model to track my movement, and then animate the actual 3d character in blender on top of the skeleton, using the gameplay capture as reference on the background. Then I could remove the skeleton from the background footage in After Effects, and composite the 3d figure on top with a cast shadow.

To get a custom character to show up in a game that supports Liv, you have to either download an “avatar” from modelsaber.com, or use a certain Unity project to set up a character prefab. I followed this guide to my model to show up in the game. Below is a screenshot of the prefab in Unity, where I’ve marked the game objects with colors. The blue ones are points that the script will move according to the tracking data from trackers, thus moving the whole character. Green ones are bend targets for arm and leg IK.

In I only have trackers for head and hands, but it would be possible to use trackers for waist and ankles as well to get a full-body motion capture. From Liv, it’s also possible to hide parts of the character model. I hid the legs because I had to replace them with a walking cycle anyway. Here are some clips of the skeleton model in use. The whole figure from the waist up is moving according to the movement of just the helmet and hands.

Rough Cut

Next step was to compile some of the gameplay clips into a structure mockup of the fan trailer. The first thing was to find a suitable audio track. Some of the tracks in the game itself would’ve worked best but I wanted to get stock music instead to avoid the video getting removed. I ended up going with the track “BreakBeat Sport” by AlchemyProduction since it has both calm and intense parts and possibility for a drop. After choosing the track, I cut it into a suitable trailer structure, and placed the gameplay clips to fit the music.

Animating over the skeleton in Blender

I modeled a low-poly female figure, trying to have it not too far from the style of the game, even though it has some very cool shader effects with pulsating polygons and chromatic aberration which I didn’t even try to mimic.

I picked the clips from my structure mockup where legs should be shown, and opened them in Blender. Then I set up the camera with the video on a background plane so that I could animate the figure’s movements to match the skeleton.

Next I exported the animations as obj sequences and opened them in After Effects. I removed the skeleton from the video clips with the built-in Content-Aware Fill tool, placed the figure on top using Element 3d plugin, and added a simple cast shadow underneath the character. Here’s a before and after clip of one of the scenes.

Rest of the animated scenes:

Animating the figure in Blender was very slow and tedious, and I wasn’t too happy with the result either. Tracing all sudden movements of a figure reacting to surrounding enemies by hand seemed too laborous of a process to use for a big amount of scenes. I ended up animating 4 scenes this way (as well as I could in a reasonable amount of time), and decided to recapture the rest of them with the character model used directly as a Liv avatar, cropped out from the waist down.

Final result

Here’s the final video:

Leave a Reply

Your email address will not be published.