The story behind The Last of Us Part II’s staggeringly realistic in-game character facial animation

22 0
The story behind The Last of Us Part II’s staggeringly realistic in-game character facial animation

An exclusive look at the tech and multi-team collaboration that shaped Naughty Dog’s latest adventure.

It’s my first day in Seattle. Ellie and Dina are exploring a side room of a ruined synagogue. As has happened with frequent regularity since the game’s start, I open up Photo Mode. Naughty Dog’s latest is quickly proving to be a treasure trove for the photographic eye.

I pan round to compose my shot, looking to capture expressions, body language. To work out the best position for lighting. In doing so I spot Ellie’s gaze seemingly drawn to a painting in the room. I can’t interact with it, there’s no dialogue attached to its presence. Yet her facial expression gives every indication that she’s mulling the artwork over. Could Naughty Dog really have created subtle reactions for everything in the game?


Spoiler alert: This post contains story details about The Last of Us Part II


The answer, in a way, remarkably, is yes. The illusion is carefully crafted from multiple game systems working simultaneously. All built to depict the most realistic rendering of in-game character models possible.

These models sit separate from the motion capture tech utilized to render real-world actors in-game for cinematics. When those end and gameplay begins, that’s when the careful handiwork coordinated between multiple teams in the studio kicks in.

Taking me on a tour of this is Keith Paciello, the studio animator who masterminded the in-game facial tech after his internal pitch was greenlit and who collaborated on numerous other animation processes.

“In that instance,” he explains during our video call after I describe the scene to him, “you as a player are aiming with the controller for Ellie to look at the painting, which is triggering a ‘look at target’ placed by a designer. On top of that, I animated small eye darts (saccades) within the character’s facial idles to try and indicate an overall thought process. So animated eye saccades sitting on top of the eye-aim, work together to create what looks like focus and thought process.”

It’s an example of that close collaboration between the different teams during development, but also helps highlight the work being performed by Paciello’s Emotional Systematic Facial Animation system. In simplest terms, it picks a facial expression from a range of nearly 20 different emotional states for any of the 25 key characters that are on-screen. That covers leads, co-op partners, enemies, even Infected to an extent.

Facial animation working in unison with eye movements, body language, breathing… all interlinked and triggered by script beats, dialogue, encounters or ambient moments, like Ellie seemingly being absorbed by a painting. The illusion of emotion is constructed with mathematical precision. “It brings a depth to the characters that we’ve never seen before,” says Paciello.

After 17 years working as an animator, nine of which have been spent at Naughty Dog and most recently working as Lead Cinematic Animator on Uncharted: Lost Legacy, Keith was already invested in finding the best way to visually capture a character’s feelings.

Yet the epiphany to create something groundbreaking for The Last of Us Part II came from, of all things, a single blade of grass.

“Everyone was stepping up their game for TLOUII,” recalls Paciello during our video call. “We were looking at and talking about how to make a blade of grass even better [in game]. In doing so, we panned up, and there was this blank face on the character. I was like, “Oh.” It was then I wondered how we could simply, across the entire game, add these emotional beats to the characters, so at any point, you can tell what that character is feeling.” 

Most of his development time would be spent solving that riddle. Using the Ellie facial model as a base, he started by sculpting expressions based on seven universally recognizable emotions: joy, sadness, fear, anger, surprise, disgust and contempt. Liaising with the dialogue team, they tagged emotions to be triggered at specific lines, building a solid foundation and allowing characters to smoothly transition from one to another. “We could emotionally pace out our characters from the absolute beginning of the game to the absolute end of the game and blend in and out of cinematics seamlessly.”

Paciello was then asked to support melee, fashioning realistic reactions for combatants. He also aided in animating the breathing system (“six animations, from small breathing all the way up to exhausted”) which would go to such lengths as to see characters switch from open to closed mouth breathing depending on their proximity to an enemy. “That was one of my favourite collaborations on the project,” reminisces Neil.

Outside of script-specific triggers, characters would also have a number of ‘neutral’ idle states with emotional overlays. He also brought his expertise to hand-sculpting poses for each character to match their emotional states. Some 40 poses for each of the 15-20 emotion sets. By the end of production, he’d totted up an impressive 15,000 individual, hand-sculpted poses.

That hard work was paying off. When the system was fully activated, positive feedback was universal: staff, playtesters, even Keith himself were finding a deeper emotional connection with the game’s cast during gameplay.

Paciello points to a pivotal moment when Ellie is at Joel’s grave as an example of the seamless transitions between cinematic and gameplay, “Flip the camera [after the cinematic ends] and you see that she’s still devastated,” he explains.”I talked to the character team to see if we could get the textures and make sure it looks like she was just crying, so everything blends together nicely.” 

Yet even with the system in place and working as it promised to, the teams continued to fine tune every scene, further polishing each character’s emotional state. Ellie’s obvious exhaustion during the Santa Barbara sequence was the subject of additional focus. “We didn’t know if you wanted to go with pain, because she’s just gotten punctured. Ultimately Neil was happy with the exhausted look.”

And certain scenes demanded additional, unique flourishes to reinforce key character beats. Ellie’s reaction to seeing the Tyrannosaurus Rex statue during a flashback “was specially made for that moment,” while for the climactic beach fight, Paciello went in and redid all of Abby’s emotional sets to give the scene the gravitas it deserved.

“We all learnt a lot from that,” says Paciello. “When I first got to it, Abby, being so emaciated, still looked savage. I talked to Christian and said ‘I’m going to duplicate all her emotional sets for melee and I’m going to make them all exhausted’. That way you choose between fierce and exhausted, so maybe she musters up enough strength but after she throws [a punch], goes into her exhausted state.”

I ask Paciello if he has any personal favourite moments from the game that shows the system at work. Understandably, he quickly rattles off several (“Dina and Ellie’s conversation back and forth on horseback, or Lev and Abby. Yara, in the aquarium talking about Lev.”), but ultimately he comes back to that climatic beach scene. He sees it as the culmination of everything the teams had worked on and proof that the system had achieved its goal. As Ellie and Abby come to the end of their respective arcs in an explosive, exhaustive clash, the emotional scars are clearly etched on their faces.

“It made me really made me feel like, ‘Okay, we’ve really pushed it.’ It’s what I wanted, it’s what I dreamt of when I pitched the idea of the system.”

Comments are closed.

22 Comments

  • I absolutely love every aspect of this game; In fact, I bought the limited edition PS4 Pro, hard drive, and controller to commemorate such an incredible experience. Thanks, Naughty Dog!

  • Great job, the game is awesome and the details (as usual with Naughty Dog) are amazing !
    Gameplay, story, lore expanded, a true sequel that doesn’t copy/paste the original and that’s why TLOU Part II is good

    • Well, I love the game, and I enjoyed it. Still, yes, there’s something that I feel in the story need to adjust more professionally, putting political agenda for me. It’s not the issue; the issue here it’s the way you put it, but over all, it’s a beautiful story. I feel people should try to play it and enjoy there own experience. Good job Naughty Dog.

  • Great read. Visual storytelling at its best!

  • The level of detail, and as a result, the extra work involved, is absolutely STAGGERING. Reading this article along with the Twitter posts about the breathing engine has been fascinating and just goes to prove that the people working at Naughty Dog are the best in the business. Bar none.
    I can’t even count the number of times I was playing that I looked over at my PS4 to make sure it wasn’t actually a PS5 running the game.

  • Impressive work! I wish every AAA story-based game has this level of animation. But, I know that’s impossible.

  • Thank you for this insight into the painstaking process of making these characters feel so alive. Naughty Dog titles have arguably always had the best animations I’ve ever seen and it’s really on another level in TLOU 2. Much respect & appreciation to all involved in doing this incredible work!

  • “It’s a mother****ing dinosaur!!!” Lol Ellie was so vulgar when she was younger. The attention to detail in this game is amazing. I first started noticing when I was aiming and my wife said “go into photo mod and look at her face”. I was surprised when I saw she was squinting!! Not a lot of games do this. I hope more devs do this.

  • Yeah this game probably has the best facial animations I’ve seen, a shame we won’t see it from games.

  • The animations were the best aspects of the game. Much improved facial animations and Horse animations, which I was delighted with. However I did not like the game overall. I loved the first game though.

    What I’d like to see Naughtydog do next is a Sci-fi Horror. I would love to see there take on Sci Fi.

  • Please do not take this the wrong way, but perhaps a proof reader would be a helpful edition when writing this type of article. I by no means intend to imply offense by this. A professional article that has grammatical issues can take away from the insightful things being bestowed upon the readers. I only offer this as constructive criticism. A polished project can do wonders with helping one’s ideals come across as confident, well thought out, and also helps with persuasion.
    Thankyou for your time.

  • Best character models and facial animations in the history of gaming. Interesting read and a testament to Naughty Dog’s insane attention to detail. This is the kind of stuff that really adds to the immersion subtly without it becoming cumbersome or too simulated in-game, like RDR2 at moments.

  • A small team of Naughty Dog employees should be given the task of essentially remaking the first game but with this new emotion system, updated textures and raytracing.

  • Seeing that clip of the fight from the end is still giving me the feels.

  • masterpiece.
    the best game in storytelling , best graphics , best game play and everything is in the best form .

    • The original is the best in storytelling. The sequel is pants for most of it. Ellie’s story was great, but could be better and Abby’s story was a snooze fest.

  • Just makes me realise the water fight isn’t very realistic. There’s no drag from the feet in the water, no splash. It would completely change the way you move your whole body.
    That being said. This game was very controversial obviously and everyone has an opinion on it. Like I’ve never seen before, for that it is unforgettable.

  • Young Ellie looks like the monsters in Until Dawn in that gif.

Please enter your date of birth.