Day 2 of F8 2019: Advances in Computer Vision, Inclusive AI and Other Highlights

The second day of F8 focused on the long-term investments Facebook is making in AI and AR/VR.

Chief Technology Officer Mike Schroepfer

In the opening keynote, Chief Technology Officer Mike Schroepfer talked about the AI tools we’re using to address a range of challenges across our products — and why he’s optimistic about what comes next. Schroepfer was followed by Manohar Paluri and Joaquin Quinonero Candela from Facebook AI, Product Design’s Margaret Stewart, Lade Obamehinti, Lindsay Young, and Ronald Mallet from AR/VR.

Artificial Intelligence

Manohar Paluri from Facebook’s AI team

AI powers a wide range of products at Facebook. In recent years, this has included Facebook work to proactively detect content that violates their policies. To help them catch more of this problematic content, Facebook is working to make sure their AI systems can understand content with as little supervision as possible. And they have made important strides, but these are early efforts and there is still a long way to go. Advances in natural language processing (NLP) have helped them to create a digital common language for translation, so they can catch harmful content across more languages. And a new approach to object recognition called Panoptic FPN has helped their AI-powered systems understand context from the backgrounds of photos. Training models that combine visual and audio signals further improves results.

[Photo updated on May 1, 2019 at 12:05PM PT] Technologies like Panoptic FPN, shown here, help AI-powered systems understand context from the backgrounds of photos

Facebook work around natural language processing is important, but many techniques work best for the most common languages. Facebook need a way to support the many languages where there aren’t enough samples to train on. Self-supervised learning can help as they are able to train models for new languages without having humans label additional datasets for those new languages. This lets Facebook better understand relevant content — including policy violations — without translating each sentence. These techniques help make sure all their classifiers are catching problematic content in more languages than Facebook previously could.

Joaquin Quinonero Candela from Facebook’s AI team

AI is instrumental as Facebook work to keep their platform safe — but they know it comes with risks. Namely, it can reflect and amplify bias. To address this, Facebook is building best practices for fairness — to ensure AI protects people and does not discriminate against them — into every step of product development. When AI models are trained by humans on datasets involving people, there is an inherent representational risk. If the datasets contain limitations, flaws or other issues, the resulting models may perform differently for different people. To manage that risk, Facebook developed a new process for inclusive AI. This process provides guidelines to help researchers and programmers design datasets, measure product performance, and test new systems through the lens of inclusivity. For vision, those dimensions include skin tone, age and gender presentation and for voice, they include dialect, age and gender. The inclusive AI process is now in use across many product teams at Facebook and baked into the development of new features.


Lade Obamehinti from the AR/VR team

One of the areas where Facebook is using the inclusive AI process is augmented reality (AR). Spark AR engineers use it to ensure their software delivers quality AR effects for everyone. For instance, some of the effects are triggered by a hand gesture, so the training data included various skin tones under a variety of lighting conditions to ensure the system would recognize a hand in front of the camera. Oculus engineers are also using this process for voice commands in virtual reality (VR), using representative data across dialects, ages and genders. As Facebook work to ensure their technology does not exclude people, they are also working to make sure it helps bring people together. And with VR, Facebook see a future whether people can interact, and come together, regardless of physical distance. But to really achieve this, people need to feel completely present in VR. That means Facebook need truly lifelike avatars, with gestures, facial expressions, and tone of voice that add nuance to our conversations.

Facebook has shown groundbreaking realism in their Codec Avatars faces, which let people interact in real time in VR. But genuine communication requires the full body. That’s why Facebook is developing fully adaptive, physics-based models that reproduce a 3D avatar with data from a limited number of sensors. Facebook is using a layered approach that replicates human anatomy and can automatically adapt to perfectly match any individual’s appearance and unique motion. Facebook design these models from the inside-out, developing a virtual skeleton then layering on the muscular structure, skin and clothing. The result is avatars that are realistic – right down to muscle movement and the draping of clothes. Facebook still have a long way to go before this research results in a product, but we are encouraged by the results so far.

Ronald Mallet from the AR/VR team

Like AR, Facebook want to make VR inclusive and safe for everyone. They have built preventive systems — like a code of conduct for everyone that uses or builds for their headsets — that foster respectful culture and interactions. And Facebook has built reactive systems — including tools for reporting or blocking users who are violating guidelines.

When Facebook released their own social VR apps, including Spaces, Venues and Rooms, they incorporated safety into the core design of the experience. An orientation video introduces people to some of the features designed to make them feel more comfortable interacting with a large group of people while in VR. For instance, safety bubble is a feature that prevents people or objects from coming closer than you’d like. If one avatar enters another’s safety bubble, both avatars become invisible to each other. Facebook also has live moderators on hand to help ensure good behavior and review reports of inappropriate behavior.

Lindsay Young from the AR/VR team

The tools and processes Facebook shared today are all part of how they are preparing for what comes next. For Facebook, this work is about bringing voice and opportunity to people all around the world and helping people stay connected to one another.

Source: Facebook Newsroom

  • Spotify
  • Branders Magazine
  • Branders Twitter
  • LinkedIn Branders
  • Facebook Branders

© 2019 by Branders Magazine