Applying Machine Learning into Your Product

Welcome, Housekeeping, and Productside Overview

Roger Snyder | 00:00:00–00:06:00
Good morning everyone, and welcome. My name is Roger Snyder and I’m the Vice President of Marketing at Productside. Thank you so much for joining us this morning, as today we’ll be discussing machine learning and whether it can help add value to benefit your customers.

I am joined today by Bastiane Huang, who is Product at Osaro. Bastiane has extensive experience in product management and business development. She currently works at Osaro, a San Francisco–based startup that builds machine learning software for robotic vision and control and works with the Amazon Alexa group. She also worked with Harvard’s Future of Work initiative and writes about artificial-intelligence-enabled robotics, machine learning, and product management for Robotics Business Review and Harvard Business Review. So I’m really excited to have your expertise and your knowledge with us today, Bastiane. Thank you for joining and being with us today.

Bastiane Huang | 00:06:00–00:06:30
Thank you, Roger. It’s my pleasure to be here.

Roger Snyder | 00:06:30–00:10:30
Great. So let’s go over a few housekeeping items before we get into the meat of our presentation.

After this webinar, you can continue to stay engaged with our product management community, and these days it’s more important than ever to get connected with peers and have some great conversations. So join our LinkedIn group – I’ll put the link into the chat box for you – and join this community because we have great conversations. You’ll also be able to stay on top of all of the exciting new assets and free resources that we provide to our community through our LinkedIn group.

Let’s talk a little bit about Productside and then some of the logistics here. During the webinar, we would love for you to engage with us. We would love to hear your questions. So use the questions box – it’s outlined here – and type your questions in there. We’re going to have time for Q&A at the end of the webinar, so we need your questions to make that a nice interactive time for us to learn a little bit more about what you really want to learn about this exciting topic and to pick Bastiane’s brain, because she is quite knowledgeable. I’ve read several of her articles and I learned a lot. Even though I have a degree in computer science, it’s been a while, and I’ve picked up a lot reading her materials. So I’d love to get some questions from you so that we can have a great conversation.

Now, our most popular question when we run these webinars is, “Can I watch this webinar later?” and the answer is absolutely yes. Every attendee will receive a link in our follow-up email that will allow you to view the webinar recording after it has ended.

Before I hand this webinar over to Bastiane, let’s go over a little bit about Productside. At Productside, our mission is to empower product managers to do great product management and become product leaders so that ultimately you have the knowledge and tools to deliver products that matter. That’s what really matters to us – helping you deliver products that matter. So if you need help improving your product management or product marketing skills, you can see we provide a variety of different services. I encourage you to check us out at Productside.com.

Digital Product Management and Webinar Series Context

Roger Snyder | 00:10:30–00:14:30
Just this week we launched a whole new course. It’s called Digital Product Management, and this course focuses on the key skills and the tools needed by product managers to quickly and iteratively discover, design, deliver, and grow digital products – and be able to do that with confidence.

This is an exciting area. There are a number of folks now who are adding a digital component to a product. For example, if you have a hot tub and you want to have an app that allows you to then turn that hot tub on as you’re coming home from work, so it’s nice and warm when you get home, that’s a digital product being added to an existing product. And many product managers aren’t familiar with the pace of innovation required to run a digital product. If you’re a hot tub product manager and you now need to get into the digital space, this is going to be a course to help you. It’ll also help folks in IT organizations and even new product managers who are just getting their feet wet in the digital space. So check out this course – I will paste a link to this course also into the chat box so you can check it out.

We’re excited to continue to offer you this leadership webinar series – especially during this time when it’s been very difficult to innovate when all of us have been separated. We’ve spoken with various peers to uncover your priorities and we love to cover various topics for you. Today, that’s why we’re bringing you this special topic on machine learning. It was a topic that polled very well in terms of interest and it falls into our theme of “implications of change.” Again, ultimately our goal with this series is to help enable you and your teams to build better products.

We always love to know who’s in our audience. Today, nearly 50 percent of our audience is product managers and product marketing managers, but we have a good scattering of over 20 percent of vice presidents and directors of product management, a nice representation of product owners as well, and a big group of other folks – probably people who are interested in product management or just interested in machine learning from a product management perspective.

And with that, now I want to get to the exciting stuff. I want to let Bastiane take over. So, Bastiane – take it away.

Speaker Introduction and Agenda

Bastiane Huang | 00:14:30–00:18:00
Thank you so much, Roger. Hey everybody, good morning. My name is Bastiane, and today I want to share a little bit of my experiences managing machine learning products.

Before we start, one thing to know is that this is still a very nascent area. It’s still evolving, and there’s really no one way of doing things. Throughout this presentation, it’d be great if you can also think about the best way to apply machine learning to your products. Maybe you will decide that it’s not the best time to apply machine learning to your products just yet. What I want to do here is really just to encourage more discussion. That’s why I also write articles and host events for machine learning product managers to exchange thoughts from time to time. I’d love to get your feedback. If you would like to connect, you can visit this link, bastiane.substack.com, or go to my Medium.

Here’s the agenda for today. I’ll do a brief introduction about myself, what kind of machine learning products I’ve worked on – this is background and some concepts for you. Then I will go through some of the basics about machine learning: what is machine learning, why is it so important, what is the difference between artificial intelligence and machine learning. Then we’ll get to managing machine learning products: why is it more challenging and more difficult than managing other types of software products, how is it different, and what are some of the best practices and key principles.

Background and What Osaro Does

Bastiane Huang | 00:18:00–00:23:00
A little bit more about myself. I studied computer science in my undergrad. Over the past few years, I’ve been working mostly on product and business development in the tech and manufacturing industries. I first worked on video recognition software that does facial recognition, people counting, gender identification as well as license plate recognition. Then I joined Amazon Alexa as a Senior Product Manager and worked on Alexa’s ELUX engine, which is the core platform of Alexa. I also worked with Harvard Business School’s Managing the Future of Work initiative, writing articles and cases about artificial intelligence, automation, and their impact on our future work.

Now I’m with Osaro, so I want to give you a little bit of background about what Osaro does, just because I will be using Osaro’s machine learning products as examples throughout the whole presentation. Broadly speaking, Osaro works on deep learning and reinforcement learning, and we specialize in computer vision – more specifically, robotic vision and robotic control.

As you can see from the slide, essentially our system takes images from cameras and uses those images to predict the optimal grasp poses for a wide range of different items. Unlike traditional machine vision, we don’t need prior information like CAD models or registration of the items. We just use the image to predict optimal pick places and also the orientation of objects. With this information, our customers – who are normally warehouse owners or manufacturers – can then use this information to automate things like machine tending, packaging, and kitting.

This sounds very specialized in robotics, but the fundamental machine learning architecture and technology is actually very similar to the computer vision deep learning image recognition technology that you’ve seen in your daily life.

Roger Snyder | 00:23:00–00:24:00
So, Bastiane, if I understand what this is about – if you stay on that slide – those little circles are not actually on those objects, right? That’s your algorithms identifying this is the right place to grasp each of these items in the bin. Is that right?

Bastiane Huang | 00:24:00–00:25:00
Exactly. And it’s also different from traditional robotic programming, where engineers program a robot and they know, “Oh, I need to go after the center of the bottles.” This is all self-learned by the machine. The machine will try to pick by itself, and if it fails it will try a different way. If it succeeds, it will continue to pick from this position.

Roger Snyder | 00:25:00–00:26:00
That’s amazing, because now you have a bin with a mix of sizes and shapes of bottles, and the algorithms allow the robot to react in real time to new circumstances, new environments, new types of bottles, right?

Bastiane Huang | 00:26:00–00:27:00
Yeah. So that’s actually what we’re about to get to. The main difference that machine learning brings is really to allow a machine to be able to learn by itself and adapt to changes in the environment.

Audience Poll and Why Machine Learning Matters

Bastiane Huang | 00:27:00–00:28:30
So let’s find out a little bit more about our audience and launch this poll. We’d love to get to know you more: How much do you know about machine learning?

Option one: I have extensive experience with machine learning as a product manager.
Option two: I have some experience with machine learning as a product manager.
Option three: I have some experience with machine learning but not as a product manager.
Option four: I’m a product manager curious about adding machine learning to my product.
Or: I’m not a product manager and just learning about machine learning now.

Roger Snyder | 00:28:30–00:30:00
This helps us get a sense of both how well the audience knows machine learning and what kind of perspective they’re coming from. Are they coming at this as a product manager, or maybe from a more general perspective?

I’ll give people a chance another second or two to vote, just to be sure, but I’m thinking we’re seeing a pretty obvious trend here.

Let me close the poll and share the results. Fifty-six percent of the audience said that they are product managers who are curious about adding machine learning to their product – and that’s what you and I kind of suspected might be the audience. Nineteen percent said they have some experience with machine learning but not as a product manager. Fourteen percent: “I’m not a product manager and just learning about machine learning now.” Eleven percent: “I have some experience with machine learning as a product manager.” And one percent: “I have extensive experience.”

So the sweet spot here is: “I’m a product manager curious about adding machine learning to my product.” Those are the folks we should try to appeal to today as we’re talking about this.

Bastiane Huang | 00:30:00–00:34:30
Yeah, that’s helpful. So let’s get into machine learning and talk about why it’s important.

As Roger and I already alluded to, the key difference with machine learning is that it enables a move away from having to manually program the machine to true automation, to true autonomy. That means that the machine can now self-learn a lot of different things it couldn’t before, and machines can make predictions and improve insights based on patterns they identify in data without humans explicitly telling them what to do. That’s why machine learning is particularly useful for challenging problems that are difficult for people to explain to machines.

It also means that machine learning can make your products more personalized, more automated, and even more precise. Here I have an example: I borrowed the definition for self-driving cars. As you can see, if you’re familiar with autonomous cars you know that there are actually six levels, from level zero – which is essentially no automation – all the way up to level five, full autonomy.

Right now most of the systems we see, for example robots in factories, where we see a lot of automation going on in factories and warehouses, those are actually either level one or level two.

Level one is mostly driver assistance, so humans still take control with just a little assistance from the system. We call it a single automated operation. Someone – mostly engineers – still has to go and program a robot, and the robot or machine will just repeat the same process over and over again without taking any input from the environment.

When you get to level two, partial automation, the machines do get sensor input from the environment. For example, you can attach a machine-vision camera to a robot arm so the robot can now react to some changes in the environment. But these changes need to be pre-programmed, so it cannot be something that’s totally new or surprising. Otherwise the robot still cannot respond to it.

When you get to level three and level four, right now because we have deep learning and reinforcement learning we actually start to see level three and level four autonomous systems – conditional and high autonomy. In this case, you don’t really need humans to go in and actually program a robot. The robot will be able to learn to recognize a wide range of different items and learn to react to them. So that’s the benefit of machine learning.

Roger Snyder | 00:34:30–00:36:00
I have a question or two here. Several car companies now are starting to have the ability to self-park. Is that parking kind of a level two, or is that a level three level of autonomy?

Bastiane Huang | 00:36:00–00:37:30
I would say it’s probably almost level three. The difference between level three and level four is that level three only works in a very limited scenario and level four is high autonomy across a wide range of different scenarios. In that parking situation, it only works when you are trying to park, and sometimes it even fails. But it’s still autonomous – it doesn’t really require people to do anything.

Roger Snyder | 00:37:30–00:38:00
It’s a pretty amazing technology. And then I assume the Tesla self-driving is more like a level four?

Bastiane Huang | 00:38:00–00:39:00
I think it’s getting close, but it still requires the driver to actually take over from time to time. So I wouldn’t say it’s full autonomy, but I think it’s pretty high autonomy in most cases.

Industry Adoption and ML vs Mobile

Bastiane Huang | 00:39:00–00:45:00
So let’s talk about why machine learning is important from a business perspective.

The adoption of machine learning has been rapidly advancing across various business sectors. Nearly half of the companies have incorporated one or more artificial intelligence capabilities in their processes, and another 30 percent are piloting AI projects, according to a recent McKinsey survey.

If we look at McKinsey’s charts, you can see that across many different industries and functions there is potential to use AI and machine learning. They also rank different industries by their potential to be automated and identify key tasks that they consider easy to automate in the near future. For example, predictable physical work, processing data, or collecting data. The top three industries that have the highest automation potential are a combination of food manufacturing, transportation, and warehousing.

We can also see this from exploding AI funding over the past few years. A lot of venture financing is going into AI, and many more AI companies are being founded. A lot of big companies like Google, Apple, and Microsoft have each acquired more than ten AI startups. Traditional incumbents have also slowly started to react. For example, Nike acquired an AI-powered inventory management startup called Celect, and McDonald’s acquired a personalization platform called Dynamic Yield. So you can really see that AI is entering every different industry.

We can also see that by trying to draw a parallel and compare the adoption of AI to the adoption of mobile over the past few decades. If we look back at history, mobile really started to grow after 2008, when Apple launched the App Store and we started to see a lot more app companies go IPO, like Eventbrite, Tinder, and others. In 2017, Google actually announced that they were going to switch from a “mobile-first” strategy to an “AI-first” strategy. So we expect to see a lot more AI companies go IPO and a lot more AI companies being founded in the near future.

Even though we’re still in a very early stage of machine learning, it’s already not hard to see why machine learning is expected to have an even more transformative impact than mobile technology. However, the transition to machine learning could also be more challenging and harder than the transition to mobile, and that’s what we’re talking about today.

Types of Products and How ML Fits

Bastiane Huang | 00:45:00–00:50:30
Before we talk about why managing machine learning products can be harder, let’s think about what kind of products we’re actually working on.

What kind of machine learning products are you actually building or thinking about? Are you working on enterprise products or consumer products?

In my experience, working on consumer machine learning products like smart speakers – like Alexa – usually has a stronger social component with its users. The user experience is really, really important and plays a much more critical role in designing consumer machine learning products. In this case, machine learning tends to become an enabler for better user experience, for better user interaction. For example, we use natural language processing to improve the interaction so that with Alexa, users can have a more natural interaction.

When I worked on Alexa, I spent a lot of time working with our user experience and UI teams to optimize user experience, and we went through what we call a “voice user interface” online and tried to make sure that we deliver the best experience. It’s a very different experience compared to my experience at Osaro, which is B2B and more industrial products. There, it’s really more about the accuracy, reliability, and robustness that our machine learning models can offer, because the key is to be able to predict the optimal pick points so that users can truly automate their processes. Our customers put less emphasis on the user experience or user interface itself. So that’s something for you to think about: when you have limited resources and you want to consider which area you want to focus on.

Another thing to think about is: Are you building a machine learning product, or are you just applying machine learning to your products?

If the core value of your product comes from the machine learning models – like in the case of Osaro – then you’re likely building a machine learning product. On the other hand, if machine learning is used to enhance user experience or performance in your products – to do personalization, customization, or optimize recommendations – then you’re more likely applying machine learning to your products. I think that’s the case for most companies, and probably most of you here as well.

In that case, it’s more important to understand the input and output of the model, but not necessarily the actual technical details like the exact architecture of the neural network. For example, you just need to know if the model takes in demographic data of users and tries to predict their monthly spending on the platform. You just need to know the input and output of the machine learning model.

A lot of companies also leverage existing solutions. They don’t really need to reinvent the wheel and build everything from scratch. There are many off-the-shelf solutions in the market. The organizational structure will also tend to be different. For companies building machine learning products, or larger organizations with heavy investments in machine learning like Facebook and Google, it’s common that they hire a lot of machine learning researchers, scientists, PhDs, and pure-play machine learning engineers to develop products.

For companies applying machine learning to their products, or smaller companies, it’s usually better to hire multidisciplinary machine learning engineers or train your software engineers to learn machine learning instead of hiring machine learning researchers or scientists.

Machine Learning Basics (Supervised, Unsupervised, and Reinforcement Learning)

Bastiane Huang | 00:50:30–00:57:30
As product managers, we need to know what can and cannot be done with machine learning, and when we should and should not use machine learning. So let’s go through some machine learning basics.

There are three main types of machine learning, as you may already know. The main type is called supervised learning. That’s actually the most common – probably 80 to 90 percent of the machine learning products you see out there are supervised learning. It’s called supervised because it’s supervised by humans. It learns from labeled data, where humans label the data, and then it predicts outcomes. You feed the algorithms thousands of pictures of cats that are labeled “cat,” and the model learns what a cat looks like. Then when it sees a new image, it can classify it as a cat or a dog.

Unsupervised learning, as suggested from its name, doesn’t need any labeled data or supervision from humans. It just identifies patterns without any labeled data, and we can use that for clustering, anomaly detection, or association.

Reinforcement learning learns as the algorithm gets feedback from the environment. You reinforce the algorithm with positive feedback or negative feedback, and it’s usually used in controlled domains like robotics or self-driving cars.

With supervised learning, you often build a deep neural network that takes labeled images as input. In each layer it transforms the input into slightly more abstract and composite representations and eventually learns to recognize a cat, a dog, or a human. When it comes to control domains, we don’t necessarily need labeled images. Just imagine that you are learning how to walk as a human. We don’t need to see labeled images of joint angles. We just try to walk, and if we fall we know we need to adjust the way we walk. It’s very similar for robots and machines. You have an agent – usually a robot – that takes an action based on its current policy. Based on the actions it takes, it receives feedback or reward from the environment, then moves on to the next stage and learns from the reward, adjusts itself, and repeats this process until it maximizes long-term return.

You can also combine deep learning and reinforcement learning so you have a machine that’s able to recognize different items and then react to them. That’s kind of what we do at Osaro: we use deep learning to allow the robot to recognize tomatoes versus chicken nuggets and where to pick them, and then we use reinforcement learning to train the robot to use different actions or trajectories to pick those items.

Of course, you can also combine a lot of other learning techniques like feature learning, transfer learning, or imitation learning to shorten the learning process. But those are the main areas of machine learning that we often see.

We also see a lot of image classification examples using deep learning. We see a lot of natural language processing, like with chatbots, voice assistants, or just pre-processing data. Facebook, for example, uses deep learning to do image classification so they can automatically generate audio captions or hashtags from photos.

And I think your last bullet is probably where machine learning is touching people the most – with these automated assistants and voice assistants. Every time you’re using Alexa, every time you’re using Siri, you’re taking advantage of something that’s been trained with machine learning.

Why Managing ML Products Is Harder

Bastiane Huang | 00:57:30–01:05:00
That brings us to why machine learning products are so challenging.

As we talked about before, traditional software requires us to go through a lot of different steps to teach the machine what to do. We have to come up with explicit rules. With machine learning, it’s much easier in some ways because we just need to define the problem, prepare the data, label the images, and train models with the labeled images. But because of this iterative nature, because machine learning takes a lot more experiments, it also involves more uncertainties.

Many machine learning algorithms lack transparency. They act like a black box that takes input and outputs a prediction. It’s really hard to take full control of a machine learning model, and it’s hard to explain to your users or other stakeholders how exactly the model works and to get buy-in. Also, because it’s probabilistic – it’s not deterministic – you also need to manage the expectations from your users, because it’s probably not going to be 100 percent accurate.

In a lot of critical domains like healthcare, where accountability and transparency are extremely important, it’s really challenging to align machine learning work with customer problems in a way that has a clear understanding of how the algorithm actually works.

Because machine learning is highly iterative, oftentimes even machine learning scientists have to test a few approaches before choosing a satisfying one. That’s why it’s often more difficult to define milestones and estimate a timeline for machine learning products. You need to give the team more flexibility and room to explore, but also guide the team to focus on the customer problem, not just on researching the fanciest machine learning models.

We can really see that building a machine learning product is not just a technical challenge – it’s also a structural challenge. It requires a totally different mindset and requires us to embrace experiments, be more data-driven, and be more open to uncertainties. It’s not just for the product managers; it’s true for the entire organization.

Another thing to remember is that machine learning is still very new and still evolving. If we look at the history of software engineering, it first appeared in 1965, which is basically 15 years after programming languages started to appear. Almost 20 years later, the Software Engineering Institute was established to manage software engineering processes. Today we have generally accepted best practices for software engineering.

Machine learning, on the other hand, only started to flourish as a separate field in the 1990s, and deep learning as a subset of machine learning only started to become more popular in 2012 after the rise of AlexNet. Compared to software engineering, machine learning is still in its infancy. That’s why we don’t really see a lot of industry standards, metrics, infrastructures, or tools. Everybody is still exploring best practices and killer application use cases for machine learning. It’s exciting, but it also means it’s going to be more challenging during this early stage of machine learning. We still need to figure out what is the best way of managing these products.

Data, Teams, and When Not to Use ML

Bastiane Huang | 01:05:00–01:12:00
Even if you’re building a machine learning product, it’s rarely the case that you only involve machine learning. It’s often interdisciplinary and involves not only models but also software engineering, back-end infrastructure, data analytics, UX/UI, and sometimes hardware too. As the product manager, the challenge is to manage a cross-functional team and deal with interdependencies and potential clashes among teams, because machine learning is really different from other disciplines.

You also have to think about your data strategy. Training machine learning models requires a lot of high-quality data. We know that machine learning models outperform traditional algorithms when there’s a lot of data. It’s really important to outline your data acquisition strategy from day one. Are you going to buy data, partner with other companies, gather data from your customers? Is there any privacy concern? Can you generate data internally? What do your competitors do? Do you have an advantage in terms of proprietary data? Those are things to think about before you start to build a machine learning product.

At the same time, you should avoid using machine learning when it’s not needed. If the problem you’re solving can be addressed with simple rules, or the solution doesn’t need to adapt, or you can’t access the data you need, then you probably shouldn’t use machine learning. If your product requires high accuracy or full transparency – for example in healthcare – and you must comply with regulation, then it can be tricky to use machine learning. In those cases, you may need to rely on more traditional methods or combine them with machine learning in a very careful way.

You also need a multifunctional team: not just machine learning engineers, but also data scientists, software engineers, UX/UI specialists, and domain experts. Communication with users is very important. The performance of machine learning models improves as they are trained with more data, so it’s great that these models constantly improve themselves. But it also means performance will not be perfect from day one. Is that okay for your customers? How do you define acceptance criteria? If your customers need a model to perform perfectly from day one, what do you do? Do you provide a pre-trained model, or a backup plan? What happens when your model fails to deliver the expected performance?

You also need to manage user expectations. For example, if you have a humanoid robot like Pepper, users will usually have much higher expectations because it looks more intelligent and more like a human. But if your technology is not able to deliver that level of natural interaction or intelligent responses, it’s probably better to position your product differently – for example as a smart speaker – to set the right expectation. Design plays a big role here, because design can subconsciously set user expectations.

Another point is building trust. We mentioned that machine learning algorithms act like black boxes. How do we help our users understand why we made a prediction or recommendation? Amazon, for example, shows “Customers who viewed this item also viewed…” to at least give you a simple one-line explanation. Facebook has “Why am I seeing this?” on posts, and you can give feedback. It happens all the time that the machine learning model makes the wrong prediction. In that case, how do you design a pleasant, elegant user interface to deal with those errors and mistakes? How do you allow users to give you feedback so you can continue to retrain and improve your models?

Q&A and Closing

Roger Snyder | 01:12:00–01:18:00
I think we want to get to Q&A as soon as we can, because we’ve got a lot of great questions. Let me start the poll, and folks can answer this last question while we’re doing Q&A.

We’ve had questions like: “How do we work out the ROI or benefits relative to less sophisticated statistical or analytics methods?” “How do we handle safety in high-risk domains like autonomous driving?” and “What practices need to be included to ensure non-discriminatory actions when the data may lead to discriminatory actions?” These are fantastic, deep questions for product managers to be thinking about.

Bastiane Huang | 01:18:00–01:24:30
On ROI, I think it really depends on the use case. In our case at Osaro, we use machine learning models to enable robots to automate things humans are doing. What we can do is first agree on acceptance criteria with our users or customers and use the existing solution as a baseline. What is the accuracy and performance of the existing system? Then we compare that with our expected performance metrics. For example, right now the current accuracy might be 80 percent and we expect to improve that to 90 or 95 percent. You can use that to calculate ROI.

For safety in high-risk domains, this is still an ongoing research area. A lot of researchers are looking into transparency and explainability of machine learning models. Right now it’s difficult to say for sure if a model is “very safe” because machine learning is probabilistic. Companies like Waymo and others are doing all kinds of different tests: offline testing, simulation, and physical testing in extreme scenarios. As a product manager, you can work with your team to identify as many edge cases as possible and design tests for them. Handling 100 percent of all possible use cases is very difficult, but we can keep improving.

On fairness and non-discrimination, machine learning can be biased because it’s learning from human data. The data itself may be biased. So we need to start from the data. As a product manager, you should look into the data with your data scientists and design test cases to see if your model behaves in a biased way. If you can, you should also look into whether your data correctly represents what you want to predict. Data strategy and data integrity are really not just the responsibility of data scientists. It’s the responsibility of all stakeholders and even the entire company. Executives, stakeholders, and product managers need to agree on a data strategy. If you don’t have a data strategy, data scientists have no way to magically come up with good-quality data for you.

Roger Snyder | 01:24:30–01:27:30
That’s a really great point. The product manager has to stay in the loop and be involved in testing to double-check these algorithms and make sure bias doesn’t creep in, but also in the larger picture of putting this whole organization together so it can holistically drive value out of machine learning without having some of these problems crop up.

We have one or two slides that we want to just go over before people depart. Next week we’ll have an “Ask Me Anything” session, so sign up to learn about getting ahead of digital transformation before it gets ahead of you.

I also want you to have an opportunity to connect with Bastiane, because she did a great job today filling us product managers in on both the benefits and potential perils of machine learning. Thank you so much, Bastiane, for your expertise and joining us today.

Bastiane Huang | 01:27:30–01:28:30
Thank you so much, Roger. Thank you so much, everybody.

Roger Snyder | 01:28:30–01:29:30
All right. We’ll talk to you all later. Have a great day, and we’ll see you soon.