Getting Product Management Sh*t Done with ChatGPT

Welcome and Introductions

Kate Fuchs | 00:00:01–00:00:31
Well, welcome everyone. Um, this is an exciting uh webinar that we’ve got for you today. Very discussion based and um we’re interested to hear what you have to say as well. So, we’ll work on on our chat. Um, but we’ve got folks from all over the place. So, good morning, good afternoon, good evening. Um, we are going to talk about getting product management done with chat GPT. So, um, if that didn’t pique your interest, I don’t know what will. My name is Kate Fuchs. I’m going to moderate today. I’m

Kate Fuchs | 00:00:32–00:01:02
a product manager with the Productside. And I will let our esteemed um discussion and presenter um individuals introduce themselves. So I’ll turn it over to Dean first and you can pass it on from there.

Dean Peters | 00:00:32–00:01:32
Thank you for that wonderful introduction, Kate. I am Dean Peters. I’m the one with the face for radio. Uh I joined Productside this past summer after about oh 20 years in the trenches doing product management and before that software engineering as a real-time systems integration integrator and engineer. Um, if my face does look a little familiar, you may have seen me speak on other places, perhaps even sing once in a while on topics like agile and product management, perhaps product tank, product camp, Red Hat, Agile Day, Pandemonium, and for some odd reason, uh, my on and off blogging has landed me in the position in the agile and product management lexicon as the zookeeper of product boards, dangerous animals of product management. So, um, you if you’re getting flashbacks from seeing

Dean Peters | 00:01:33–00:02:06
me, that’s where they’re from. Uh, Elad, I’ll hand that off to you.

Elad Simon | 00:01:33–00:02:06
Thanks for that, Dean. So, I’m Elad Simon. I’m the CEO and co-founder of Craft.io, a product management platform. Um, I’ve been doing product management and go to market management for the past 15 or 20 years or so. Um, just don’t want to disclose my age, so I’m going to say just going to keep it vague. Um, and I’ve uh I’ve been uh excited to be participating in several webinars, including a couple with our friends from Productside. Um, and so really looking forward to talk about this very exciting topic. So, um, I think we can get going. Do it. All right.

Audience Poll: Generative AI Proficiency

Kate Fuchs | 00:02:07–00:02:46
So, just to get you guys, uh, started, we kind of want to know obviously who our audience is. So take a moment um to take this little quick poll. What is your level of proficiency of generative AI? So do you use it on a daily basis? Do you use it just time to time? Um rocking the API for fine-tuning or you have no clue what we’re talking about. So go ahead and fill that out for

Kate Fuchs | 00:02:47–00:03:26
us so that we can see who is in the room. We’ll give you a quick minute to do that. And we are working on the chat. I believe our host, can you see in settings you can change that? Sorry. Can you see if in the settings for the Zoom call if you can change the um ability for everyone to message each other? That sounds quite complex to me, but I’ll try. Oh, it’s working now. Oh, wait. Okay. Can Can you guys see each other’s comments? That’s the [laughter] question. Uh, no. Okay. So, we’re going to work on

Kate Fuchs | 00:03:27–00:04:01
that. Let’s make sure that we can see each other’s. [clears throat] And people may have to look at the bottom of their Zoom, the lower right hand corner where it says may say host panelist. There’s a little drop down for everyone. I think if they select that, they might might be able to do that as well. All right. I think we got our results in. All right. Nice. Let’s see them. Um, can you guys see the results or no? Um, nope. Not yet. How can I show the results? I can share the results if you want.

Kate Fuchs | 00:04:02–00:04:23
Yeah, let’s share them. All right. So, first of all, um, there’s still a few people kind of answering. So, let me just kind of end poll. And that’s it. That’s it. Come on. Last last one. Last one. All right. Done. All right. Share results. Ah, there we go. There we go. Okay. Interesting. So, we have what about where are we now? Okay. So, about 63% of people use this, you know, in some shape or form 11% like 10% or so or on a daily basis. Okay. So, you probably know more than us. That’s good.

Dean Peters | 00:04:24–00:05:23
Um, and then, um, from time to time, 50% of people, and then 35% of people are honest enough to admit they have no clue what this is. So, that’s fantastic. Um, that’s great. And then rocking the API, which is, you know, the super expert. Um, I think they’ve just and went last last night went live for general availability. Um, so for, you know, so big news there, too. Indeed. All right. Well, I think we got our we got our understanding. Should we get Should we should we move on? Let’s move on. Thank you guys.

What Is Generative AI?

Elad Simon | 00:05:24–00:05:56
All righty. So, let’s get started. So, I guess starting with what is generative AI. So, in case I mean I I thought you know the most appropriate thing to do in the world is to ask Chad GPT what is generative AI? And so there you go. Here it is uh in front of you. Um in case I mean Dean, you want to you want to take this or you wanna I mean I’m not going to read this whole thing. Feels like a very long-winded uh a long-winded paragraph. Do you do you want to talk through what your your view of the world of of

Dean Peters | 00:05:57–00:06:31
generative AI? Well, yeah. I mean I like I said for me generative AI um just you know is really synthesizing responses based off of. So imagine uh it’s a sounding board. You’ve you it’s been listening. It’s been listening. It’s been listening and gathering information and gathering information here. And now when you’re ask start asking questions, it’s it’s synthesizing responses predictively based off of of what it knows uh in general as well as what it knows uh what you fine-tune that with.

Dean Peters | 00:06:32–00:07:05
So that’s just sort of it. If I were to go to chat GPT and say, explain it to me like I’m like I’m 10. Uh perhaps that would have been the explanation there. Uh so that’s that’s sort of the layman’s term if you need to describe it real fast to your bosses who may not be technical. Cool.

Elad Simon | 00:07:06–00:09:25
Yeah. And I think I mean for those who are more let’s say uh into the actual thing of course you know a large a large language model um which which we are seeing more and more of now nowadays. Um so the open this is CHP specifically is the open AI one. Um I mean based on the open AI one and we have now how many now? Four or five, right? So Lambda, we have Llama which is the recently announced Facebook one I believe. Yeah. And Lambda is the Google one and then we have of course open. Those are the three I would say main ones um out there. Of course there are many many many more uh LLMs out there on which this type of um type of uh technology is built on and one [clears throat] of the things that you know obviously uh I think chat GPT took from generative AI as a concept is this idea of actually you know using like language and text and like you know really synthesizing synthesizing um the capabilities of generative AI in principle because geneneral AI is not necessarily for for text, right? So it’s like it’s a general concept that says I can take anything I can learn from that and I can then create you know similar um similar patterns based on you know instructions kind of thing right so that’s like that’s the that’s the concept behind that um specifically chat GPT you know takes that notion and applies that to to language which ends up being quite a cool if nothing else quite a cool um thing that you can interact with and it feels at least feels um quite intelligent and quite con quite conversive uh if that’s a word um yeah and and if I were to add to that just from a you know a zoo animal perspective here think of the chameleon all right where where it can sort of blend in to its environments here or even maybe the octopus that that blends in and uses its environment to to respond to to whatever the conditions in the situation are the t times they can generally uh and and so they’ve taken this huge amount of data whether what whatever these models are you know whe you know and and I think you did a good job of enumerating just some of them we could spend all day going through four models you talked about uh going through whatever the AI what do you call open AI mafia whatever they’re working on those 10 individuals so we we could go all day talking about it but the general idea here is this this uh um yeah someone has parrot. In fact, I think there was uh at some point somebody had made a talked about it being a sort of a sneering parrot uh at some point. I think I like chameleon octopus only in that it it you know uh fine-tuned the right way. It blends in as opposed to being uh confrontational. Although I suspect we’ll want to cover that topic a little as well.

Risks, Hallucinations, and Fact-Checking

Elad Simon | 00:09:57–00:12:20
Yeah. And I guess I this is probably a good opportunity to ask people to like submit their questions I guess in the chat. Um I think we right you’re you’re you’re uh you’re driving on this. What you prefer question? Yep. You can you can put them in both places, but I am monitoring the chat. That sounds that sounds great. And I I see some um some questions already, so throw them in there. Um I’m wondering if either of you guys would like to touch on the underlined piece here um that even chat GPT will admit uh I so in order in order to avoid uh creating drama by our own perils we’ve decided to use this quote um directly from horse’s mouth as they say um and so um and so I um one of the quotes that I I like and this interesting it raises concerns about potential you know, content. I’m not sure about, you know, without going to the whole misinformation, manipulation of public opinions, I would just say that as a um as a source of data, let’s say it this way. Um it’s uh it’s still in its infancy. um you know that’s the that’s that’s I would say that I’m going to give one example of that if it’s okay just to just to uh before going in there um which is you know one of my own my own my own examples um and so I’ve asked Chad GPT and you know this is just a personal question for me because I’m I am as far as I’m concerned I’m the CEO of craft.io the [clears throat] as far as I know and so I asked Chad GPT on two different instances um who I mean I’ve most of you know and if you don’t then it’s a good time to mention that um Chad GPT is trained on a data set that ends in September uh 2021 so it doesn’t it isn’t um it isn’t yet uh um updated with real time and there is a big reason why it’s not updated with real time and there’s a there’s an issue with large like language models being updated on a real-time basis. We’ll leave that in a minute. But um regardless of even with that um I’ve asked him who’s the CEO of craft.io O and he gave two different answers as you can see and both of them are wrong and so um and I’m not using that of course um not at all using that example to say you know it’s not useful or you know it doesn’t have uh that ability to do things but as a source of data and Dean I think we’ll talk about that in a couple of minutes when we talk about use as a I love that you’re setting the table like this and and I know some of you just want to dive right into the uh but without understanding some of the things that Ellet is talking about here. For example, he’s talking about what they call hallucinating. Um if you don’t understand, you know, how some of this works here, you might find that you’re what you’re doing in the how and the what might put you in trouble. So, you know, we this is important ground we’re covering here in terms of here you have the CEO asking chat GPT who is the CEO and it gives two different PE. Did you know either of these individuals? No. No. No. And so that’s that’s they don’t and I think that’s um I think that’s one of the things that is interesting about like understanding LLMs, right? So yeah, they are giving statistically plausible answers based on their knowledge rather than the facts. um and which is a diff very when you think about that sentence that’s a um that’s a very very very different thing um than the actual the actual facts anyway so that’s just um I just want to give that as I wanted to give that as an example um but I I see a lot of I see a lot of people really dying to dive into the actual uh um getting done I know before we do if it’s okay. It’s just worth mentioning just for a moment and I think everybody knows that and I know it’s a little bit gossipy and I know it’s you know but this is indeed like you know the next battlefield when it comes to uh when it comes to large companies and large tech um obviously I suspect you’ve all read about Google’s you know drop in you know hundred billion dollar in valuation in market cap market cap because Bard um made a mistake uh basically again, which is not not an impossible thing to happen, but um [clears throat] made a mistake. And then Oh, we have other people on the call. That’s interesting. I’m not sure what’s in there. I’m hearing whispering, but yeah, you might want to mute your mics, peeps. I am. Um and so and so that’s a so anyway just to say that’s like definitely the next battlefield. Of course Elon Musk always has to jump in as well and he says he’s going to build the best one and he also mentioned and again that’s an in super interesting topic which we’re probably not going to go deep into. Um he meant he basically blames uh chat GPT of being too woke if if nothing else. Um and so he wants to have something more I don’t know I’ll leave I’ll leave the inter right this gets into some of what we talk about in product management you know this this and for those of you who are working in data science and artificial intelligence as product managers how many times do you have the fight over narrow vertical slice over engineers and data scientist saying well

00:15:47 — Using ChatGPT Without “Delivering Crap Faster”

Dean Peters | 00:15:47–00:17:28
let’s build the whole infrastructure first let’s get it right first I I think that’s some of the the slings and arrows that Sam Alman over at the CEO of OpenAI is taking and where he’s taking the stance is how are we going to get this to work in the wild responsibly without releasing into the wild and getting getting feedback here. I I tend to lean on let’s get this out into the wild. Let’s make things happen. Let’s let’s go ahead and cut our teeth, but in the process let’s do it without making these feric victories. Let’s win these battles with you know let’s keep as I teach in my classes here let’s keep the blast radius small and so the things that is talking about is you know for example you get hallucinations so oh well Bing had mistakes in their demo too and Google had mistakes in their demo and these mistakes and let’s take a step back and think about the the the chat GPT and other types of generative AI here understand that The five years ago, no one was talking about general purpose AI. Here now we have chat GPT and we can start getting real work done with it here. But as we get real work, how do we how do we get the right thing built as to build, you know, building crap faster and how do we do it? How do we get uh the the the signals back without colluding it with a bunch of noise? I I think someone made a comment up here way earlier like, “Oh, you want to start with why? How novel. And that’s interesting. You know, as you start work, as you think about it and you sit down, let’s say you’ve plunkked down your $20 US a month to uh start playing with, you know, chat GPT plus uh and and so now you’re trying to solve for a problem. So I think let’s go to the slide here. Uh I think it’s slide. Yeah, there we go.

Amazon Working Backwards: Workflow Example

Dean Peters | 00:17:29–00:29:05
So as I was preparing for this uh and was just exploring I did this a couple months ago actually I said can I pull through the work here? Can I pull through a a actual work that a product manager does? I think what was frustrating me at the time was seeing a lot of parlor tricks around prompts. Oh, you can fill out the lean canvas. And so someone says, “Oh, here’s a prompt to do the lean canvas. So, please fill out the link canvas for this solution. Now, it was interesting because around the same time I was listening to a podcast. I think it was Lenny Rich Lenny Lenny’s podcast. I think it’s Lenny Rachky. I hope I said the name right. He was interviewing Melissa Melissa Perry. Two names you should know in product management. And Melissa Perry was talking about well, you could fill out the lean canvas, but if you fill the boxes full of junk, then what you get in the end is junk. Chat GPT is no difference. All right. It’s a It could be a It could be a huge accelerant for your work and amplification work or it could be a huge, you know, garbage in, garbage out machine for you, delivering crap faster. So, for me, I I start from when I as I found myself getting more productive here, I’d say, okay, as I try to solve for a set of problems here, let’s not just dive into the solutions. We got a lot of persons here like, well, how would I write a prompt for this? Well, how would I write a prompt for that? And it sort of reminds me what I teach my classes in in in both digital product management, agile product management, our customers, our stakeholders, our engineers, our data scientists, and other people. All they often come running to us with solutions. I would imagine you’ve had a boss run into your office, hey, hey, hey, hey, we need to add this, you know, generative AI here. Um, and just six months ago or maybe a year ago, they were the same people running as your office. Hey, hey, hey, hey, we need to do NFT and before that blockchain and before that. So you you see people putting the c, you know, the cart before the horse here. But as product people, we need to take a step back for that. We need to take what they’re they they know solutions speak. That’s the language they speak. That’s fine. It’s not because they’re dumb. It’s not because they’re hurtful. It’s not because, you know, they’re they’re stupid or anything like that. It’s the language they speak. But it’s our job to translate that and and move that conversation into the problem space for a while. So how do we do that with chat GPT? Well, think about your storyline. Think about chat GPT almost in a theatrical manner here. You know, um what is let’s give it an So that you you talk about its three stages here. It’s right on the the chat GPT. It talks about the three steps. You you you give you basically help set the context. Then you go through this this fine-tuning of train and reward and then finally you start asking it to fill in the boxes. Problem is we all jump to step three. So so uh I don’t know any thoughts there all so before I I talk about this particular uh Amazon working backwards method and how I walk through it end to end. Well, I I would say I mean just one thing to add before you jump in, I would say and again to add to your to to your points earlier. Um when you once you start playing with it and I see there’s quite a lot of people who’ve been playing around with the tool for a little bit, right? You know, and and and some people have been using it for on on a quite on a daily basis. Um you’ll get used to the notion that you kind of have to scan through what you one you have to to to your point, Dean, you have to think through what you’re actually asking. And of course, by the way, that doesn’t mean if you want to play with it just for your own recreational activity, which I do ever so often and that’s super fun anyway. Um, so I do that and that’s, you know, that’s why I saw that I showed you the the CEO example from before. But be beyond that, um, you want to ask questions in a thoughtful way that so that, you know, so the act it actually gets as much context as possible rather than, you know, be crypt cryptic about it. Of course, it it does the language model itself is quite powerful as in you know it is quite responsive. It is quite like collabor communicative in a way but you still want to give it as much content as possible. And the second thing and for me that’s one of the most important things is um you’re going to need to verify you know what’s coming out of this and you’re going to specifically need to be very careful because that’s one of the areas that I see a lot of people using this in terms of market research. um you’re going to need to be specifically careful when it comes to factchecking and and fact validation. There isn’t as far as of now there isn’t a source or validation um kind of functionality. I I think that’s coming and I know Google is adding that to Bard as part of the solution. Uh and that’s one of the biggest hurdles I would say right now especially again especially when it comes to facts. So I think when you’re using things like methodologies and you’ll see that now in Dean’s example in a minute or so that’s less I would say sensitive but if you’re trying if you know if you’re working in a company you know that is a a music app and you’re trying to do your market research on what music apps are out there you know you’ll need to be very careful with chat GPT to a degree that it might not be your best solution to use right so so I don’t think chat GPT is here to replace for now at least not to based search engine as uh as source of sources of of of curated information. Um that’s just my you know my two cents of advice when I agree. I I entirely agree. In fact, I tell people at this point two couple things. Your job is not going to be replaced with AI at this point in time. However, your job is going to be taken from people who’ve learned how to use the tool where it’s at right now. And so what a lot is talking about something really important. Think about 10 years ago, maybe a little longer, when we were all learning Ruby on Rails, even just dabbling with it, and it had this concept of scaffolding. I see as we try to get real work done in the world of product management using this as a similar scaffolding tool. And just like when you used a tool like Ruby on Rails, you gave it a data model, you gave it other certain information here, you told it about endpoints and then it generated code, but you were warned con constantly, do not move that code into production here. Take the same stance when you’re trying to pull through work and to end. You’ll notice in this scenario, I actually, you know, the worstke kept secret on the web. It’s funny you mentioned music, is my past career as an opera singer and and being in the theater for a long time. And with that, you start then with the why. And you say, look, first let’s gather our thoughts about a problem, not the solutions. Let’s first ask questions about what are some of the pain points here. So you see this session I started here and there’s actually several questions you the the Amazon working backwards I think it’s like six or seven steps depending on who you ask and I would imagine some people said I’m using Amazon working backwards give me the you know press release for um a solution like that you’re not going to get anywhere with that what I did was ask did a series of questions first like first of all do are you aware of this process I didn’t know I have train or not no I said yes I’m aware of it okay then I gave it a persona stance basically stealing you know okay take on the character imagine yourself chat GPT like being myself a senior product manager parachuted into an initiative where I’m going to be having my next two months are going to be all these conversations here can you simulate for me an Amazon working backwards method pro you walk through the steps of the process to help me shape the type of communic communications, conversations, artifacts, and and other interactions I’m going to have on this topic. And I I use so you’ll see on that that second prompt they’re saying, “Hey, I want to do this as a simulation.” And I that means I understand some of the rules are and some of that reward and response are now. Okay, we understand this. So, we’re treating this just like the fourth wall theater. We’re going to remain in character. some of the information we’re going to give you is to fill in the blanks and you’re we’re we’re leaving you to be the human and the adult in the room to figure that out. However, think about it. My first day, first two days, I’ve been in a job for a week. We’ve [clears throat] all been there as product managers. We’re parachuted in. What if we could say, look, here’s tell me about what you know about some of the problems. Tell me what you know about some of the pain points. Who is impacted by these? What are some of the things are they doing now to solve for these? So, you notice I’ve gone through a series of questions with chat GPT simply getting into the why and into the who. At that point, it’ll probably give me some solutions that are valid or not. And I don’t say give me a, you know, let’s talk about all these solutions. I’ll actually zero in and do some fine-tuning. So now we’ve taken this top of this funnel, the problem space, and we understand the who and the what, and we’ve talked about how they’re solving for it. Now, now I’m getting into the neck of this this funnel here. All right, let’s zero in on a particular problem. In this case, it was eggs being transported to a store. And I was like, okay, well, let’s now let’s do the press release on a group. And you’ll notice I didn’t say the solution that’s looking to solve this problem. So you notice that third prompt there. write me their press release from the stance of we’re about to start unpacking this problem. What it did then is as I was able to ask a few more prompts and we got I got a series of slides on these where it’s like okay now let’s talk about the FAQ. There’s where I would look for factual inaccuracies. There’s where I might look for gaps. We don’t have any further slides for that. And yeah and and then I would ask it a few more qu so as we got to each stage by the end I spent I did this on a weekend about one hour you think about one hour of investing time in a productive conversation like that that’s going to help me guide me in real conversations I’m going to need to have and you could do this you’re following the optimal product process here you’re going through hey help me start shaping my business case what type of conversations but you need to start with the why then fine-tuning and narrowing down the how and then get into the what. Just don’t jump into these here’s the prompt for filling out the business canvas for user story mapping. I have taken story mapping all the way in the user stories into creating a a CSV that I could import into Jira and I would imagine within the next few months you’re going to see Jira plugins. They’re going to help you write your user stories, but they’re going to be garbage in, garbage out if you don’t, you know, make sure the personas and the is are are well understood and the how is around the problem space, not the solution space. So, I’ll get off my my my pulpit now. Um, and my burninating questions here.

Q&A: Practicing PM Skills vs Doing PM Work

Kate Fuchs | 00:29:06–00:29:51
Thanks for that, Dean. K. Do you want to take some questions or do we want to Yeah, absolutely. Um, thank you, Dean. And, um, I think that was a really helpful just looking at the chat and through some things in the Q&A, you know, that was helpful to see sort of the progression and how you could use it to to work for you instead of, you know, against you. Um, so we do have some questions. Um, some are are in the chat for everyone and some in the Q&A. So, um, I do want to start with, um, you know, it sounds like sort of at this point, chat GPT could be used for practicing PM skills, which is sort of what you just, you know, rolled through. Um, and you did say you’ve used it to create, you know, user stories and whatnot all the way down to that level. So, what would be kind of your advice at this point um, to either use chat GPT to hone your skills as a PM or would you say, hey, jump in and and try to get it to actually do some work for you?

Dean Peters | 00:29:52–00:30:23
For me, I’ll I’ll speak, but I’m gonna give a lot some time too and talk about it because I know his whole product helps answer that question. For me, learn how to speak to the problem space first. Learn how to reframe those people running into the room, that hippo, that that zebra, that seagull manager running into the room with a great idea and reframe. And because honestly, I could show you a bunch of parlor tricks. Well, here’s how you do this form and here’s how you do Terresa Torres’s opportunity solution tree. Here’s how you do jet patent style um user story mapping, but it becomes a big garbage in garbage out machine if you’re you’re not taking time to fine-tune. This is why the API is exciting to me. Uh you know, I I I’m probably going to be feeding it user stories, blog posts, you know, things that I’ve written in the past saying in the voice of Dean Peters, write the following.

Elad Simon | 00:30:52–00:33:23
Yeah. So, I I think I mean I’m gonna not contradict you, but compliment you if it’s okay. Dean, [clears throat] that’s That’s great. I so I so chat GPT I mean again for everybody who’s played with it um it’s a fantastic tool for me it’s a fantastic learning tool um in in a way again to a degree I I think because it can give you I’m again this is my learning style and it’s not like everybody’s learning style I love learning from examples and I think one of the benefits and the abilities that I would say probably the most impressive abilities of chat GPT is to dress up a concept with an actual example. I mean, it’s one it really is it really is one of the nice and I think you’ve shown that here. I mean, but you’ve done it in a very, you know, you know, we we’ve met we met we met a while before and you know, you’re a very methodological dude and you’ve done it, you know, you’ve done it the right way and I and I commend you on that. But if you just want to figure out how does something work, you know, you’ve heard a topic or you’ve heard a thing, not necessarily to get chat GPT to do it for you, but you just want to educate yourself on it. One of the nicest thing you can do is like take your own use case, you know, whether it’s your app or your feature that you’re working on and just tell chat GPT to dress it up with whichever methodology or you know, you know, again, whatever it is that you’re trying to learn and that’s a very very powerful way of learning things. So, I’ve I’ve done this, you know, with some of the stuff that, you know, some of the things you know I’ve agree. And so I think that’s really a nice way of doing things because one of the hardest things when you read materials on the on the web is that you know you read something it might have an example but that example might be so far removed from what your life is you know if you’re working on a fintech product for you know for B2B and you know you get an example of a of a music app for consumers you know you’re just like okay I can’t rewire this into my head well and this is really a place where chat GBT goes in so so well and like just like dressing it up without the effort in you know invested in doing all that right so so I you’re absolutely right you’re absolutely right we got to we got to take some swings we got to have some fun with it too I mean I was I was stuck on an airplane on a tarmac and I’m sitting here with chat GPT almost no almost in a wouldn’t say call it therapist mode but it was sort of like okay what are some jokes I could crack with the person next to me or I was actually exploring your you know little things tiny what I call tiny acts of discovery so I actually agree with you. Look, I talked about doing the workflow. Now, could I have done the workflow that I did about a month ago back in November when the when the early adopters came out? No. I was probably like, “Hey, show me what you can do with the story map just to see its limitations.” Um what I found out real quick was scaffolding and simulation would you know it large or small but especially with tiny acts of discovery like you talked about a lot help you learn the nuances that help you do the entire workflow. So you’re absolutely right with that. Absolutely. And then I think the second use case I suspect I mean I see a lot of people in the chat mentioning this as well. I mean the second use case which I think is I would say bluntly obvious when it comes to generative AI especially in the language space is to use it for like copy stuff as in like creating creating copy and and this is and I’m I’m not I’m specifically not saying you know write your user stories or your epics because that’s not creating copy that’s very different that’s actually doing it job in a way but which by the way it can do as well but That’s that to Dean’s point. If you just parachute it into that, it’s gonna it’s going to be a you know a proxy. Do it at your own hazard without solve your problem. But if you want to create like microcopy like for a release notes or micro or sorry copy for release notes or micro copy for like a a call to action button on your app. chat GPT I mean it’s splendid for this and especially for people who are not like you know they’re not comfortable in writing you know this is really a very classic use case if you haven’t tried it again I mean for those who you on the call who haven’t tried it by all just do it I mean it’s really really powerful and of course and again I I I keep saying the same warning again and again always check it always make sure that you know you’re kind of so that it doesn’t end up like you know spit spitting out something that is completely inappropriate or completely irrelevant for for your use case But it really helps you get started when it comes to writing. I’m not a I’m not a big writer. I’m not like I’m not a very comfortable writer. I would say I like speaking more than writing I would say for sure. Um and so for me like it really helps when it comes like just writing a skeleton of a blog, you know, it’s fantastic. It really is. And I will never take that and copy it into a blog, but it’s a really good starting point and like gives you some ideas of structure and all this. Absolutely. I’m I’m all about the seven, you know, points of a story arc. Uh, in fact, where I’ve gone in as is in and been a director, a senior director, a head of product where where I have these geniuses who who who report to me and I’m certainly not the smartest person in the room, but the one area I’ve been able to help them and the one area that I I think I’ve been able to keep their career from being replaced by AI has been able to make them great storytellers. Well, chat GBT experimenting with like way a lot’s talking about is a great way to do that. I was messing around with, hey, how would I tell this? For example, let’s say you give it five user stories. You’re going to fine-tune it and say, okay, I want to write a related user story on this about the mobile app and you give it some a personality. You know, you’re a product manager. Now, you decide, let’s look at it from a slightly different angle. Hey, how would I write this in the voice of Lewis Black, the comedian do uh doing the Netflix special? How would I write it in the voice of Garrison Keeler doing you know and and so you and what it does it starts actually just like a user story map when you hear Jeff Patton talking about it first of all it helps you take those big rocks and smack smack them into you know what do you call it the boulders and smash those into big rocks smack those big rocks into smaller rocks and then ultimately smash them into the pearavevel that are user stories here um it also can help you find those gaps what oh I would I didn’t even think about presenting it from this angle here. to do that type of exploration in different voices in different again scenarios and scaffolding give it to ask it for for example what would be a simulation I’m in a brainstorming meeting I heard someone talking about in product market in one of the comments here there in product marketing why not have that conversation saying hey your your four person personalities in the form of a table give me what each you know what each person might brainstorm on this particular topic here just to prepare you for the meeting it could you walk in the meeting, you still get blindsided, but maybe the blindsiding isn’t so severe. Yep. Thank you. That’s that’s perfect.

Q&A: Confidence, Proficiency, and Measuring Accuracy

Kate Fuchs | 00:38:28–00:44:42
And I think we’ve talked a lot about, you know, examples and storytelling, and that’s something that, you know, PMS take to heart and and know that that really speaks volumes um along with our our data. So, I want to get to um the proficiency level and the confidence level of um our generative AI. So we’ve talked a lot about things that you know hey garbage in garbage out. So you know how can we um understand the proficiency level and the confidence that um we can you know accurately uh give to some of these tools. Is there any way to measure that um as as a you know recipient? Ask Yeah. Go ahead. I’ll give my two cents about it. one at the moment there really isn’t a good way it’s the it’s the shortest and most real answer that you have you can have right now there really isn’t a good way of of measure and that’s why I don’t like love to use it as a fact as a as a matter of fact one of the things I think this is obvious but I I’ll mention just like very briefly one of the things that people the reason why people love chat GPT and chat in general is that it is uh it is very decisive it gives you a single it gives you a single like one of the biggest advantage like po like quote unquote advantages it has versus search engines is that it’s definitive it gives you one answer here is it here is here it is that’s it that’s what I think however because because we can’t validate it you end up you end up being and I I’ve you know I know the limitations and I’ve known that from the beginning and it’s really easy to go in that into that trap anyway like of like you read it and oh my god that’s so persuasive that’s it I this is this this is probably True. The reality is again for facts and for like again everything to do with um like market research etc. I just wouldn’t recommend it. Um when it comes to methodologies I would say it’s accuracy is fine but in any case it’s not again there’s no way of saying oh this is like 95% accuracy etc. That’s not there yet. Nor is by and again that’s a conversation going on right now with Google as well in terms of like you know validated answer versus nonvalidated answers right. That’s again those are things that are probably coming. When are they coming? Yeah, I I I would agree. I I you know, this reminds me, you know, I I have a pretty long background in in natural language processing technologies, you know, from from the rudiments of lucine up through solar and and and elastic. And I remember one of the things that was happening was people were saying, well, put the ranking score out there for people to see. uh they found that was a horrible usability uh active usability there. Uh you you found people actually starting to argue why did this only get 86 you know ranked as 86%. Why did this get ran? And so now the conversation starts moving away from trying to shape the conversations and that’s what a lot of our work is is influence without authority to shaping solutions through greatly shaped conversations. And now the argument became well you’re scoring at this or what about this or gaming the algorithms here and I think we’re going to see that the a repeat history repeat itself here I haven’t had enough time the API has only been out what GA for about I don’t know how many hours so I don’t know if they are going to offer any sort of relevancy score but just understand we’ve all been down this path before a couple times it never worked out well u I think Elad is is you know I don’t I wouldn’t even care if it said it was 100% % sure even though I’ve asked it give it some certainty here and it’s given me u I would still fact check I would use it for scaffolding for simulations for helping me fill in the blanks for helping me to understand what I need to do next in my workflow here but if I’m looking for a high degree of accuracy right now good luck um it’s it’s not there and they they say that it’s not there in fact GPT quite often says at the bottom uh now if you’re pre-training it with user stories you’ve written and you you look at the user stories and you can generate them which I’ve done okay that now I’ve narrowed the topic down so so narrow that it has less wiggle room but in those more uh ambiguous areas yeah you you put yourself at peril if you you know that’s five minutes before a meeting you generate a few slides bang you walk into the meeting here and and then you find out that there’s a huge error or you know gaping hole in in the Sure. Yeah. Thank you. So, so really what I’m hearing and I see a lot of chatter um in in our chat um about you know relevance and how can we use this and in our day-to-day and you know what I’m hearing from you is is really making sure that we train it in the way that we want to use it. So we have to make sure that we’re giving it the you narrow down. Yeah. So you know speak to start with the why start with the problem space. Take that funnel, get narrow, get very narrow, get very tiny. Um you’ll you’ll you at least the errors will become much more conspicuous much faster. Sure. Is there any um people are asking for example of that type of storytelling or getting down to the narrow. I know we shared maybe we can go back to that slide of the Amazon one really quickly if if we have a chance um like that example like like open somewhere that you can kind of share it. I’m just kind of curious because a lot of people are asking for that I think on the chat just yeah so I know that I know that there’s if you go to um I think it’s the actually chat GPT how it works they have a page there which gives you the three major steps within those steps they give you some substeps there so and it almost I think there’s a I’ll have to find I’ll find that link real quick here yeah how it’s just but it’s you know it comes right out of their out of their page there on how it works there um and introducing. So, let me let me see if I can find that and get the link up here. But, but yeah, yeah, it’s right here on the page here about halfway down. They talk about the three major steps here. I would look at that from the the stance of that first step is problem space, you know, the why and the who. Maybe that second step starts getting into the narrowing down process and then that third step is where you start saying, “Okay, fill in the blanks. I’m I’m confident enough here.” Sure. Thank you. So, I know we’ve got about 10 minutes left. Um, and I do want to get to a very thoughtprovoking question that I’m interested to hear what you guys um say. I don’t know if we want to skip ahead a couple slides to um share [clears throat] our um ending poll as we talk through it because I want to get to that as well as the special offers that we have for both um craft.io and Productside. So um let’s go ahead and share this and we’ll as we share and you guys can answer I would love to hear both of your takes on you know considering the very rapid process and the evolution of um these tools through the years. What are your um thoughts on projecting ahead five years from now?

Poll: Where Does AI Take Product Management Next?

Kate Fuchs | 00:44:43–00:51:13
Um how do you think chat GPT or other generative AI can change product creation and growth? So, um, yeah, I’ll let me share this poll and not sure who wants to take that first, but we’re trying to look into the future. I’ve done all the talking so far. I’ll let it allow I’ll let him pick it up from here. I’ I’ve been overly Gabby, you’re you’re uh you’re launching the poll kit. Yes. Yep. [clears throat] All right. Do you want to take that? Where do you see us us going next? um whether five years is might be a little too long but where do you see the next steps? Well, I think let’s get let’s give let’s get people’s answers first. No, before I give I share my thoughts or not. We can I want to I want to get you know Sure. Yep. We’ve got we’ve got folks lots of answers coming in. Starting to slow a little bit. Yeah. Thank you for the links. Um Dean did put the the link to um a number of articles and examples for those of you looking for sort of the um you know example of how you might go through. I think I think Kate regardless I think we’ll probably share with the audience here a couple ofam like live examples on slides. I think it sounds like a lot of people are just looking for that. So I think maybe you and I can paste a couple of them on a couple of slides and just share with folks. That makes sense. Thank you. Yep. And and Dean just dropped another one, too. So, all right. Um, looks like we’re slowing down, so I’m gonna I’m gonna end our poll in three, two, one. All right. Here’s our share results. Um, so here we are. Wow. Now, see, this is where I want to have the conversations on on the no there, but a lot of you go ahead. I I’m I’m [laughter] curious. So I am not on the I think I’m I’m with I’m with the audience. I would say product management as a role is is quite a complex role for I think almost any type of AI. I mean specifically generative AI which is very you know it’s very specific format. um in the sense I mean in two senses. One eventually what product managers do and I think Dean touched upon it like very briefly before is influence. I mean that is the key skill um of product managers or the key job of a product manager. I know it sounds weird to say that because product manager also need to build things and you ship features and you know and and talk to customers and build road maps and do a lot of other things. But at the end of the day, if I had to synthesize like is rallying people around decision- making and like making sure that people get where where where where the the product is going and why. And so that skill for me is very difficult to replace with the current AI you know that exists in the world. I mean I’m ignoring really ignoring and I I hear some people here saying and I I agree. I mean of course this technology will evolve and I I have no doubt about it. it will evolve. It’ll become better, more accurate. I don’t know by the way if it will get to be a live technology and I again if you if you learn a little bit about like LLMs and their limitations like the at least currently the computing power needed to keep that model as live learning um is very complex um and and also difficult again from a like you know you might end up with very weird results if you go with unstructured data which is what the web is when it starts And so that really is a tough tough nut to crack and it’s not by coincidence a coincidence and I think the fact that Bing is doing now a mix of live and like curated results is actually search plus open AI kind of results. I don’t think it’s like I don’t think they’re actually updating that. So anyway, long story short, I think that in the foreseeable future it’s not going to be a replacement in any shape or form. Um, by the way, there’s I would say it’s going to become a big aid. I think it’s going to become I think probably going to become one of the biggest aids um to uh to product managers. Um and so definitely a space where um if I were you as in you the the audience here, I would definitely invest the time in learning and using it as much as possible. um not because not just because to be trendy and to be you know you know like because it’s cool etc but I really think um that it’s uh it’s a space that can really assist a lot of people in doing their work. Um again if you use you know use it in the right way rather than in the you know in the hasty way. Um, so I agree with the audience. It sounds like a lot of people are either optimistic about it about the future as humans or uh if not if not that at least um I don’t believe that’s going to change. I don’t know Dean what your thoughts are about that. Oh, I agree. It look more than likely your job is not going to be replaced by the technology. It’s going to be replaced by that person who learns how to use this technology better than you. So if you stick to building trust, you know, influence without authority is about trust. So if you think about building trust with your partners, your stakeholders, so forth with storytelling, with study, with simulation, with scaffolding, with shaping around the problem space, you’re going to do just fine. And you’re going to find that a tools such as chat GPT and others are going to help you augment your work. But if you’re all you are is a jury slinging ticket monkey. If people are just pushing features down and all you’re doing is slinging tickets, you might want to think about changing your game a bit.

Special Offers: Craft.io and Productside

Kate Fuchs | 00:51:14–00:52:53
Yes, I agree. And um and okay, I think uh with that we have last five minutes. Um should we go to the next slide? Yeah, absolutely. Let’s do it. Yes. So, we’ve got some very exciting special offers for both of our groups. So, please check these out. Um, Craft.io has 20% off on all plans. So, um, I don’t know, Ilad, if you want to speak to that for a minute or not. Yeah, I mean, basically, I mean, again, as part of this, uh, special webinar, we’re offering 20% off of all the plans we have on craft.io. Um, so, uh, we will I think we’ll share the link um, on Yes, it’s she’s got it in there. Yeah, there you go. So, we have we have the link uh, on the chat. So, uh, we have a landing a special landing page for that. So, if you want to take advantage of that, just go ahead and and and of course, you know, feel free to look around craft.io on our on our website and and see what it can help you with. We’re a product management platform helping product teams build product with confidence. So, that’s on our end. Uh, hopefully that people find that useful. And Dean, on your side or Kate? Yeah, absolutely. So, 20% off um all inerson and live online courses during the month of March. So, um this is uh product pickum [laughter] is the code and we would love to have you in our courses. Um learn more from folks like Dean or Dean Self and we um we’re excited to see you in our classes. Um yeah, perfect.

Closing Remarks

Kate Fuchs | 00:52:54–00:54:10
Well, thank you for joining. Um Dean or IA, do you have any last uh final words to to part with? [snorts] Yeah. Very good. Stay in touch. Let us know your experiments. It would be great to start like I mean with the folks on the call it would be great to start a thread. We we’ll share some thoughts on a on a doc or a slide. But absolutely if we can make this a community thing and everybody can share their own like prompts or ideas of how to use it. You know we can make this a nice a nice collaborative environment for everybody to kind of share thoughts and this amazing tool. Love it. Well, thank you both for your insight and your passion on this topic and I appreciate all of the folks who joined us today. Um, it is certainly an interesting and an emerging [laughter] um topic. So, thank you so much and have a wonderful rest