Transcript 0:00 The big laugh in the room was that women were outperforming men in human and machine collaboration in terms of, on a lot of different measures, but they were underrating their performance. Mm-hmm. 0:12 And men were overrating their performance. Mm-hmm. [upbeat music] We have a fabulous guest today on Women Splaining AI. Welcome, Jo. 0:30 Thank you. Lovely to be here. Jo McKinney is a boss. [laughs] This woman is incredible. She is the co-founder and president of Corix. Um, she spent 30 years in the agency world. 0:46 She's CE- she was CEO of Burns Group, working with clients like P&G, Coca-Cola, Pfizer, and honestly, she really should be co-hosting this podcast. 0:57 [laughs] Like, I think I'm just gonna step down and I'm just gonna hand the mantle over to Jo. Okay, just for today. Because- Just for today... 1:03 she is so, she is so spot on in terms of all of the things that we talk about here on Women Splaining AI, around how to maintain your judgment and your center going into this- Mm-hmm... new AI era. Welcome. Thank you. 1:18 And we're so excited to hear about your journey from, from, you know, the heights of, um, of leadership to, in all different disciplines, to now the heights of leadership in this AI, uh, collaboration space. Yeah. 1:34 Um, and I do have to just give myself a humble brag here that I think everyone should go out there and take the AI collaboration assessment on the Corix website. Thank you. Um, which I aced. Oh, awesome. 1:45 Ooh, yeah, you did. A+ baby. And you would have no idea if I didn't. [laughs] I could have just said that I did. That's, well, we could go back and check. [laughs] No, the results are anonymized, for sure. I know it is. 1:59 It is. It is. Jo, welcome to the podcast. We're so happy to have you here. Thank you for inviting me. 2:05 I love the whole idea of this podcast and- You, you were on the stage at the [laughs] at the, at an AI powered women conference at MIT a few weeks ago, and I, in the what, 11 minutes that you were talking, I immediately- [laughs]... 2:18 said to myself, "I am going to get Jo on the podcast," and here we are, so... [laughs] It was a very short keynote, for sure. [laughs] Your research presentation genuinely really had the room kind of, like, fired up. 2:35 What do you think really landed? And if you can give a little sneak peek and then we'll dive deeper, but what do you think landed with the audience in that moment? 2:41 If you can kind of bring our listeners into, into the room with you. Yeah, and I will say, like, I've, I've spent the last year, um, doing a lot of that kind of rabble-rousing, evangelizing on stages- [laughs]... 2:54 because I feel like we're in just a really critical moment where, um, there's an entirely new workplace forming. Mm-hmm. And there's a lot of people who somehow have deemed themselves underqualified to be- Yeah... 3:09 leading in that workplace, which is making the leadership of this design of the next step, um, more narrow than I think it really should be. 3:18 And I, so I, I personally am just really concerned that, uh, this new workplace be designed by the most diverse set of voices we can get out there. 3:26 Right now, this tech kn- this, you know, technology evolution, revolution that we're going through feels very directed by a very narrow set of people. [laughs] Um- Yes... 3:37 and um, so my goal was to actually rabble rouse and evangelize- [laughs]... to get people to realize that they have what it takes to actually lead right now. 3:47 Um, because being human is the thing [laughs] that we need to bring to this technology. 3:54 So I think the moments that really land for people when they start looking at this is, um, the fact that we're kinda myth busting a lot of the assumptions around- Mm-hmm... 4:04 who is actually getting good quality work out of AI. Mm-hmm. Not who's using it. You know, a lot of- Okay... 4:10 the research to date or a lot of the things that you see, um, you know, quoted in, on LinkedIn or in articles or in the news is really about adoption, um, and usage of AI, not about the quality of the work. 4:23 And where are- Yeah... the humans and the machines [laughs] getting to outcomes that perhaps neither of them could get to alone, right? That's really the big question in this new workplace. 4:33 Like, so I think being able to land research that basically says every assumption you might be making, uh, in your organization or in your own work about who actually has the capability to collaborate with a machine well, um- Sure... 4:47 aren't true. 4:47 [laughs] And so when we stand up and we say, "It's not this person, it's not that person, it's this person," it's actually this behavior, and this is a behavior that, to this group of women in the room, um, women seem to be doing more frequently than men, I think is, is really interesting. 5:03 Also, I think the big, the big laugh in the room was that, um, women, um, were outperforming men in human and machine collaboration in terms of, on a lot of different measures, but they were underrating their performance. 5:18 Mm-hmm. And men were overrating their performance. Mm-hmm. And you know, obviously this is, this is an issue overall, um- Mm-hmm... 5:26 in terms of self-rating versus actual ability, and I think women get very activated when they realize that, um, they potentially are not raising their hands for something because they are underrating their abilities. 5:41 And the men in the workplace might be doing the opposite. They might be raising their hand- Yeah... and overrating their abilities, and that seemed... I don't know, Logan, that seemed to get quite a reaction in the room. 5:51 Well, it's funny 'cause in the last, like, five minutes of our, of a episode that Maura and I did last week, I referenced that same thing, and I, I posed it to Maura as, like, "Which do you think, you know, did, did, which group do you think underestimated themselves- Yeah. 6:03 [laughs]... and overestimated themselves?" Yeah. And Maura was like, "Well, duh, like, the women underestimated them." She was like- [laughs]... "Is this a trick question? I think this is-" Is this a trick question? 6:10 Yeah. Yeah. [laughs] It feels like you're identifying something that is a universal-... 6:15 truth, you know, but it's, it's, it's, it's once you bring it out in the open, and I think especially applied to this, to this very fraught moment where people- Yeah... 6:23 feel like their, their whole identity and, and, and expertise and value is being questioned, it feels like it's even more important to acknowledge, and it's like a collective... 6:33 It was a laugh, but in my mind in that room- Mm... it felt like a collective sigh of being seen. Yeah. I think, um, if, if we don't recognize that this is a, 6:44 you know, pattern, um, in the workplace, we are potentially, um, making, making companies miss contributors in, in terms of- Mm... putting people into roles to make decisions, right? 6:57 Like, so this all is so related to who gets to decide, who gets to design, who is trying to figure out where human work and where machine work are gonna live side by side in a workplace. 7:08 If women are underestimating their own abilities, um, in that area, they're therefore probably not raising their hands to lead. Um- Yeah... 7:16 and therefore we're perpetuating this earlier problem we just discussed [laughs]... which is the narrow set of voices designing the future. 7:23 So I, you know, I hope it's a really motivating piece of information for people, not just a like, uh, "Oh yeah, of course that's always the case- Mm-hmm... but hey, what should I do about that? 7:34 Like, how do I actually see myself as a leader in this space?" Um, I just wanted... I w- wondered if you could dig in more on the gender lens and- Yeah... 7:41 specifically how you saw kind of the quality distinctions in terms of, uh, the outputs that, um, men and women were producing. And this- Sure... 7:51 your research helped to validate something that's been kind of a hypothesis for me for a while, which is, like, the combination of two different, um, research pieces that I've read before, and one is, and we talk about this one all the time on the, on the pod, which is Thinking Fast, Slow, and AI. 8:06 Mm-hmm. Right? Which is the- Thinking Fast, Slow, and Artificial. [laughs] Thinking Fast, Slow, and Artificial. Yeah. 8:11 [laughs] Um, which was the paper that talked about artificial intelligence being the third kind of, what is it, mode or something of- Mm... of thinking, and how the... 8:21 One of the conclusions to that research is that the people who are going to win in this space are people who are more skeptical of the r- AI's results. Yes. Right? Yes. 8:31 So that was, like, point A, and then point B was King Co- King's College London put out research earlier this year that showed that, um, between women and men college students, women thought that AI was making... 8:44 The women college students thought A- AI was making us dumber- [laughs]... and men thought it was making us smarter. And so I kind of felt like there, there- Yeah... 8:51 is a hypothesis that landed in my head that was like, I think these two things are conflating in a way that's really positive in terms of women's leadership skills, but I don't have proof, and then you walk in. Yeah. 9:01 [laughs] And so... And I think that's what's so interesting about what we were trying to study at Corax, right? 9:06 We're, we're measuring, um, the, uh, different, different aspects of the collaboration between human intelligence and machine intelligence, right? 9:14 So, you know, the whole point of the research is, um, to figure out what the person added. So we compare their work at each turn with a simulation of what AI would have produced without them. So as... 9:26 And it's a self-selected conversation, so you're in a conversation with AI, and for every turn you take, we're also having AI collaborate with AI to see- Yeah... where that would've gone, right? 9:37 So we look at whether the person stayed engaged, whether they kept thinking for their self, whether they maintained their judgment, um, as the work continued. And, um, interesting... 9:47 First of all, I would say people's self-assessment, um, they can't self-assess their ability w- uh, in collaborating with AI, period. So there was... 9:55 Forget the gender side of it to begin with, there was absolutely zero relationship between how people think they are at AI- [laughs]... and, and what they did. 10:02 So that just means in general, like, no one really has a clue, right? Go ahead. [laughs] Well, I'm just so curious. Do you think... What do you think the reason for that is? 10:11 Is there something about working with machines that, that causes us... I mean, that, that paper you referenced, Mara, is Thinking Fast, Slow- Yeah... and Artificial- Yeah... 10:18 is all around cognitive surrender and kind of the, the, uh, false confidence that you get with working with AI. I wonder if people are good at self-assessing their own work, period. I, you know, I, I don't know. Mm. 10:29 And I don't know the answer to that, Logan. I think maybe people just aren't very self-aware, and I don't think- Yeah... we all know what it... 10:36 I don't think we've given deep thought about what does it take to get good quality work out of a collaborator that's a machine. Okay. 10:42 We have spent the last three years passing out tools to a lot of people in the workplace, asking them- Mm... to just experiment, use- Yeah. Mm... um, create their own use cases. 10:52 That's not the same thing as really understanding the difference between human intelligence- Yep... and machine intelligence, and figuring out where those two types of intelligence come together to get to quality work. 11:02 Yeah. And this is, the way it's been rolled out hasn't been around, um, what does it take to collaborate with a machine? The training has been around prompts and, you know- Right. Right... 11:13 how do you use tools, which is not the same thing about thinking about a new form of intelligence, to your point, Mara. Like, um- Mm... this is a different type of intelligence. 11:21 It doesn't work the same way as collaborating with a human. I think people will, and as you said- Mm... 11:27 there's a lot of really interesting research [laughs] coming up now from lots of different angles on social sciences, you know, psychology, uh, cognition. 11:35 Uh, I read a, I read a really interesting, uh, research thing this morning from the MIT Media Lab- Okay... that found that ChatGPT users showed weaker neural connectivity, and 83% can't recall their own work, right? 11:49 Um, like, you're like, "Okay, so, like, we really need to be studying [laughs] this because something, something really interesting's going on in many ways." Yeah. I've read the research that you've- Yeah... 12:00 that you've sent, and we'll share it in the show notes and, and link everything, but what really surprised you? 12:06 I think the general myth-busting that, like, years of experience of working with AI, having a technical background- Mm. Yeah... um, frequency of use of AI were not correlated with the performance With AI, right? 12:20 So, and I think actually that's part of the disqualifying piece, right? Like, if I haven't worked in AI for a million years, I'm behind. Yep. Or I don't have an IT background. Mm-hmm. I can't do this. 12:29 You know, basically the research says th- none of those things predict how well people collaborate- Yep... with AI. So, and we also found, like, it's really an open and level playing field. 12:39 Like, there weren't a lot of, like, demographic or racial or, [laughs] you know, uh, a- there aren't a lot of things that made people better qualified. It wasn't education level. It's [laughs]... 12:51 So, y- you think about it, if this is the new workplace that's forming, it's a more level playing field, right? Which then begs the question, you know, so what does it take to get good work out of there? Mm-hmm. 13:00 And then I think the other piece that got really interesting is, like, what was at the root of getting good results out of AI, you know? 13:07 So we looked at a variety of different things, from people's resilience, like how long [laughs] they stay in it versus- Yeah... giving up. 13:15 You know, we all are used to getting very polished-looking output fi- from AI- Mm... that's very confident and looks really good, and so you can easily accept that. 13:24 But, um, the people who accept that output versus challenge it and question it and keep questioning and challenging as things go along- Mm-hmm... are the ones that get good quality work out of AI. 13:36 So, um, if you bring an efficiency orientation to the AI relationship, meaning you're just looking for a quick answer, you're not getting better quality work, right? Mm-hmm. 13:46 Um, which I think is encouraging in terms of, like, are these behaviors we can learn? Can we learn to challenge AI? Can we learn to- Yeah... keep questioning and keep our resilience up? 13:58 Can we protect our human judgment and figure out how to bring it? Um, you know, all of these things, I think, in the research were really, really, um, revealing in many ways. 14:08 Gives you a, a real more open-ended sense of who's qualified to work- Yeah... in this space, meaning everyone could be. 14:17 I'm, I'm so excited that you're bringing this to the table, because I, a lot of the work that I do is with women, um, who aren't sure how to enter the AI- Right... space, right? 14:30 And the, some of the objections that I hear are, "I'm not techy enough." Right. Right? Like, "I can't possibly keep up." I'm late to the party. [laughs] I'm late to the party. Yeah. I can't possibly keep up. 14:41 Um, and this is exactly what you're identifying, right? Right. And saying actually that those things aren't, um, aren't, like, legit... not legitimate concerns, but they're not definitional- They're not predictors-... 14:57 in terms of your engagement... of better quality work with AI. And, and- Right... 15:00 the other piece I would say to women who think perhaps they're under-qualified is, based on our study, women were bringing behaviors to the AI collaboration instinctively that are the right kind of behaviors. 15:13 So women- Yeah... were outperforming men on average, um, because of really a couple of big differences in behavior. 15:21 Um, they challenge AI more often, [laughs] and that, and that gets back to the skepticism point and, you know- Mm... um, maybe thinking about the third and fourth order consequences. 15:31 Um, and they, they're, they don't decay in their behavior throughout the session. Mm-hmm. So men and women start out very evenly in the way that they're collaborating with AI. But women keep, 15:43 keep the pressure and keep the challenge behavior longer. Yeah. And men tend to drop off, um, and accept a response sooner. And so those two things women are, seem to be doing instinctively more than men. 15:56 It doesn't mean that everyone can't do that. Like, I, I don't- Sure... point it out to say that women are just always gonna be better at collaborating with AI. 16:02 I don't think that's the point, but that some of these instinctual behaviors that women are bringing to AI are the behaviors that are the right behaviors for getting good quality outcomes with, with a machine. 16:13 And so we should be saying to women who feel like, "I'm late to the party"- Yeah... or, "I'm not qualified," you could say, "You probably already [laughs] have what it takes to be- Exactly... 16:21 the right kind of machine collaborator." And combined with the fact a lot of the women in my network, um, you know, I'm, I'm talking to people who have really deep knowledge of their own area of work. 16:31 And that's the other- Right... really big advantage- Yeah... in working with AI, is your deep domain- Yeah... expertise or functional- Yeah... knowledge. 16:40 And the truth is, AI is trained on a lot of information, but not on the subject matter specificity that we all know- Mm-hmm... from our own work. And that's the difference-maker in getting to good outcomes with AI. 16:52 So when you look at, from our research, where, where humans add value in the process, 'cause that's really what we're measuring with Corix, we're adding value by bringing information AI doesn't have. And that- Yeah... 17:04 information, um, and that improves outcomes more than anything in our study, right? 17:08 [laughs] So that, that could be, you know, you've worked in a field for a long time and you know something doesn't work, even though [laughs] it should, or you, you understand the politics of the organization that you're in, or you understand the category in a way that AI couldn't. 17:24 Um, and that type of- Or you have context, or you have context that AI- You have context... can't possibly have... you have judgment, you have deep knowledge. 17:31 And the truth is, if you're a beginner and you're thinking about how do I design the future of work for my area of expertise- Mm-hmm... it's that knowledge that tells you where machines belong and where humans belong. 17:43 It's nothing else. Yeah. Yeah. So it's not that an, it's an IT person who's worked in tech for a long time. It's someone who has deep domain experience. 17:51 So, like, one of the big arguments that I think our research supports, and is an argument that I feel like I've been making in my own work prior to this research, is that 17:59 the design of new workflows, like applying AI to existing ways of working is not the unlock, right? It's actually- Mm... reimagining work with two forms of intelligence- Yeah... [laughs] in the workplace. Yeah. Yeah. 18:09 And the act of designing that in your area of expertise should only be done by people who are experts at that kind of work and know what good human work looks like. 18:17 So if I'm spent my career working in marketing- Someone like me should be designing what does the future of marketing [laughs] look like- Yeah, yeah, yeah, yeah... 18:24 not someone with a technology background, because only I know what works, what doesn't work. I have that context- Totally... I have all that thing. 18:30 So I think the research really, really supports that assumption, because it shows that, um, 18:36 that we- you're able to actually improve the outcome of AI materially, uh, with these type of messages that you give it, which are that kind of very specific information, and they improve the outcome, I think the data said, by six points. 18:51 Um, so th- you know, the- these are, like, one of the most consequential things you can do to improve your work with AI, is to bring your knowledge from your specific role. Love it. Yeah. 19:01 I think, and I agree so much with that in terms of, you know, all the research showing domain expertise, and I think- Yeah... it's a very empowering message. I think- Yeah... 19:08 I think when I talk to people, they kind of intrinsically get that they know more. Maybe just to, to double-click, although I hate that phrase, it feels so corporate, and I hate that I just said it- [laughs]... 19:21 and to absorb- You could delete it. I'm not gonna delete it. Logan, Logan, I'm just gonna say double-click to you- I'm calling that out publicly... all the time now just to piss you off. 19:28 [laughs] No, I will not delete it because I hate it. Logan, I'm double-clicking on that. I'm not gonna double-click on anything. I would like you to say more- [laughs]... Jo, on, on the, the pushback. 19:39 Do you know what I mean? Where it's like- Mm-hmm... for me, I'm pushing back and I'm being skeptical with Claude, and for me, what that looks like- Yeah... is me saying to Claude, "What the fuck are you talking about?" 19:46 [laughs] Like, and, and me- Yes... literally cursing at my LLM. Yeah. Like, it's [laughs] like it, like really rudely. I would never talk to a person like this, but- [laughs]... 19:56 w- but cursing at Claude is an opportunity for learning. And that- Yeah... is a, is a little bit [laughs] of a disconnect for people ent- Yeah... 20:05 'cause I think a lot of people just entering this think it's about prompt engineering, and let me really plan- No... the, the, the ways in which I want to approach a session. 20:12 But what you're saying is that it's actually the good work comes out and the judgment is expressed over the course of that session, meaning over the course of those interactions with the LLM. Yes. Not giving up on that. 20:24 Like, like, look- Hmm... I think people can, obviously, the opening context that you share is, is really critical. I think- Mm-hmm... 20:30 the way people open up, if you just give AI a task, you're not gonna get a good quality output. Sure. Y- you have to actually know what problem you're solving, you need to frame it. 20:39 Um, y- I, I, me- the context that you give upfront can be a huge contributor. Um, and it's one of the weakest behaviors that we measured, is- Yeah... how people articulate the problem or the opportunity. 20:51 That's fantastic. But the, um, but the challenge behavior is, is so interesting, 'cause I think we all may be... It, look, I, I, I... Jeff and I have been talking about this, and I'm curious to get your takes on this. 21:02 I feel like there is two different types of interactions that you have with AI. One where you have knowledge and judgment, where you are leading and you're able- Mm-hmm... to challenge, right? 21:11 You, you know, you see a response, and Logan, you wanna curse at it 'cause you're like, "That's crazy. That's not right." And you know it because you, you're working in a, in an area that you know well, right? 21:20 And therefore- Yeah... you challenge, and you challenge, and you challenge, and you get to the output that you want. 21:25 But there's a lot of instances where you're working with AI in subject matters that you have no, no expertise. Mm-hmm. Sure. Like I, I, you know, put a legal document into AI, but I'm not a lawyer. 21:34 So what I get back maybe looks really good, and I have no way to challenge it, right? Mm-hmm. 21:40 So one of the things that we've been advocating for, and, um, I recently put a whole post on LinkedIn about this, that has some very specific ideas of what to do when you don't have subject matter expertise- Mm-hmm... 21:51 and you need to challenge AI. 'Cause you still have to keep- Yeah... that challenge behavior up, and there's a lot of behaviors you can use to do that, right? You can, you can ask AI to show you its sources. 22:04 You can ask it to critique its own work. I often take two different LLMs and play them against each other. I do too. I'll take Claude's response- Yeah... and give it to ChatGPT, and vice versa. I'll at... 22:13 Or even within Claude, let's say, asking Claude to critique its own work, um, tell me what an expert in the field would find wrong with their response. It's amazing- Ooh, that's a good one... 22:23 how they identify their own flaws in the response. Like, this is the thing, like if you're an expert and you're challenging AI, you know what to challenge. 22:30 If you don't, but you ask Claude to do it for you, it challenges- Yeah... its own work and points out its own flaws. Yeah. I mean, so- Jo, I, I, I put my cl- AI collaboration assessment into my Claude chief of staff. 22:42 Yes. And it was like, "Yeah, I don't really believe this one. I don't think that this is quite right." And I'm like, "Oh, so you're arguing with the AI's assessment of the AI's-" Yeah. [laughs]... assess- You know? 22:54 [laughs] It's like so meta. Right. I was just laughing. I was like, "Oh, you think you're smarter than Corix apparently." [laughs] That's so funny. Well, but I think it's interesting. 23:02 And then I think the other piece is, like, asking the AI what, what, uh, what's at stake if this is wrong. You know? Mm-hmm. When you're dealing in an area, so- Oh, I like that. Yeah. Mm-hmm... 23:11 and, and then course correcting route. 23:12 But in our research, you know, the things that you can do to get to better outcomes with AI are setting up the context and the problem first, like saying what you want to achieve, putting the guardrails around that. 23:24 O- one thing that we've really learned is that, um, it's not just telling AI what you don't like, like, "I don't want contrast framing. I don't want- Mm-hmm. Sure... am dashes," whatever it is. 23:33 It's showing examples of good work. Yeah. So for instance... And it might be good pre-AI work. Like it could be that you say- Yeah... "This is a great example of a research report, and this is the..." 23:45 So giving it models of what good looks like. And that might be from you, your best writing samples. It might be voice notes that really indicate who you are and what your point of view is, whatever it might be. 23:55 Um, the second thing is bringing only what you know. We know that people's human value in the collaboration comes very specifically from their knowledge. 24:04 So when you're writing or in, in conversation with AI, making sure that what your contribution is, is something that only you know. You telling AI what it already knows doesn't help anything. 24:14 But really looking deeply at your own knowledge and critiquing your own work to say, "Am I bringing enough- Mm-hmm... of what I know here?" Right? Um, the resilience, keep bringing it. 24:24 Like- Don't stop challenging 'cause you're at the fourth or fifth turn. And Logan, I'm sure this is where you start cursing at AI- [laughs]... 'cause you're like, "Oh, it's so frustrating." Um, 'cause it's a lot of work. 24:32 Like, it's not like- Yeah. Yeah... I, I mean, if you use AI to just crank out crap, that's work slop. I mean, that's why- Right. Right... 24:38 people don't wanna read your LinkedIn posts and that's why people don't wanna read the summary that you sent them in Slack. [laughs] It's 'cause you're not doing that behavior, right? Mm-hmm. 24:47 Um, and, and Mary, to your point- Yeah... 24:49 I think one interesting thing when I did the Quorix assessment for the first time, I took the output and I put it back into Claude, and now Claude doesn't let me get away with some of the behaviors that were- Yeah... 24:59 wrong for me, and it- It was, it was helpful in that way... it forces me- To like strengthen, help, help, help strengthen kind of the adversarial nature- Yeah... of our thinking relationship. Yeah. Yeah. Totally. 25:11 I wonder, 'cause work slop is a very hot topic [laughs] for many people- Yeah. Yeah... right now, and, and just 'cause it's so... It's tiring to read a lot of- Uh-huh... AI crap, right? You know, you get- Yeah... 25:19 some, you get a report from somebody and you're like, "This is eight pages. Like, I might as well have just asked ChatGPT [laughs] my, myself." Right. Like, I don't need you here. 25:26 What does the opposite of work slop look like right now? Mm. 25:30 Well, I think first of all, anyone who's using AI to quickly get to stuff or to synthesize information, um, and then they're sharing that information but they haven't read that work themselves or taken... 25:43 They, they're wasting their colleague's time. You know? Yeah. Like, the truth is there's studies that show that people who receive work slop rate the sender, [laughs] 25:52 um, like their, their assessment of them goes way down in terms of trust and intelligence and all that sort of stuff. Yeah. 25:58 So you're not doing yourself any favors by sending people stuff that you haven't bothered- Yeah... to review- Right... or work. And I think everyone needs to kinda just check their own behavior and say, like- Mm-hmm. 26:08 Yeah... that it's just lazy. I mean- Yeah... I mean, truthfully. Like, have you... Pre-AI, you checked your work before you shared it with other people. [laughs] Right. Like, you did. 26:17 Just 'cause this looks good doesn't mean it is good, and I think- Right. Right... um, and, and the last thing I- I, I remember being like in graduate school- Yeah... 26:25 it's 2:00 in the morning, and I'm going through like with the, with the m- with the spelling- Yeah... you know, the spelling checker. Yeah. Like, looking at every single word. I would... 26:35 I'm literally falling asleep while I'm doing it. Right. But I'm doing it. Right. But that's like, that, those behaviors are stopping, right? And so people, first of all- Yeah... 26:42 they're generating a lot of work that they don't understand or own, right? Yeah. So their ability to even... 26:48 And you can see this in people where like maybe you go to a presentation and someone starts presenting and you realize they have no idea what's on their slides. You know? Like- [laughs]... 26:57 it's, like it's AI generated stuff and th- they're really struggling with that, right? So- It's-... I think there's a whole quality of work issue, and it's gonna really make no one wanna work in [laughs] 27:08 you know, in any capacity. [laughs] Well, okay. I wanna... This is where I wanna get- I don't wanna work in any capacity anymore. I wanna, I wanna- I'm done. We're done. We're done here. [laughs] No, we're, we're good. 27:21 But, but like when... 'Cause we started this conversation with, with kind of the, the future of work in the workplace- Yeah... is changing. 27:27 I think if I'm optimistic, I see a, a kind of opportunity for really, really high caliber, interesting work at the frontier for people who have nailed this kind of human machine collaboration- Yeah... 27:39 where then it's you're actually operating on another level, right? And then you have- Mm... this, this other end of the spectrum where it's work slop and it just degrades trust, and all of a sudden I don't wanna... 27:50 I don't care what Janet from accounting... [laughs] What does the future of work look like, I guess? 27:55 Well, look, and I would say too, right now most of what you talk about is in human machine collaboration at work today is more generative AI, right? Right. It's less agentic- Right... systems, right? 28:05 So AI is gonna start handling more and more multi-step work, right? Mm. And, and in that, um, how people contribute is gonna change [laughs] because if there's more- Yeah... 28:17 um, autonomous work happening, it, a lot of what we're talking about here is, is a whole new level of, of issues, right? 28:23 So I think truthfully, human judgment is gonna matter, and it's gonna matter much more than just what is the opening prompt. [laughs] It's gonna matter- Mm... in the design of these systems overall, right? 28:36 So agentic systems are gonna run across more steps, so we need to decide where people are, are in that system and where they review. 28:44 I think our humanness and what makes us uniquely human and our human knowledge is gonna become more and more valuable, because it's gonna be the only thing that LLMs don't have, [laughs] right? Mm. 28:54 So that piece is really gonna matter. Um, newer workers are gonna, um, have a real crisis. They're gonna need, we're gonna need to build in ways for them to learn judgment- Mm... 29:06 because we're taking away the grunt work for them that taught us- Yeah... all judgment. Yeah. 29:10 So I think we're gonna have to keep learning tasks intact in some way, and I don't know what the answer to that is, but I, I may almost radically think that you shouldn't be allowed to generate with AI, um, until you reach a certain level of knowledge and judgment in that particular area. 29:28 Mm. So maybe you could be learning with AI, but you couldn't, you shouldn't be generating work- Yeah. Yeah... because you have no ability to judge it. So I think we're going to need to really figure that out. 29:37 And then I think, I don't know, have you guys read this, um, this piece by Rishad Tobaccowala about the future of work? Um- No. It's really interesting. But I love a reading assignment. Please. 29:48 [laughs] Oh, it's a, a really interesting Substack. Um, but he talks about work being completely different, and like being organized around tasks and outcomes and smaller, more flexible organizations. 30:00 So learning- Okay... across your whole career, less tied to one company, less tied to- Yeah. Okay... one work chart, right? I've heard theories around this. Yeah. Yeah. Which is like less boundaries in how we work. 30:09 So I think when you think about the future of work, you also as an individual have to be thinking about, what is my learning mode right now? 30:16 Like, one of the things that I've done professionally, and you know, I, I, I left my role as a CEO three years ago with the, with the specific intent of being a practitioner. I f- I saw my life's- Yeah... work changing. 30:27 Everything I had worked on in my whole career, everything I was considered an expert at was gonna be changing, and I felt like only people with real deep expertise should be on the front lines of that change, and I really wanted to be in practice. 30:38 Yeah. And when you're a CEO, you're not in practice. You know? [laughs] You're not- Right. Right... an individual contributor. Mm. 30:43 And I, I felt like, what could I do over the next couple years that would put me in deep, deep practice? And that's why I've been working fractionally across a lot of different things. 30:50 And, and it's made my learning and my ability to collaborate with AI, AI, and, and start to understand where AI really fits in my work much clearer to me- Mm... than if I had just been in a leadership position. 31:01 And by the way, the Quorix research says this too, that individual contributors get far more out of AI collaboration than managers. Um- Yeah. Yeah... because you're staying in the work. 31:11 And when you're in the work and you're learning how to collaborate with machine, it's really important that you know the work and you know the work really well versus you're managing it from a distance. 31:18 So, um, I think we're all gonna have to be thinking about, how do we learn? How do we lead? What is the cycle? 31:25 And what are we putting in our daily, daily work that's allowing us to experiment, fail, not worry so much about achievement and expertise, see ourselves as beginners? And, and I think this is a big mind shift for work. 31:39 Like, I don't think- Yeah... work is designed around celebrating a beginner's mindset or celebrating experimentation and failure. Work is h- traditionally designed around being rewarded for being an expert, being- Yep. 31:52 Yep... being paid for being an expert. And I don't think- Yep... there's anything called expertise on AI right now. Like- No... you know, anyone who calls themself an expert I, I'm very skeptical of. Yeah. 32:03 And I, I think s- I think women super suffer from this, is like I call it Hillary Clinton-ing a situation, where you like, you know, work yourself up into a level of expertise that you're kind of like divorced from your own authenticity- Yeah... 32:17 because it's just so- Oh, that's such a good point... the risk of failure is just so high. Yeah. 32:22 Um, and so given that, within this amorphous context, I think it's that much more important to be inviting women in right now. And so I wonder, 32:31 you, you say that Quorix's strategy is to shift to a more representative set of voices. Like, what is Quorix's theory of change- Hm... 32:38 in terms of how change happens to move to a place where we invite different voices in and they feel empowered to have a say in what happens next? Sure. 32:52 Oh, I say in general at Quorix we feel like companies right now are measuring the wrong things when they think about the role of AI in the organization. 32:58 And what we're trying to say is skill and ability in collaborating with a machine is measurable, it's growable, and it's something anyone can do. 33:09 That's a much more inclusive statement right off the bat, because if you think about, um, thinking that you have to have a tech background or you have to have years working with AI- Yeah... 33:16 oh, right off the bat you've just eliminated a huge percent of the population [laughs] from- Right... contributing to AI strategy. But if you say- Right... 33:23 actually the ability to work with a machine is something that we can measure and something that we can grow... 33:29 And actually we have an enter- enterprise dashboard that shows with anonymous data across an organization what does human-machine collaboration look like within the company. Um- Mm-hmm... 33:38 what type of tasks are happening, who's leading it? Are there certain departments that are doing more or less work? Yeah. What should be, what should be automated and obsoleted? 33:47 Because clearly people are doing too much of it. You know, all of that kind of work. Because I think if the organization is transforming, we need to be able to see the nature of the work that's happening- Yeah... 33:56 between humans and machines, and we haven't been able to. And I, and I think that's why we're committed to this. 34:01 We're sort of saying like, if we can, if we can reveal what has been hidden, um, over the last couple of years and give people a chance to grow their capability... 34:11 And I would say one of our big theories in that is that this will never be a static skill set. Right. This is why I think in our study, people with corporate AI training did worse than people who didn't have it. 34:21 I was gonna say, generally. And I would love, in general, this isn't the kind of thing where it's training and one and done, right? The, the, the models keep changing. The nature of the work is gonna keep changing. 34:30 We're gonna be moving to more agentic systems, which is gonna change how we collaborate with machines. 34:34 So I think we have to, uh, really help people s- make a commitment to say, "This is, uh, an issue that we gonna need to measure, we're gonna need to grow, we're gonna need to teach, and help people learn always," like on a day-to-day- Hour to hour... 34:49 basis. I don't think I have a fixed skill set with AI. I think, um, you know, every time I take the Quorix assessment, for instance, I learn something new about myself. 34:57 We've also developed, we're in beta right now with a live desktop version of Quorix that as you're in interaction with AI, it is stopping you before you hit submit on your- Oh... response. Yeah. And saying- Stop... 35:09 "Do this, do that," like challenging you. Like, I think we need, we need to think about the learning side of this as a live- Right... always-on intervention [laughs] in some ways, because- Yeah... 35:19 every time the models change, the way you work with it has to change. 35:24 What the future workplace is gonna be, what the role of humans is gonna be, what the role of machines is gonna be, and we think it, think of that as a moving target, not a fixed- Mm-hmm... 35:32 target, perhaps we can think more inclusively about, um, what kind of skills are we developing, who belongs in the mix, how do we mean- So what I'm hearing, and I love so many... 35:45 For the record, I m- mis- applied for a PhD last year. I, I never should've done a PhD, because it is not... [laughs] I'm too ADHD for it. 35:52 But like, but I literally wrote my whole proposal around the need for metacognitive forcing functions. [laughs] Hm. Interesting. 36:00 So the fact that you have, like basically like you need to be able to stop and think about your own thinking, but that needs to be like built into the UI/UX of, of your work with LLMs. Anyway, so like that- Mm. 36:10 I would love... We could go on for a whole nother hour. I love that. That's a thing that Jeff and I talk about all the time, is like the preservation- It, yeah... of your human judgment. 36:18 And when you have multi workflows going on, like if you're- Yeah... are building a lot of different AI workflows, your, your just erosion of- Yes. Yeah... 36:26 your human capability across this becomes really, really obvious to you, and you can feel yourself losing a lot of- Yep... what you know you need to do great work, right? And so- Exactly... 36:38 I think the question is how do we preserve human judgment and human resilience, um, through- Yep... this transformation cycle, and then through the, the work life of working with a machine? Because it- Yep. Right... 36:50 you're gonna need to do it always, right? I'm wondering about one thing, and I'll, I'll approach this from a, a listener question, actually. 36:57 So we have some, some listeners and readers who, who like to write in and pose us questions. We call it- Oh, fun... kind of like the drinks, the drinks round. It's a li- Yeah... 37:03 a little bit of a newer segment, but we wanna bring in, bring in voices. And so I'll, I'll read aloud from this question, and I would love... I don't have an answer for, for Melissa here, but I would love- Okay... 37:11 to hear your take on this. So- Let's hear it... one listener or reader, Melissa, works in research compliance, so relevant perhaps. 37:17 But she's, she knows that her colleagues use AI all day long to draft policies and checklists, but nobody talks about using it because everybody's a little bit embarrassed. 37:26 So how do we fee— how do you see this working in organizations in your, in your work? 'Cause it adds that third level of com- complexity. 37:33 It's human and machine, but then it's also human, machine, Mara and I have done this- And human and human... in our podcast work. 37:38 It's human, it's, I, my Claude talks to Cl- Mara's Claude, and sometimes we have to intervene and be like, "No, you guys are, you guys are messing this up." [laughs] Like, so- [laughs]... 37:46 so, you know, what does, where do you see this? 37:47 You know, it's sort of a fascinating question in many ways, and it speaks to actually an issue for women as well, because there's some research that shows that women are shamed more if they use- Yeah, yeah... 37:59 AI in their work. Um, and, and people say women are more reluctant to use AI at work, which I don't think is really true. I think women are just potentially, um, using judgment and caution a little more overtly- Mm... 38:11 in their work. And I do think we're in this very weird stage right now where people are getting better, um, at developing their own systems for collaborating, um, with AI. 38:22 But, um, you know, um, they, they are getting to their own ways of working. And some people are revealing that to colleagues, and some people are not. Some people are really- Right... highly secretive. 38:32 And I think even just out in the world, like, you go to all these, like, AI conferences, and you go see a panel discussion, and no one's saying anything. Like, you know- [laughs]... 38:40 people are, like, talking about their work, but they're not really talking about their work because- Yeah, yeah... people are holding confidence. 38:45 But I think you're seeing that same dynamic colleague to colleague in many ways. Some people- Hmm... are really excited to show what they're building. 38:51 Like, "Hey, here's a, here's an artifact that I built that is the dashboard of all the BD that I'm doing, and here's what it looks like." Yeah. 38:57 And then others are, like, they wanna, like, use AI, but not reveal that they're using AI, and I, I don't know what changes that. 39:05 I, have you, like, seen anything recently psychologically around, like, how we get past this weird stage right now where people aren't sure whether it's a good thing or a bad thing to admit, um, in terms of how they're using AI? 39:17 But to me, what you just described is somebody showing their dashboard to colleagues. I, it, it takes one person, and that could be a leader, but it's... 39:26 I think back to your point around you left a CEO position and now you're an, an, you wanted to be a practitioner. Yeah. And I think this is the moment for practitioners to shine. Yeah. Meaning- Mm-hmm... 39:35 you have to take a little bit of a risk, potentially. And as long as you're operating within the bounds of your company AI policy, et cetera, and you're not, you know, you're being careful around PII. 39:44 But, like, people should be coming forward and just saying like, "Hey, raise your hand. This is what I'm experimenting with." 39:48 It doesn't have to be to your entire organization, but at least show it within your team and say like, "Listen, I don't know if what I'm doing is right, but this is what I have made," and I guarantee- Yeah... 39:58 most people that you're further ahead, back to your research, you're probably further ahead than you thought that you were. Yeah. 40:04 I work with a company called Board of Innovation, um, which is a AI transformation studio, and one of the things I think they do really well as an AI native company is, like, that kind of internal show and tell- Yeah... 40:14 where people come and they show what they're building, either personally or professionally. Um, and it puts the builder mindset into- Mm-hmm... the spotlight in a good way. 40:22 The truth is, for a lot of us as knowledge workers, um, we've, we've not been able to build in the past. Like, you know, someone- Yeah... with my type of expertise, I've- Yeah... 40:30 al- my output has always been strategic- Yeah... um, not as executional. 40:35 But with AI, I can turn some of my strategy into execution, um, which is really interesting for a lot of people who haven't been able to do that in the past. 40:43 And I think we need to honor and kinda celebrate that a little bit. 40:46 Not saying that, look, I don't think I can work with Claude and create a more beautiful website than someone who knows how to create a beautiful website does. Yeah. 40:53 But, um, but I think we could show the kind of work that we're doing and out in the open experimentation is where you kind of- Mm-hmm... fuel use case design and where people go, "Oh, that's really interesting. 41:04 I didn't think about doing it in that way. Let me try that." And I think putting people together in actual practical builder groups, I'm part of a- Mm-hmm... 41:13 AI working group with a, with a group of women that meets weekly. 41:16 And, um, you know, even though we work in really different industries, the kinda stuff that we're talking about doing in a very candid environment is really helping everybody. And so- Yeah, yeah... 41:25 I think the builder mindset needs to come forward a little bit more in the workplace. 41:30 And I think, yeah, I think, I don't know how companies can create the right conditions for people to see that as a good thing, not a bad thing, and I'm not sure people are doing that very well right now if- Mm. 41:41 I don't- I can't remember who said it, but someone said, um, AI is a team sport. Yeah. And I think that- Yeah... like, and I really think that that's true. 41:49 I think leaving it up to the individual to be kinda, like, suffering through this alone is incredibly lonely. [laughs] As opposed to, like, s- which is how we started this podcast really. Yeah. Yeah. 41:59 Because Logan and I were like, "We need to suss this together." 42:02 But I think companies can set up, they really should be giving employees time during the workday to be working together and sharing together in small groups- Yeah... 42:12 where, where you can build trust so that you can experiment and show and fail and be vulnerable, right? There's, like, an EQ part of this- Yeah... 42:22 that I think doesn't come naturally when you're talking about a tech implementation. Yeah. Right? Yeah. 42:26 But it's somehow, it- it's somehow more kind of environment- um, emotionally sensitive than, like, rolling out the next version of Microsoft Word. 42:35 And so people need kind of these spaces, I think, to share and to grow together. That's, uh- Um, but they need time during the workday in order to make that happen. Yeah. That's exactly right. 42:44 I think treat, people treated AI like any other tech rollout, [laughs] right? Yeah. Or software rollout. Yeah. Which was hand out the tools. What happened in the workplace- Mm-hmm... 42:52 at least what I've observed, is a lot of individual experimentation, which doesn't scale to enterprise. Mm-hmm. 42:57 Like, people, no one's getting shared learning, and everyone's developing their own skills with or without, you know. And I do think, um, you can't treat AI in your organization as a software rollout. [laughs] Um- No... 43:09 getting people into working groups and actually building together- Yeah... I think is, is a huge unlock. Um- Yeah. 43:15 And from the time, I mean, I'm, I always repeat this like, uh, forever, but because women have more time poverty- Mm... 43:21 building that opportunity within the workday and managers giving people who are, you know, individual contributors- Yeah... space t- um, is kind of the only way you're gonna be able to do that equitably. Yeah. Yeah. 43:33 Totally agree. Yeah. Well, I know we're almost at time. Where, where can people follow you, Jo? Where should people find you? What would you like any listeners to do? Well, um, I'd love people to follow CoreX. 43:46 We're on- Mm... uh, we're only on LinkedIn, and our website is corex.ai, and we, we love to have more and more people take the assessment and give their feedback. 43:53 And we also will be opening up beta for our desktop app and for our enterprise. So we're- Sign me up. Yep. [laughs] Yeah. 43:59 So we're looking for people who wanna participate and understand what it means to, like, have a machine collaborator that helps you get to be a better, a better human machine collaborator- Mm-hmm... in the mix. 44:09 Um, and then follow me on, on LinkedIn. I'm Jo McKinney, and, um, heyjoemckinney on TikTok, where, um- Yeah, you are... I have my more embarrassing content. [laughs] The best kind. The best kind. 44:22 Well, thank you so much, Jo. I think we could probably keep talking- Thank you so much, Jo... for hours and hours. This was wonderful. So fun. Loved every minute of this. Appreciate your time. 44:29 Thanks for helping me womansplain for the day. I appreciate it. Yes. [laughs] [outro music]