Will AI kill the essay?

CLIP You’re listening to a frequency podcast network production in association with City News.

Jordan Heath-Rawlings If you have heard of ChatGPT, and if you are interested in artificial intelligence at all, then you probably know that this program has passed some tests. It passes more. Every single day from just the past week or so. Here are a few tests that ChatGPT has passed. It has passed a basic university exam, a basic social sciences essay. It is passed Google’s coding jobs test, the entry level position, which starts at $180,000 a year. It has passed the University of Minnesota’s Law School entrance test, though the university is quick to note. Only just, it has passed Amazon’s software position, interview test. It has passed something called the Nazi test, which means you can’t make it praise Nazis, so, that’s good. And it has passed the US Medical Licensing exam, and there’s more. I just got bored of looking them up and writing them because I didn’t use AI for this task. Suffice to say of the many ways in which these programs will impact the world, one of the first ones we’re going to have to grapple with is our idea of measuring intelligence and knowledge and qualifications with simple written tests or essays. And that is pretty bad news for a lot of professors at a lot of institutions because that’s like most of how they grade. And the AI would also score perfect on the 10% they hand out for attendance. So what comes next for universities and colleges if assigning an essay or a series of written questions can be gamed by a machine. How are these places trying to find and stop the use of AI programs like ChatGPT, and how futile is that effort in the long run? What does higher education look like if we aim higher than what a machine can do?

Jordan I am Jordan Heath Rawlings. This is The Big Story. Jeff Schatten is an associate professor of business administration at Washington and Lee University. He has written extensively about artificial intelligence and also incorporates it into his coursework. Hello, Jeff.

Jeff Schatten Hi there. Thanks for having me on.

Jordan You’re welcome, thanks for joining us. This is a really complex topic and we could go down a number of rabbit holes, but I. What’s attracted a lot of attention is how radically these programs could change, like, I guess the stereotypical model, uh, of teaching and lecturing and essay writing and rinsing and repeating. Do you think that that’s a legit concern?

Jeff Schatten The latest iteration is ChatGPT, which came out in late November. The biggest implication in higher education is really on student work. It’s garnered all the attention and deservedly so. When I first got interested in the space about 18 months ago, which was the initial, uh, one of the initial products was for GPT3, I started messing around with it. It was put out by OpenAI. And it could do very limited student work. So I would type in to a college level essay. And what it would put out was just, you know, uh, high school level, middle school, riddled with errors, right? Repetition. It was hard to even get the user interface to work at times.

Jordan And 18 months later?

Jeff Schatten And 18 months later, I will walk you through some examples. It will blow your mind what this can do. And so some of the attention, a lot of the news articles that have come out just in the last couple weeks have been, uh, ChatGPT passes a law exam at a particular university, or it gets a C in the MBA program at Wharton on an exam. For this to be able to get a C at Wharton is already very impressive, but that’s just when you take it at a glance. That’s in first gear. So the way that those tests are all done is that they input the essay question, whether that’s gonna be an accounting question or whether that’s gonna be a question on strategy. They input the question as they would give it to a student. They put it into ChatGPT. ChatGPT puts out the result, and then the professor grades it and sees that it’s about a c. that’s in first gear right now, what ChatGPT does, that is such an amazing improvement upon any other AI is that it’s a conversation. So you can put in an initial question and you get an output, but then you get to iterate with it as if it is your own, your very own assistant. I’m gonna go through an example I just put in for something that would be, you know, normal in my class, I asked ChatGPT to do an essay about personality and Elon Musk. This is for an organizational behaviour class. And so I can say, you know, write the introduction for a college level essay about the personality Elon Musk. And what it pumps out is, I mean, it’s not, it’s not all that impressive, right? Just like the before examples that I gave, it would effectively get a C in one of my classes. And so if you stop there, you can say, oh, well, oh, that’s it. That’s just a C student. But you can iterate with ChatGPT. So then I say drawing on this, can you write it in the style of someone who won a Nobel Prize in literature? So now it takes the original input — it’s not gonna get a Nobel Prize, but I’m gonna just give you some of the just the first line that it pumps out. In the realm of business where numbers and profit reign supreme, the traits of narcissism and charisma are off coveted by those who seek to lead. Now those two parts where numbers and profit reign supreme in the second part, often coveted by those who seek to lead, that’s already above the level of my students.

Jordan It’s creative writing.

Jeff Schatten Yeah, exactly. I mean, it is, it has taken what was a boring essay and that line is now Yeah, that, that, that can kind of grab you, but it doesn’t stop there. If I then ask it to iterate again and I say, you know, rewrite this paragraph, but use lofty metaphors, and use language that is more complex. And can you add one piece of wit in the style of Christopher Hitchens? In the cutthroat arena of commerce, where the almighty dollar holds sway, and the bottom line is king, the allure of narcissism and charismatic leadership is often too great for those who yearn to ascend the throne of corporate dominance. But when these traits are indulged to an excessive degree, as is the case with Elon Musk, the ramifications can be akin to a contemptuous storm that wreaks havoc upon all those in its path. A recent analysis of the Tesla CEO’s modus operandi lays bare a man be set by a profound and incessant craving for validation and an inclination to place his own interests above the wellbeing of his companies and stakeholders. It appears that for Musk, the only thing more important than being in charge is being able to tweet about it.

Jordan Just a little less flowery. Now you have to iterate it until to dial down the lofty rhetoric now.

Jeff Schatten That’s fine. So I can have it do that. So you mean there’s parts that we can pick apart. Yeah, but I mean, here’s one thing. I mean, I can’t write that. My colleagues can’t write that. Maybe some in the English department, but I really don’t know many people that can do that until, let’s take, let’s just take that last line. It appears that for Musk the only more important thing than being in charge is being able to tweet about it.

Jordan Yeah. That’s a good line.

Jeff Schatten That’s a great line. That’s Christopher Hitchens. And I can then ask it to do the last line in the style of Thomas Friedman and it says, The true test of a leader’s effectiveness, however, lies in the ability to balance their own ambition with the needs of the people they lead. In Musk’s case, it seems that this equilibrium remains as elusive as one of his reusable rockets.

Jordan Also good. Yeah.

Jeff Schatten Very good.

Jordan You’ve kind of proven that this thing can be incredibly malleable. It can mimic styles, um, it can iterate in the direction that you want it to. How is it being used by students right now? Do we know that it’s being used to submit essays? How would you as a professor try to determine, uh, if your students were handing in ChatGPT written essays? Like what do we know about the pervasiveness of this thing, I guess, you know, just a few months after its release.

Jeff Schatten So there is a study done recently by, I’m not very familiar with this website by intelligent.com to take it for what it is, that showed that 30% of college students use chat c p t for homework and for assignments. And those are the ones that are admitting to it. And that’s two months after it’s release, right? So at least we have an idea that a lot of students are using it, um, for good reason. It’s free. You just log on and it produces results that even if you don’t know how to use the tool well, are still decent for college purposes. So, in terms of what to do about it as of today, uh, the cops are much better than the robbers. There’ve been about a dozen AI programs that are designed to catch ChatGPT. And they’re to varying degrees, but most of them are over 95% effective. Right now, if a professor did suspect that a student was using it, they would be able to, you know, put it into one of these anti plagiarism softwares, um, and most likely catch the student. But the thing is, I don’t think it stays like that. And the reason why is, all right, so we have CHatGPT and ChatGPT also has leaves breadcrumbs in its writing for these plagiarism softwares to detect. But Google’s developing theirs. China is developing one. Russia’s developing one. I mean, what happens when there’s, let’s say 30 of these? And many of them are not gonna be coded in such a way where they’re in, in intentionally leaving breadcrumbs, um, for these anti plagiarism softwares to use.

Jordan So what is it about high school, or in your case, university level essays that can make them so vulnerable to replication by this AI. And I, you know, I was a newspaper magazine editor. I write scripts every day for this podcast and, and other podcasts. I have played around with this thing, trying to get it to mimic the stuff that, uh, my colleagues would do or that I used to do, and it’s off, it, it doesn’t make sense because it’s not being trained on real world examples happening at this very minute, and it doesn’t quite have the style down. But it seems from both what you’re saying and from what I’ve read elsewhere, you know, there’s something about the essay style that is like, perfect for this thing.

Jeff Schatten Well, it’s not perfect for anything. I mean, so I haven’t talked about the limitations I asked it to do a literature review for some organizational behaviour questions. And it took me down the road of two different journal articles and I kept asking it more and more questions about theory development, about statistics, about the research background, and it gave me infinite depth about these articles. Except ,one of the articles doesn’t even exist.

Jordan Huh? Did it make it up or what?

Jeff Schatten Totally made it up! It absolutely made it up full scratch from nothing. So it, it speaks with the same confidence right now about an article that does exist with an article that doesn’t exist. Hmm. So it can be, I mean, it can be really mischievous. I mean, I, I did a thing where I said, um, what is five plus five. And ChatGPT says 10. And then I wrote, no, it’s. , and then it said, I’m sorry, I was mistaken. Five plus five equals eight. And there’s all kinds of quirks within this system, but my assumption is that it’s two months old. And when you look at the exponential rate of change that has happened with this tool, I mean, these things are gonna get figured out. So I’m less concerned about the quirks of Oh, the people who are freaking out. Oh my gosh it wrote an entire essay about an article that didn’t exist. Sure. I mean, it did. And so, yeah, if you’re using it today, you certainly need to be very careful that you’re talking about things that are real and you need to back it up. So right now the killer combo is actually humans with AI. You just can’t hand just hand full, you know, slots of of assignments over to AI at this point. It has to be integrated with humans. But my big question is for students and for the rest of us, I look, I don’t think this is just college. I mean, I think this is for journalists. I think this is for doctors, this is for geneticists, this is for researchers. You know, we’re ju you know, we happen to be talking about college, but the question is, what is our role? Once this tool and these tools get so much better than they are today. Cuz at this rate to change, it’s hard to see what our role is in a, in a lot of these process.

Jordan So if we’re gonna take that as a given, that these programs will continue to get better and better and better at evading detection through the software you mentioned and better at not integrating completely false things. What role should they play in the educational process then? If this AI is eventually within a year, 18 months or whatever, going to be capable of a fully formed factual college essay, what do we teach in college? How do you, how do you change what you do?

Jeff Schatten Yeah, so that’s a great question and it’s really the, the central one. So I’m gonna go through the options that I think are out there, and I’ll give you what my perspective is. These are all being hotly debated, um, on campus. The number one option is to just, you know, some people just wanna put their head in the sand and pretend that it doesn’t exist and hope that it goes away. That is unlikely to happen. The second option is to outright ban it. We’ve already seen this. There’s universities in France, there’s universities and India who have done outright bands. W’ve seen this in several, uh, major school districts in the United States where they’ve done outright bands. I don’t think that works either simply, you know, like the war on drugs or the war on obesity, or the war on porn or the whatever you wanna put your war on, it tends not to work. You know, anytime there’s something, a technology or any other thing that has a huge demand in society, it makes its way through. As soon as the, as soon as the use case is, is high and the, and the cost is low enough, it’s gonna make its way through. The third option is to just embrace it. And I do know people, I know professors who’ve embraced it and said, um, yeah, we’re just gonna allow it free form into anything where if people want to use, might even know somebody who basically allows full use of AI and then uses AI to grade it. My proposal is, I think we have to find a more sensible way and that we’re, there’s no sense in banning it, but there’s also no sense in really embracing it. I think professors have to do the very, very, very hard work and that is to design courses and design pedagogy around AI. Where AI can be used as a, something for brainstorming, but not that we can just give it to students to do all of their work for them.

Jordan So give me an example of what that might look like?

Jeff Schatten So, for example, in my class, um, I have them do a full consulting project. So they have to step into local businesses and do a consulting project. What can AI do? I mean, it can help them generate ideas, but at the end of the day, these are local businesses where they can’t go to the AI and ask about, you know, the hardware store in Lexington and about their particular problems. They’re actually gonna have to go into the hardware store and interview the managers and interview the employees there and develop a plan to, uh, to work with the problems that they’re facing. I give a lot of simulations. I have them do in-person negotiations, I have them do TED Talks. You know, at this point I have made my courses, you know, largely about, uh, I’d say maybe 70%, where, you know, AI can help you through strategizing. It can help you as brainstorming, but it’s, it’s really not gonna do the assignments for you.

Jordan You have a very practical approach, which is cool and very usable in places like real world business problems or face-to-face negotiations or all that kind of stuff. What would your take on incorporating it be for something like history or social sciences where, you know, by and large, and this is, I’m not saying this is a criticism of it, but by and large it is, you know, study, learn in depth about a subject, think critically about it, and present an essay or an exam on it?

Jeff Schatten Yeah, I I think the business subjects are gonna be, To work around the age of AI, full stop. Nonetheless, I, I think, uh, simulations will still play a role, public speaking, right? Where students have to come and present and you set limits on PowerPoints so that they’re not just reading off off of a PowerPoint slide, which, you know, I think in general is a weak form of presentation. They too can come up with experiential assignments. So, yeah, I think there’s still gonna be options, um, for the more liberal arts side of the equation, but those professors will naturally need to be a little bit more creative. My big concern about this is that for 400 years, we’ve really had one tool for people to make sense of complexity. And that tool is writing: business plans, relationships, stories, speeches. The mission statement at Washington University says that we teach students to think freely, critically, and humanely. And so much of that happens through writing. And my overarching concern of all of this is that the neural pathways that you and I took, you know, that were developed in our trajectory as we aged as adolescents and came to our twenties, a lot of that, all the critical and deep thinking that we did happened through writing. And I believe that we’re gonna have to have a, you know, a range of tools that we give to students so that they can still get those same, you know, neural developments that you and I had through writing. It’s gonna take a lot more creativity. I don’t think all is lost. I mean, I’m optimistic. I think the future’s gonna be great for colleges and for college students. But it’s gonna take, you know, kind of a sober awakening, um, for professors to rethink what are we in this business for? I mean, we’re really not in this business for teaching writing. We’re in this business for teaching thinking, right? We’re trying to equip students with tools so that they can have a life well lived so that they can go have a career that that’s successful for them so that they can, you know, engage in deep relationships. So they can be a members of community, so they can have civic engagement. And you know, we have one tool that is potentially under threat and it’s probably just gonna be time for us to wake up and develop new tools.

Jordan That was gonna be my next question, and it’s one of my last, which is you’re a member of academia. Anybody who’s had experience in academia, Would maybe say that it perhaps is not the most welcoming to quick change in the way of doing things. Like how resistant do you think institutions will be on mass at moving away from writing as the main tool for critical thinking and learning? What have you seen, you mentioned that they’re pretty adamantly either against this or sticking their heads in the sands. Like how do you do that seismic shift?

Jeff Schatten I think it’s gonna take a long time. Academia is, I mean, exactly as you said. You know, it is notoriously slow to change. Um, and because these institutions are so decentralized, you have to realize for most for most universities, professors have their own fiefdom, and so I can come up with that. However I wanna approach it, it has no bearing on how my colleague approaches it in the office next to me. In most universities professors have what’s called academic freedom, and they can effectively teach however they want. The university can ban ChatGPT, they can allow it, but at the end of the day, it’s really gonna be up to the professors individually to decide how they’re gonna teach particular courses. It’s gonna be up to departments to have these conversations. It’s gonna be up to entire schools to have education sessions. I just gave a, a session, a large, a session at Washington and Lee where, I mean, it was packed as professors are trying to grapple with, you know, what does, what does all of this mean? How do we actually think about. You know, crafting courses that are gonna work over the, you know, in the, in the decades to come. So, yeah, I mean, I think, I think it’ll be slow, but at the end of the day, professors care about teaching. They care about their students and they’ll start seeing the results, which is, I think in a lot of universities, they’re gonna see the students just aren’t doing the work they used to do. You know, think about it this way, a a a lot of technology that ha that has been a threat to universities has actually not turned out to be a. You know, I think of online education has not actually done much to traditional universities. I did a survey of my students, you know, how much effort did you put into the online classes versus the in-person classes? And it was, you know, 10 to one the effort for the in-person classes. Students just don’t tend to do much when it’s an online class. They’re lazy. They don’t do their, they don’t do their assignments, and so it was, it’s never, we thought, you know, 20 years ago that maybe online education was gonna be a, a threat to the traditional, um, university system, and it’s just turned out to just sit on the periphery. This is very different, right? This tool allows a lazy student to excel, and so I think it’s gonna take a, a reorganization. I wanna show you one more thing cuzI think it really captures where we are at this moment. You know, right now these AI tools have a lot of limitations and as I mentioned earlier, it really takes the integration of humans with AI to produce something stellar. Well, Beethoven produced nine, symphonies in his lifetime, and he was about 5%. Of the way through his 10th when he died, and Beethoven scholars fed the first 5% of the Beethoven’s 10th Symphony into an AI and that AI produced hundreds or thousands of different full be Beethoven symphonies. Just like the examples that I’ve given, some of it was junk, but some of it was really good. And so with the combination of Beethoven scholars and AI, they produced Beethoven’s 10th Symphony and, and this is what it sounds.

MUSIC

Jordan Well, s***

Jeff Schatten Yeah,This is the world that we’re entering into right now. It’s human and AI, but that balance is shifting and it’s shifting pretty quickly. It’s gonna be universities, it’s gonna be everyone to sh shift the way that we start educating and preparing the next generation, cuz then world in the next generation gonna be very different than the world of our generation.

Jordan Last question then. That will happen eventually the world will shift. In the meantime, how messy is it gonna get on the actual like ground level of university with, you know, programs innovating faster than programs that can catch them, can catch them, versus evolution of this technology and students cheating and some students not like — it’s gonna be a disaster for a little bit. It’s gonna be a disaster for a little bit, no?

Jeff Schatten Yeah. Yeah. The next few years I think are gonna be very tough in the space as universities figure out, um, how they’re gonna approach it. I’m optimistic that in the end, that we figure out, um, kind of collectively about a series of best practices. Um, I’ve kind of laid out some of what I think that might look like. But in the process for, you know, for your average college student that has this tool that they can just use to, uh, skirt by their assignments, it’s just too easy. And yeah, they’ll, they’ll, they’ll use it until there’s either, you know, sufficient crackdown. But then there’ll be other tools to, you know, skirt that or when I think ultimately when professors start changing the way that they teach.

Jordan It’ll be fascinating to watch. And I was gonna say, I’m glad I’m not in school anymore, but maybe it would’ve been easier, so I’m not sure.

Jeff Schatten I mean, I think it’s exciting. I find change in general to be just very exciting and interesting and fun to be a part of.

Jordan It’s all well and good till they take your job. Jeff.

Jeff Schatten I have tenure.

Jordan Can’t machines get tenure?

Jeff Schatten one One day. One day.

Jordan Thanks so much for this.

Jeff Schatten All right, thanks a lot.

Jordan Jeff Schatten of Washington and Lee University. That was The Big Story. For more, you can head to the big story podcast.ca. A quick note, I am out of here for the next few days. I am trying to navigate whatever hellscape our airports have become for a little vacation. We’ll report on that when I get back. Friend of the podcast and someone whose voice you know regularly. If you listen to this show, Donnovan Bennett will be in the host chair for the rest of this week. I’ve already seen some of the stories he’s got planned. I hope you enjoy them. They’re really great. If you wanna talk to me or Donnovan or the rest of the team, You can find us on Twitter at the Big Story FPN. You can write to us hello at the big story podcast.ca, and you can call and leave a voicemail, (416) 935-5935. The Big Story is available in every podcast player, and of course you can get it on your smart speaker by asking it to play the Big Story podcast. Thanks for listening. I’m Jordan Heath-Rowings. Enjoy Donnovan for the rest of the week. We are off for family day here in Ontario next Monday, so I will talk to you on Tuesday if I’m not still stuck in an airport.