Fighting Fire with Fire: Scalable Oral Exams

I did, however, pepper my answers with statements like "it is widely accepted that the industry standard for this concept is X". I would feel bad lying to a human, but I feel no such remorse with an AI.

You've piqued my interest!

To the extent of wondering what value the human instructors add.

Students had and still have the option to collectively choose not to use AI to cheat. We can go back to written work at any time. And yet they continue to use it. Curious.

In my BSc and MSc we were all basically locals who are in all aspects about the same except from the aptitude to study. In the university where I did my PhD there were much more divisions (aka diversity) in which every oral examiner would need to navigate so one group does not feel to be made preferential over another.

I wish that wasn't a thing.

Interviews are similar, but different: I'm presenting myself.

Next steps are bone conduction microphones, smart glasses, earrings...

And the weeding out of anyone both honest and with social anxiety.

We have already been there. A student asked whether they could use an app to "translate" the examiner's instructions. The app was ChatGPT, prompted to solve all questions in the conversation.

The real problem is students and universities have collectively bought into a "customer mindset". When they do poorly, it's always the school's fault. They're "paying customers" after-all, they're (in their mind) entitled to the degree as if it is a seamless transaction. Getting in was the hardest part for most students, so now they believe they have already proven themselves and should as a matter of routine after 3-4 years be handed their degree because they exchanged some funds. Most students would gladly accept no grades if it was possible.

Unfortunately, rather than having spines, most schools have also adopted a "the customer is always right" approach, and endlessly chase graduation numbers as a goal in and of itself and are terrified of "bad reviews."

There has been lots of handwringing around AI and cheating and what solutions are possible. Mine is actually relatively simple. University and college should get really hard again (I'm aware it was a finishing school a century ago, but the grade inflation compared to just 50 years ago is insane). Across all disciplines. Students aren't "paying for a degree", they're paying to prove that they can learn, and the only way to really prove that is to make it hard as hell and to make them care about learning in order to get to the degree - to earn it. Otherwise, as we've seen, the value of the degree becomes suspect leading to the university to become suspect as a whole.

Schools are terrified of this, but they have to start failing students and committing to it.

The two solutions to this are (1) as some commenters here are suggesting, give up entirely and focus only on quality of output, or (2) teach students to care about being more than appearance. Make students want to write essays. It is for their personal edification and intellectual flourishing. The benefits of this far surpass output.

Obviously this is an enormously difficult task, but let us not suppose it an unworthy one.

In reality, they cheat when a culture of cheating makes it no longer humiliating to admit you do it, and when the punishments are so lax that it becomes a risk assessment rather than an ethical judgment. Same reason companies decide to break the law when the expected cost of any law enforcement is low enough to be worth it. When I was in college, overt cheating would be expulsion with 2 (and sometimes even 1 if it was bad enough) offenses. Absolutely not worth even giving the impression of any misconduct. Now there are colleges that let student tribunals decide how to punish their classmates who cheat (with the absolutely predictable outcome)

I suppose there are other fields where the degree might be used mostly as a filtering mechanism, where cheating through graduation might get you a job doing work different than your classes anyway. However, even in those cases it's hard to break the habit of cheating your way around every difficult problem that comes your way.

They're even more possible if you do an oral exam only on the highest grades. That's the purpose, isn't it? To see if a good, very good, or excellent student actually knows what they're talking about. You can't spare 10 minutes to talk to each student scoring over 80% or something? Please

> And here is the delicious part: you can give the whole setup to the students and let them prepare for the exam by practicing it multiple times. Unlike traditional exams, where leaked questions are a disaster, here the questions are generated fresh each time. The more you practice, the better you get. That is... actually how learning is supposed to work.

It depends on how frequent and how in-depth you want the exams to be. How much knowledge can you test in an oral exam that would be similar to a two-hour written exam? (Especially when I remember my own experience where I would have to sketch ideas for 3/4th of the time alloted before spending the last 1/4th writing frenetically the answer I found _in extremis_).

If I were a teacher, my experience would be to sample the students. Maybe bias the sample towards students who give wrong answers, but then it could start either a good feedback loop ("I'll study because I don't want to be interrogated again in front of the class") or a bad feedback loop ("I am being picked on, it is getting worse than I can improve, I hate this and I give up")

if we want to educate people 'how people work', companies should be hiring interns and teaching them how people work. university education should be about education (duh) and deep diving into a few specialized topics, not job preparedness. AI makes this disconnect that much more obvious.

This is why we need to continue to educate humans for now and assess their knowledge without use of AI tools.

CFR 46.104 (Exempt Research):

46.104.d.1 "Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods."

https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-...

So while this may have been a dick move by the instructors, it was probably legal.

But isn’t the whole point of a class to move from incompetent to competent?

Isn’t the poor performance on those exercises also part of their overall performance? Do you mean just that their positive work outweighs the bad work?

Also, IMO oral examinations are quite powerful for detecting who is prepared and who isn't. On the down side they also help the extroverts and the confident, and you have to be careful about preventing a bias towards those.

> I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading.

Do you, like, really think this is the best way to assess someone's ability? Can't we find some place in between the two extremes?

Personally, I'd go with a school-provided computer with access to documentation. And no AI, except maybe (but probably not) for very high-level courses.

And why is this a flex exactly? Sounds like an extortion scheme. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass.

It's like some malicious compliance take on both teaching and studying.

Mind you, I was (for some classes) tested the same way. People still cheated, and grading stringency varied. People still also forgot everything shortly after wrapping up their finals on the given subjects and moved on. People also memorized questions and compiled a solutions book, and then handed them down to next year's class.

For comparison we had lengthy sessions in a jailed terminal, week after week, writing C programs covering specific algorithms, compiling and debugging them within these sessions and assistants would follow our progress and check we're getting it. Those not finishing in time get additional sessions.

Last exam was extremely simple and had very little weight in the overall evaluation.

That might not scale as much, but that's definitely what I'd long for, not the Chuck Norris style cram school exam you are drawing us.

The old ways do not scale well once you pass a certain number of students.

You have a very weird idea of education if a teaching method that results in a 99% failure rate is seen as good by yourself. Do you imagine a professional turning out work that was 99% suboptimal?

And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off.

This is clearly someone who had a conclusion to promote regardless of what the data was going to show.