Short answer: Jordan Metzner says it sends a problem to Gemini 2.5 Pro, GPT-5, Claude Sonnet 4.5 and Grok 4 Fast, which write Lean proofs, then ChatGPT judges them. He also built an API so Axiom's team can run their own questions.
Watch this answer from Episode 22 (starts at 2:19) · Episode 22 notes and transcript
From Episode 22: Let's do the Math!, starting at 2:19.
Jordan Metzner [2:18]: Yeah. So, hey, Carina. Welcome to the show. Hey, Sam. Another great episode. Another crazy week in AI, obviously, with the news of Gemini three dropping this week and a bunch of other things. So, it's almost never a dull moment, in the AI world. When we first spoke with Carina, we talked about some ideas of just building some kind of vibe coding applications that could be cool for math and could be even cool for Axiom in general. So I built an LLM Math roaster. Carina, I hope you like it, but I wanna show you how it works. And it's it's productionalized, so it works like pretty well. Okay. Cool. So I have it in light mode and dark mode. Pretty cool. Right? Alright. I have some style. Alright. So dark mode is better for math. Right? I feel like that's like a little on mass.
Carina [3:02]: No. Well Light mode?
Jordan Metzner [3:04]: Light mode? Okay. Light mode for math. Alright. So here's how it works. So I collected a bunch of different problems of different levels of difficulty. And then I have different models. I have Gemini 2.5 pro, GPT five, Claude Sonnet 4.5, and Grok 4 Fast. Now I wanted to add Grok 4.1 and Gemini three, but these models just came out like yesterday, and I don't have API access yet. So and even 5.1 is out, but I just didn't get it to like put it in time. But before I run it, me just show you a little bit about what the application does. So you put in the math problem, it'll score it, roast it, and we'll go through that in just a second. You can actually submit a custom problem. So if you have your own problem, you can simply do that. That's awesome. We have the history. So if all the maths problems that have run-in the past, we can see how well they performed. And then I thought that'd be fun for your team, but I did a little extra and I built a little API. What your team can do is generate their own API key and using my API link here, they can pass in a request of a question and then what they can do is have in the background it run, have all the models go do their own results and then they can compare them. Let's jump into it.
Jordan Metzner [4:19]: So first is, I'm okay at math but I'm certainly not your level. So the first thing I wanted to do was just build something really easy to prove whether or not my model works. So I did something, let's just you know two plus two equals four and at this type of math is you know, probably at my level and I think I can solve it. So what I have all these models doing is I have Gemini, they're correcting, they're writing kind of a lean proof. You can here see the response times from each of the models.
Carina [4:44]: Wow.
Jordan Metzner [4:45]: We can compare the results of each of the models And then, obviously, you have this problem which no model knows which model is smartest. So I chose ChatGPT. So ChatGPT will then evaluate the responses from all of the models. Let's just get this finished with two plus two, and then we can go into a more complicated question and see how well it does.
Carina [5:03]: Yeah. That's great. Oh, evaluating.
Jordan Metzner [5:05]: It has to get processed each one of these against, like, the core source. So, ChatGPT's feedback is that all of them are correct. Okay. Great job. So fantastic. Two plus two equals four. All models know that. I think we're on good shape. So so far, the basis of truth. Right, Carina? I'm not a perfect scientist, but you know, I know a little bit. I have a few different problems. Maybe you wanna choose which one? I have a
