Short answer: Jordan Metzner demos a dermatologist candidate scored 89.5 overall, with chaptered video playback, multilingual subtitles, playback speed, per-question evaluations, a full transcript and whiteboard analysis. Sam Nadler notes the avatar asks follow-ups based on previous answers.
Watch this answer from Episode 6 (starts at 9:30) · Episode 6 notes and transcript
From Episode 6: From Avatar to AI Hire: Building NTRVSTA, starting at 9:30.
Sam Nadler [9:30]: Yeah. I would love to see a quick demo of what the actual tool looks like.
Jordan Metzner [9:33]: We can jump into a specific candidate here, and we'll have to block some of these details because they have some personal information. But, Ntrvsta works across all different types of interview types. In this case, this is actually a dermatologist who's interviewing for an RLHF role in regards to dermatology. And, the reason I found this role, so interesting, and I selected this interview, because the candidate just absolutely fantastic. She gets a 89.5 as an overall score, but a ninety in a technical score and 85 in soft skills. And so when a candidate takes an Ntrvsta actual interview, the first thing we do is we ask them their background, where they're from, what their interests are, what their hobbies are. In this case, Sarah likes to play tennis. We take us we take a headshot of the candidate, and then we also talk about highlights and lowlights of the interview. So, we see some of the areas here where the interviewers did great, and then maybe some other areas where they can improve. And then when I was I jump into it, what I think is really exciting is that you can watch back the entire interview here, simply by either watching it in a full playback mode, but you can actually just jump to any chapter in the interview and see the candidate's feedback.
Jordan Metzner [10:50]: I chose this chapter, but
Dermatologist [10:52]: Terms of the possible diagnosis. So it wasn't always very straightforward what the diagnosis would be. As we talked about before, it can be difficult to tell between an acne papule and a squamous cell carcinoma unless you have the appropriate history or you're looking at surrounding photo damage, on the patient. How do you handle cases where a lesion doesn't fit neatly into standard categories? This can be difficult. When this occurs, I use my knowledge to make the best annotations possible based on the information that is given. Typically, you are able to tell color. You can tell the primary lesion. And if other options are indeterminate, that indeterminate is selected so that the AI is not being trained to make guesses that are not accurate.
Jordan Metzner [11:39]: Great. So I just want to show out some features here besides chapters. Obviously, we've got subtitles supporting multiple languages. We've got playback speed, so you can actually watch the interview at a faster speed, which is also pretty awesome. Here we have every question with an answer and some evaluation. So if you jump around, we have the availability for you to copy and review the entire transcript of the interview. And for some interviews, we also have whiteboard analysis. So the candidate will be asked to sketch something. In this case, a simple workflow diagram that standardized the annotation process. And here you can see the image that the candidate drew during the interview as well as some analysis there. So really some awesome features when you're going to review a specific candidate.
Sam Nadler [12:21]: And as you can see from the video, yes, I don't think it's hiding the fact that it's an avatar. But, like, pretty quickly, it becomes pretty normal. The latency is is very reasonable. Yeah. It's maybe not as fast as a conversation with two humans, obviously, but it's it's, not awkward. And I also noticed at the end, like, it's dynamic. She asks you know, she takes the avatar takes your answers and asks a follow-up based on your previous answer.
Jordan Metzner [12:56]: Yeah. It's incredibly dynamic, and that's, think, I what makes it feel so natural for the candidates as you saw here with Sarah. And, here is a selection of dermatologists, and you can see their different scores and how well they performed. And in this case, Sarah's the top performing. But I would say that, 89, 85, these are still really, really good scores that we see overall. I could also jump into some other features. Theoretically, you could have, whatever, a dermatologist job opening post this role, move people through, and, you
Sam Nadler [13:27]: Know, depending on how fast people take the interview. But within a day or two days, you could have five candidates, their scores, their full interview, and wasted no time. Before we jump into the other features, I see that we have these dermatology candidates, and they were asked specific questions regarding dermatology. We have engineer candidates. We have legal candidates. How do you know that the avatar is gonna ask, dermatology related questions and legal related questions, like, versus something completely off base?
Jordan Metzner [14:00]: Yeah. So we've done a lot of prompt engineering to really get the interview experience to be really tight to keep Carolina, our interviewer, on track and make sure that she's able to both be friendly and communicative while not being inappropriate and not falling offline. We've done a lot of work to try to break that, see if we can say have her, say something inappropriate or something she shouldn't shouldn't be saying. And we've actually found that we've had really great results so far. And, most candidates are trying to get a job, so, they also act professional. And we find that, the feedback loop is is pretty accurate on both sides, both from the recruiters as well as from the candidates.
