Dr. Alan Leichtner, Dr. Benjamin Zendejas, and Dr. Ariel Winn - Tools for Addressing the Pain Points in Graduate Medical Education

you you you you you you you you you you you you you bajrek Nadal F Smet me you Thanks, Laura. It's a pleasure to be here, especially for combined rounds. I was just thinking back because when Dr. Hendren was chief of surgery used to invite me yearly to give surgical grand rounds and would promptly send me a VCR tape and I have a stack of them like this. I have never once looked at myself talk but I can't throw them away. I'd like to thank Laura, the organizers on Dr. Naveita was also essential in getting me invited to this. I'd also like to take the opportunity to thank Dr. Hickey because in my role as chief of medical education at childrens, I realized that there's been no one who's been more supportive of educational innovation than Dr. Hickey. So thank you. I've been also grateful to the people that I spoke to. I have spent hours in the OR but don't really know what it's like for surgeons or anesthesiologists, let alone what it's like in the ICU or on the pain service or in the pre-op clinic. So I spoke to a number of people including Mike Hernandez and number of the other faculty and also some trainees to get an idea of what this was like. I have enlisted two rising stars in medical education at childrens to help me. Ariel Wynn is an assistant professor of pediatrics and a hospice in the division of general pediatrics. At childrens, she's the associate director of the pediatric residency and also an associate director of the academy at childrens. Ben Zendéhas is one of our junior surgical faculty and an assistant professor of surgery at HMS. Ben completed a research fellowship and surgical education emphasizing simulation before he came to childrens. I'm going to start with a very brief overview of the Department of Medical Education because I think it's something that could benefit everyone in the room because everyone in this room teaches whether you're a trainee, whether you're a physician, whether you are a nurse, whether you're a nurse anesthetist. And I really would like people to be aware of what we offer. The Department of Medical Education was founded in 2015, relatively new and has a purview over undergraduate students who are on campus from Harvard and other schools. GME and you see lots of trainees, but at childrens, we have 500 trainees who are based here and another thousand, as you know, who rotate through. And then continuing medical education. And I really should call this continuing education because we are fortunate to be able to provide joint accreditation for continuing education courses where the first children's hospital to achieve this. And this means that we can accredit courses for physicians, nurses, pharmacists, and now physician assistants, optometrist, social workers, and psychologists, all in one fell swoop. So you all work in teams and now we can educate people in teams. The other thing I'd like to make you aware of before we start the program is the academy of teachers and educators at childrens. This was founded in 2008 with the mission of improving clinical teaching and I would also add the learning environment, fostering educational scholarship, and supporting career development of those of you who want to not be primarily researchers and not be primarily clinicians, but also have a focus on education. You can see the aerial is down here as one of the associate directors and please contact us at the email address below. So the academy provides a number of activities including seminars and retreats. We will provide pure observation, coaching, and mentoring for teachers, help with career development, support research and scholarship through pilot grants. We also have statisticians and qualitative researchers who can help and we are trying to boost recognition of those individuals who make a contribution through teaching. So in 2008, we had about 100 members. We're now up to 259. I'm very proud to say that we have become much more interprofessional. So you can see that we have 40, 15 nurses who are currently members of the academy. So I'm to what really drove us to put this program together and that's, I don't know what Dr. Burdy's here, there was no pun intended, but to discuss pain points in graduate medical education training. And even though it says GME, I think that all professions who are in this room can relate to the same issues. The first pain point is work compression. So our cases are getting increasingly complex. There's an increasing burden for volume. And this means that there's less time to teach. And some of us, I would say this is maladaptive, but some of us adapt by doing a minimum of teaching. It's also difficult to find time for feedback. At the end of the day, people are splitting off to go to different places. Some people are on call. Other people get pulled to other cases. So it's difficult. So if I rate surgery and anesthesiology on the pain scale, you're really quite up to the top. The other pain point is that we have limited contact with an increasing number of trainees. So it's difficult to even know where people are in their training. And therefore it's difficult to figure out what to teach them and how we can entrust them, how we can figure out what they can do. And for this, you guys are off the scale. You might see a training for one day and never see them again. So the learning objectives for this session, we want you to be able to understand the current thinking in the process of entrustment, how trainees can demonstrate entrustability and how faculty can assess it to optimize learning. We also want to give you some examples of options for feedback and physical setting how you might build this into your workflow or use apps to ensure this happens. And then if there's time, I'm going to address the issue about lack of continuity. Do we need another kind of handoff, not of patience, but of trainees? So there's continuity. So with that, I'm going to turn the podium over to Ariel. Ariel, it's you. I can go. Go on. All right, thank you for having me. I am honored and humbled to be here. As Alan said, I am one of the pediatric hospitals here at Children's and I also wear a bunch of educator hats and so may intersect with you and any of those domains. And I'm going to spend the next 15 minutes or so talking about entrustment, which is a particular academic interest of mine. But before I'll even tell you what entrustment is, I'll try and make the case briefly for why it matters. And I won't be labeled this point, but over the last several decades there's been an increased emphasis on patient safety, which has been welcome. But with that, there has been a change in the way that we train our residents and our fellows and our medical students. And there has been a call for increased supervision. And I think this is particularly true in the pediatric fields. And so what it has left us with is a tension between the safety of patients during training versus the safety of patients after training. Because on one hand, if I was only worried about the safety of my patients during training, that I might afford a lot less autonomy to my trainees, trust them less, in the name of sort of keeping that patient safe. But it then leads to a problem because if we're never giving autonomy to our trainees, then they might not be able to develop enough competency by the time that they are done training to safely care for their patients, which is why we're left with this tension. And in fact, across specialty is nearly in every field in pediatric surgery and anesthesia. There's been some sort of study, which they have asked residents, and then they have asked program directors, at the end of your training, do you feel prepared for independent practice and by and large, that answer has been no, which is a problem because we need to be feel like we are graduating competent physicians. And so a way to frame this problem and a potential solution to this problem can be understood through this educational theory called the zone of proximal development, which I'll describe to you. But if you can imagine in this sort of bluish purple circles, that this represents all the tasks that a trainee could do. And in the middle is this sort of deeper, bluer circle. And these are the tasks that the learner can do unated, that they have mastered. And we spend a lot of our time and training in this area, and that's quite appropriate to be doing what we are able to do. But if you contrast that to the outer rim here of this sort of lighter circle, and these are sort of the tasks that a learner can't do or shouldn't be able to do yet, because they're not even close to being competent. And then if you look in the middle, there's a sweet spot, which is called the zone of proximal development. And this is the area of tasks that a learner cannot, has not yet mastered, but that they can do with guidance and with help. And it is extremely important that we try and get our learners there as much as we possibly can, because this is the only way that we can increase the size of this inner circle, this dark blue circle. And so, as much as we can, in order to increase the competency of our graduates at the end of their training, we want to be in the zone of proximal development. And this theory is not unique to graduate medical education. I have a first grader. And in their classroom, they would call this their learning edge or their growth edge. And so it's important for my son, Jonah, to not be just be doing the same simple math over and over and again, that with his teacher's assistance, he starts doing more complex math, multiplication, et cetera. And that's how he's ready for second grade or whatever he's supposed to do by the end of first grade. And so in order to be in the zone of proximal development, we need to be able to entrust our learners to do different patient care tasks. And so, entrustment can be defined as the act of confiding the care of a patient or the execution of a patient care related activity to a learner. So in my clinical day, I am making thousands of entrustment decisions with my learners. And I am sometimes conscious of them and sometimes not. So for example, I am deciding whether the medical student that I'm working with is able to take a history by themselves without somebody else in the room. I'm deciding when I have a resident, and they're going to do a lumber puncture, whether I am standing next to them, whether I am in the back of the room or whether I'm not in the room at all. I'm deciding whether one of my senior residents is able and competent, and I'm going to trust them to deliver a new cancer diagnosis, for example. And these entrustment decisions are not unique to my job as a hospital. Everybody is making them. They're just feel a little bit different depending on what your specific job is within the hospital. And where entrustment is related to the relationship between a supervisor and learner, so me as a supervisor is going to trust a learner, trustworthiness is actually inherent to the learner. And so can be defined as characteristics of a learner or aspects of a learner's performance that would make a supervisor want to place trust in them. And we'll talk about that a bit more. And so then the question is how are these entrustment decisions made? If I'm trying to convince you that in order to be in the zone of proximal development and eventually graduate competent physicians that we need to be able to trust our learners when trust is deserved, how are these decisions made? Ten Kate is one of the leading researchers on this, and he describes sort of five domains that have factors that relate to entrustment. And so the supervisor is important, so from me as a supervisor how experienced I am I, how comfortable am I with the task that I'm going to trust somebody with, how risk averse am I. Then there are factors related to the trainee, how competent are they, it's simplest, there are factors related to the relationship, and then there are factors related to the task and the context itself. So for example, for the task, so how complex is the task, so of course I'm going to trust somebody more in a more straightforward task than a more complicated task, how complex is the patient, how urgent is the task, how frequent do we see the task, and what are the risks for complications. And then there's the context of the circumstance, so there's culture, there's resources, there's staffing, so our pediatric residents love to sort of poke in tease at the fact that we give them very little trust or autonomy during the day when we are fully staffed, but on the nights and weekends we are more likely to trust them just by the way that the hospital is staffed and then competing tasks as well. The task in the context is somewhat out of our control, although that point can be argued, so what I want to spend the rest of my time doing is talking about factors related to the supervisor and the trainee and the relationship between them to that can help promote when an interestment is deserved, and I should say that lens is that the point of this is not just to trust our trainees to do anything, it's to trust them when they deserve to be trusted. And it's important to acknowledge the tension that exists and it's been described in the literature and I think we all feel whereas trainees often feel, gosh, I really want more autonomy, I want to be trusted more than my faculty member is willing to give me, and faculty are often feeling, oh gosh, that trainee wants more autonomy and wants to be trusted more than they deserve and this tension exists. So now I'm going to talk about the trainee perspective and I'm going to tell you a story to sort of talk about this, and so I can tell you a story about resident A and resident B, and these pictures are not true of these people, but I want you to imagine that resident A and resident B come into residency or fellowship, whatever you want to imagine, and they are equally competent, okay? So they have great medical knowledge, they are great problem solvers, they have great technical skills, but in this case resident A makes her way throughout residency, and she is trusted all the time to do things, she gets to do many more things unsupervised, gets to do more in the operating room, more wherever the task is than resident B, and I think that this both from a trainee perspective, I remember this feeling in seeing co-residents and I remember this from faculty, there was just these residents were able to be sort of make their way through and be trusted more, and what is it about resident A that makes her so successful? There has been some literature on this that basically described four domains of trustworthiness, and the first is the obvious one, this is a competence, right? So what is the medical knowledge like? What are the problems of solving abilities like reasoning and technical skills? And this is obviously incredibly important to trustworthiness, but then there are these other domains, so the second one being truthfulness and honesty, and I think that this can be best understood in the converse, so if I ask a trainee, did you get consent for this procedure? And they say yes, but the answer is no, then I am unlikely to trust them again in the future, so honesty and truthfulness is important. The third is conscientiousness or reliability, so how likely is a trainee to do something that they say they are going to do? How detailed oriented are they? Are they going to dot their eyes, are they going to toss their teeth, or are they going to let things fall through the cracks? And the last and arguably the most important is this idea of recognition of limitations and willingness to ask for help, so not only do I understand where I'm getting past my zone of proximal development, am I able to acknowledge it and ask for help in those situations? And so we said, gosh, all of those things are important, but what if resident A and resident B are still sort of equally trustworthy? What is it about resident A that then pushes them towards entrustment? And so in order to answer this question, we did a research study where we talked to a bunch of faculty and a bunch of residents and qualitatively analyzed those transcripts using granted theory, and we found that there were basically two other pathways that were important in this moving from trustworthy needs trustworthiness to entrustment. And so the first is intuitive and obvious in a lot of ways, but that demonstration was important. And so there were some trainees who were superstart smart, super, and a bunch of knowledge and good problem solving, but they were never able to sort of demonstrate that to their supervisor. And then there were the residents who on rounds, for example, would be able to get to the right answer, but be able to describe their reasoning, et cetera, and they were able then therefore able to be trusted more. And so there's this idea, this active demonstration, that was a critical key that some residents were missing. And when you think about recognition of limitations and willingness to ask for help, so some sample behaviors that would demonstrate that you had that component of trustworthiness would be to ask insightful questions, just simply to ask for help when I'm sure, or to establish expectations with your supervisor about comfort level with handling situations in your own, maybe even beforehand. So this is the idea of active demonstration that is important. And then we found that there were these several modifiers that could either push you towards entrustment or away from entrustment. And these are self-advocacy, self-management, relationships, and patient-centeredness. And I'll talk about each of them. So self-advocacy is proactively asking for entrustment or a dependence by showing interest, taking initiative, or establishing goals related to entrustment with a supervisor. So as you probably all can remember instances where you had a trainee who was rotating through anesthesia, for example, and they wanted to be a future anesthesiologist, maybe a medical student, they showed lots of interest, they came prepared, they had done reading, and you're more likely to trust them. And the same can be true for my trainees. So if I have a patient who needs a lumber puncture, and my intern comes up to me and says, gosh, I would really like to do this lumber puncture. I've watched too, I've done a simulator, I have all the materials, I'm going to walk you through what my approach is going to be, I'm much more likely to let them do it than to the intern who's just going to sort of be thinking those things, be just as competent, but not be able to show that initiative and advocate for themselves. And obviously, this can go the other way, if somebody is asking to do things that they are clearly not competent to do, that that can push me away from entrustments as well. The second thing is self-management, defined as the demonstration of personal qualities or actions that influence how one has received in the work environment, such as emotional intelligence, ability to manage stress, confidence, and humility. And there are residents in this, it might have been the case for resident B, who when she is stressed, when she is busy, when her work compression is so much that she falls apart in some way, she shows it, she becomes snippy, she becomes short. All of these things can make you less likely to be entrusted. And the conversation is true, so the resident that in the middle of a code or stressful surgical situation can appear calm and collected, they are more likely to be entrusted as well. And so this is how you are projecting yourself to the outside world. The third is relationships, so the training ability to build and invest in relationships with colleagues, patients, and their families. And I think this is sort of best demonstrated by just saying that if I am taking care of a patient and that patient really seems to trust my trainee, the nurse that we're working with, trust that trainee, if the trainee has developed all these relationships of trust already with other people, I am therefore more likely to trust them, which in the converse obviously being true as well. And the last is patient centeredness, and this is prioritizing patient care and demonstrating that priority by making decisions based on the best interests of the patient, taking ownership of every detail of the patient's care, building a relationship with the patient and family, and spending time at the bedside. And so all of these things matter. And I think what is important to say about this is that I think that some of these things are intuitive to trainees and some are not and could be better explicitly taught to our trainees throughout their training. We are not the only people that have done research in this area, so this is from the surgical literature, so this is called the OPTrust tool, and it describes both resident and faculty behaviors that lead to increase in trustability in the operating room. And there are a few domains here, there are types of questions asked, operative plan, instruction, problem solving, and leading by the surgical resident. And so if you take the lens again then of the resident, I'll just going to use one of these domains as an example, but for types of questions asked, there are low interestability behaviors medium-high and full. So for example, a low interestability behavior by a resident would be, does not ask that's a question for majority of the case, whereas a high interestability behavior would be, ask questions about upcoming steps in the procedure or procedural flow. So it would not be unreasonable that when you're training a surgical trainees or you're anesthesiologist, that you would essentially be looking at these behaviors and trying to coach these behaviors. So we talked a lot about sort of the training lens or the resident lens, and so what can you do as faculty? And so for one, you can help try and teach and foster and give feedback about some of these softer skills, or not softer skills, which is important skills that are under-toss, but then there's also behaviors that you can do yourself. So if you took the same tool, you could then look in the faculty behavior section, and so I'll just, I won't go through all of them, but I'll give you an example. So for type of questions asked, there are low interestability faculty behaviors, and then there are higher, full interestability behaviors. So a low interestability faculty behavior would be, does not ask resident questions for majority of the case, whereas high would be to extend knowledge with open ended questions. So I think that there are behaviors that you can do yourself that were foster interestability. And this study was really interesting, and I made this graph up because the actual graph is too complicated, but basically they looked at faculty and trustability behaviors and how they correlated with the resident and trustability behaviors. And they looked at a bunch of faculty, a bunch of residents over a bunch of cases, and they showed that faculty that had higher and trustability behaviors were more likely in that same case to have residents that showed higher and trustability behaviors, and sort of suggested or hypothesized that actually faculty behavior could influence resident behavior. And I think that the conversation is likely true, that the resident that comes in and has these strong and trustability behaviors will also likely influence faculty as well. And they looked at a tragic control for a bunch of things, and also found that it was not associated with faculty experience, which was interesting. So now I'll leave you with two other thoughts, which I think will be, this first one will be tied in a little bit later. But familiarity matters. So this study, they looked at roughly 50 residents, 75 faculty, 225 cases, and they asked faculty how familiar are you with this resident prior to the case, and then they used that opt-trust tool to talk to look at the faculty and trustment behavior, and not super surprisingly, they found that faculty that were more familiar with the residents, in those cases, they were more likely to show this higher and trustability behavior. And so this familiarity matters, which I think is quite intuitive and quite a challenge in surgery and anesthesia when you may have shorter experiences with any individual training. And the last thing I will throw out there is the issue of gender bias. And so I think we see now, see these studies in the business world and in medicine all the time. But this was a very large study across 14 sites, which looks at, you know, roughly like 500 to 800 residents, faculty, you know, a bunch of cases, and they basically showed that when you control it for almost everything, that male residents were more likely to have operative autonomy than their female counterparts. And this was most true in the last year of training. And so I think this is just something that we could think about as a potential bias in intrustment. So I'll conclude there in my sort of messages, if you can remember, too, are that as much as possible, trainees should exist in their zone of proximal development, and that training and faculty behaviors, influence intrustment decisions. And I do believe can be taught but are usually under emphasized. Okay. So I'll conduct about three passion as well as pain points for me and for many of you as well, teaching assessment feedback in the OR. So I'm not going to stand here to say I'm an expert at balancing this chair of teaching in the OR. I know many of you who do a lot better than I do, and I've learned from you how to do some of this. But my goal of this talk today is to share some ideas and suggestions on how to make this easier, not to fall off this chair. Now, I'm a big believer in in order to get to where we want to go, we need to be able to measure ourselves and assess our progress in order to get to where we want to go. And the quality of this assessment is very important. We'll get into a little more detail about that. But I think that in order to get to where we want to go quicker and better, we need feedback. This has been true in every single discipline you can look at. Feedback just makes you better. It's just like strapping a jet engine on a fast car. It makes you better quicker. So I know we're all busy. And this is how I feel some days with, and I'm sure you do as well. And so how do you fit in teaching in the OR within our clinical busy practice? Couple suggestions for you. One is making an assistive, madic approach. I know most of you already do some of this. And this is a framework that a very simplistic talks about briefing and trap teaching and debriefing. And I know you all do this. That you may do one component of it. You do two of them. You may do the last one. Let's try to find a way to make it a systematic and incorporate all these three elements. And I'll show you how. So anesthesia team probably does a better job than the surgeon. Because they often talk to their fellow or resident in advance. They already know the case they're doing. For surgeons, we often don't even know who the resident is going to be helping us or the fellow is going to be that day. So it's hard to prepare ahead of time. But still, I think if you find a time ahead of the patient interaction to talk about it, to establish this learning objectives, to establish a degree of trustworthiness of this resident or faculty. To see how much in trustment I'm going to have with them going back to what Ariel was telling, set those expectations for that level of involvement and establish trust and climate. I think that'll lead you into this zone of teaching. And to be able to target and focus on those objectives that you negotiated already with the resident. What are we going to work with? What do you want to get out of this encounter? What is that zone of proximal development of that resident that we need to focus on? And then at the end, don't wait until you're like, okay, let me go talk to the family and then come back and we'll talk about this because that's not going to happen. You're going to move on to the next thing and the opportunity for feedback is going to go away. So while you're still in the case, after that zone of the intense moment of the case when you're closing skin or doing something, ask the learner to self-sense, how do you think you did? What can we do better next time? Address the concerns and always try to provide a take home point. What is the rule of this encounter? When you face with the situation again, what are you going to try to do? Okay, plan for next time. So moving on to assessment. Not that long ago, this phenomenal group of surgeons trained in pediatric surgery. And if you were to develop a way to assess their performance as a fellow, you'd probably start off with the paper form, right? And so this is an assessment form of an inguinal herniar pair in a child. And you know, as the form goes on, you can see it's sort of five pages long, has a lot of sort of words on it and a lot of anchors to it. And the most probably valuable part of this form was these last two boxes, which free text, right? But unfortunately, the majority of the time, these are less empty. So that up to you for feedback was not the best. And I was in this sort of cohort of between, you know, 2014 to 2017. And we were still using these form that we, on average, had only about 9.6 evaluations per fellow per year. We operated a lot. That's only about 1 to 2% of all eligible cases. So I think that's a lot of lost opportunity for assessment. Yes, we got feedback throughout training. Yes. But this assessment of where do we stand was very, I think, subpar in terms of just documenting where we are. And the reason why is because every time I would, you know, come up to one of our faculty, not always, but you would get this face of, oh my god, five pages long. Oh, hey, leave me the form. I'll fill it. I'll give it back to you later. And then so you can envision how that goes. And we're busy, right? And the thing is we need a lot of assessments to get reliable estimates. You know, research has shown that about 60 ratings are needed to get reliable autonomy ratings, about 40 for performance ratings. So with just 9.6 per fellow per year, we're never going to get there. Okay. So I have a wish list of what an ideal operative assessment look into. And I'm not going to go into detail with that. But I do have a, we have been using an option that gets pretty close to this wish list in my, in my book. And it's called, it's an app. It's a simple strategy. And it stands for System for Medjuim Procedure Learning. It's based off a nonprofit research collaborative called PLLC. And it essentially goes with three questions. The autonomy or the switch scale, which will go into detail, which is the meat in the backbone of this assessment system, performance scale, and the difficulty scale. It's allows the opportunity for dictated feedback at the end of the assessment. It takes less than 20 seconds to complete literally just like a text message. And it provides the opportunity for the trainee to self assess themselves as well. So it's a, for the most part, an effort based off a general surgery that 11 different sub specialties, including pediatric surgery are involved with this. There's more than 2400 trainees within this effort. So it's a large effort. Now this wish scale. So Dr. Schwishenberg, he goes by swish as a cardiac surgeon out of Kentucky. And he has been using a version of this scale for about more than 20 years in his practice. And he, instead of focusing on what the resident was doing in the OR, he switched the tables a little bit around and said, okay, how much am I doing as a faculty, how much guidance am I providing in order to know how good this resident is? So that's the key difference or change in framework in the way we think about assessment. And so he says, okay, so if I'm doing everything, that's sort of show and tell. If I'm sort of leading the dance, doing the majority of the case, it's active help. If the resident or fellow is leading the dance and taking the operation on their own, pretty much on their own, that's pass a felt. If I'm there just there, either a computer signing notes or looking over their shoulder and not doing much, that's sort of supervision only. And the key message here is that guidance does not have to be supervision is independent from guidance. That's one key thing because some of you might be thinking, well, I'm not going to completely let loose of the control operation. That's true. And I have a hard time doing that as well. But I think you have to separate those two things that as guidance can increase or decrease, supervision can be constant. Okay. So a few of the minutes just to illustrate how these work. So the attending here screen left and the resident in term. So show and tell exactly how it looks, right? So the attending is demonstrating and explaining the steps of the procedure, doing everything essentially. And the resident or intern or fellow is watching the system sort of the first case or the situation or the first time you're working with them. Active help as an attending, you're the one with the right angle, you're the one intubating, you're the one controlling the anesthetic and the resident. Maybe doing some of the technical skills, maybe putting some stitches, maybe even the doing the anestimosis, but you're exactly telling them where to put the stitch, how to put the stitch, what goes next, you're leading the dance. That's the key with active help. Passive help, things change a little bit. You're there, you're scrub, capable first assistant, you're attracting, you're making it look easier for them, you're with the sucker, et cetera. The resident of fellows leading the dance, they're in a little more control that you're there. Supervision only as it sounds, you know, you're supervising, you can be scrubbed or not scrubbed, it just depends on your personality on how you want to manage that. But I want to get to this point that yeah, we all want to be independent as fellows, but it's more of a near independence because you're still there as a faculty, you're still supervising. So it's this concept of mimicking independence that being there is supervising, right? So I know a procedure as well as an anesthetic has different stages and you're probably wondering well, what if there are one point in the operation really, really good, but the other two that just don't know what they're doing. We try to think about this in greater than 50% of the operation, the critical portion, what were they at? This particular resident, for example, in the meat of the operation, yes, he was able to open the abdomen on its own, close skin on their own, but in the middle of the operation, he was passive help for the majority of it, had a little active help here for this critical part, but was able to do more than 50% in that critical portion as passive help. So that would be passive help. Now we've been using this system at children's for the pediatrician fellowship for over the last two, almost two and a half years and in this two-year time frame, we've accomplished about 1,200 assessments, completed assessments, 22 faculty for five fellows. That goes up to about 206 evaluations per fellow per year, it's about 40% of eligible cases, still far from perfect from ideal, I would left them see 100% of eligible cases that were getting there. We're working on it, but it's a lot better than 1 to 2% of eligible cases, and I'll show you why. It captures a lot more cases, a lot more, so more than 200 unique procedures. It doesn't, it's not a form that sits on somebody's desk for a month and then gets turned into the program coordinator two months later. You get the feedback right away, an average response time is about in a half hours, but anything from a few minutes to a day or two, there is a time limit on the assessments, they time out after 72 hours because we have done some research and there is a substantial amount of recall bias if you wait more than three days. This is sort of the more useful thing for trainees, I think, and also for faculty as you reflect on it, a thick, dictated feedback at the end of that assessment is key. I'll show you sort of why. On average, about a little more than half have been getting dictated feedback. It's interesting to see that the first year fellows are more likely to get dictated feedback than the second year fellows, significantly. I think that goes with trying to guide that first year fellow a little more to get them where they want to go. So not surprised. The tool has a really good job of distinguishing between different levels of performance. I think when you look about meaningful autonomy, which is anything above passive help and supervision only, it does a good job. Second year fellows consistently rated to be with meaningful autonomy and practice ready performance. The way our surgical training instructor, here, first year fellows and second year fellows, for the most part do very hard cases. I think there's not a lot of difference in terms of the case complexity between the first year sick of your fellows and this is sort of demonstrated by the complexity scale. This is one of my favorite grass because it sort of shows the concept that I like to call the autonomy set point or where do you start? Each line represents a fellow. Some of them are standing here in the front row and blue the first year fellows and you can see when they start, the amount of ratings that they're coded as meaningful autonomy either passive help or supervision only. It's roughly about maybe 30 to 45% of the cases because they have their general surgeons. They come very well trained. As they progress over the academic year quarters, you can see how that quickly goes up to the level of the starting second year fellow. You can envision a way to potentially establish thresholds to detect opportunities for early remediation. Let's say if you at the third quarter, if they're not above 60%, you start to get worried or if they're not at above 65% at the time of their second year, you may want to intervene on something. Same thing for that graduating second year fellow, if they're not above 90% for example, in their percent of autonomy ratings, then there may be an issue there. I think this allows you to track and detect early opportunities for remediation. Now, we do a lot of exotic cases here. I'm very complicated. You wonder, is a trade complexity going to mean something for an assessment like this or not or etc. etc. But the majority of the assessments actually is a pretty much a bread and butter. It's G-tubes, Pellaromy, Central Lions, Ingolharians, even Appendectomies. Those are the majority of the producers that got assessments completed on. When we look carefully into one of these, the Pellaromy Learning Curve, I like to see this, I'm going to show you three lines of three different fellows. These are the ones that we have data on from the very beginning, the very first essentially Pellaromy. This is one fellow. You can see how their first Pellaromy, you've never seen one, never done one potentially, and pretty much show and tell. A couple of active help and then a few going up and down between supervision only and passive help and then consistent supervision only after the tenth. Now, look at the second fellow. Show it on active help back and forth, back and forth, back and forth. Then this consistency can sort of start to see it happen. This fellow hasn't recorded any more assessments, but we're going to keep tracking this. You can see how you can start to get a sense, this is a third fellow, probably got up a little quicker than the other two, but still a little less, not that consistency quite yet. You can start to see how over time and with more assessments, you can start to develop a sense of what's the minimum number of cases that you truly need in order to develop some degree of competency for this procedure. Now, since this app allows you, I'm going to now be updated to the trainees to self assess in some self introspection and to compare themselves with the faculty. We're very interested to see how those comparisons between the fellow and the faculty were, and on average, the faculty are actually rating the fellows a little higher. So I think our fellows here are a little humble and they don't think as highly of themselves as we think of them, but overall a very good consistency between these terms of interquest correlations. So not a whole lot of difference. We surveyed our faculty and our fellows at the end of the first year for some comments, some insights on how this works. The majority says that they're willing to take up the minute to complete the assessment. It takes 20, 30 seconds, so I think we're pretty good. The majority do agree that all cases should get an assessment as opposed to only few index cases. The majority love the dictated feedback that's for the most part the most useful thing, I think. Some dislikes see people don't like that 72 hour time limit. Some people initially we didn't have any reminders on a daily basis. Now we do. It was initially not available to case log at the same time, or and they sort of need an internet connection at the beginning, but now we send daily email reminders. Now you can log your cases with ACGME, and now you don't need internet connection because that particular moment in time for the assessment is it logs it after when you get internet connection later, but you can complete the assessment. So a few other barriers to use, mostly behavioral. It's very interesting to see how the fellows would change their behavior on who they would send those assessments to based on if they got dictated feedback or not, or whether they had responded to previous workquest. And for the most part, the majority of people just sometimes forget. And I think that's where the the reminders are key. This is like this graph as well because you can see how the faculty's, oh yeah, we always get feedback, oh yeah, always. You compare it with what the fellows say and they say that well sometimes they're rarely, right? So completely the opposite. So my point to the beginning was if you don't specify it, you know, make it a systematic approach that hey, I'm giving you feedback, then they're not going to recognize it as such. I was worried about the impact of the implementation of symbol on face-to-face feedback. I think that it doesn't replace face-to-face feedback. Don't take this as a replacement for that. I was very happy to see that some faculty felt that it actually made it better because it provides this framework and to have a discussion and say okay, in order for me to move on to supervision only next time, this is what I want to see you do or why was I being active help in this case and not in that case? So it provides this framework to have a conversation, it's a common language, right? A few comments. Some faculty brought the point which I thought was very good about well, fellows don't send me the evaluations when I see them struggle. Well, now they don't have an excuse because now the faculty can also trigger the evaluations. And so even if the fellow forgets to send the evaluation faculty can also do that if you really, really are urging to give feedback. This goes again to those comments that I provided earlier where the fellows really want the dictated feedback because you can get a opportunity for improvement based on your scores that if you don't get a not how to make it better than that really sort of you're losing that opportunity, right? So to wrap up these pain points and suggestions and solutions just establish a routine for teaching, make it systematic. The BADM model is just one of those. You may have a different model that make it routine, establish it in your practice. I do think assessment is key to understand where we're standing, understand where we are. And I think the simple app is a tool. It's not perfect that I think it definitely increases the quality and potentially the quality of the assessments and feedback. It does provide a really good opportunity for longitudinal performance assessment to detect early remediation opportunities, potentially even make decisions about PGI advancement and graduation. We need a lot more data in order to support that bit. I think it also will serve itself as a great platform to be able to identify sort of the minimum number of cases needed for particular procedures as opposed to just taking that out of your hat or your sleeve in terms of thinking how many procedures you need. Feedback is very important and I think it's key for us to be able to get to where we want to go and it really is the most valued item by trainees. So with that in mind, give it up to Alan to wrap it up. So the original reason that I was invited to give this talk was to talk about educational continuity. And this is something that is actually going away. In the old apprentice model, you might spend all your time with your supervisor, but now it's really only in the longitudinal ambulatory experience that we see a lot of continuity and you might work with someone once a year and our GI fellows do this for three years. Inpatient rotations have gotten shorter. So used to be four weeks was the standard and two weeks. Now we're at about one week of time and obviously the extreme example is the OR or emergency department example where you might work with someone for a day at a time or even a single case. And that stresses the system because continuity is helps in the establishment of relationships and it's been shown that having a relationship really is key to all the aspects of teaching in terms of determining what teaching activities you do. We've heard Ariel talk about intrustment, being able to assess someone, and also how to give them feedback. It's better if you have a relationship with that individual. So there is some data on this and this is not what you guys deal with. This is an internal medicine rotation where they went from four weeks to two weeks and then surveyed the residents in the faculty. So the resident said that having a shorter rotation had a negative feedback on the ability for the faculty to evaluate the resident on the team's work process, on continuity of care and length of stay, and actually the faculty agreed with all those things. But what was different is the two week rotation had a positive impact on faculty personal life, wellness, and overall faculty productivity. It's unlikely we are going to go back to longer rotations. So part of this building, a relationship depends on getting to know your training. And ideally maybe we would want to know background, what program they are in, because different people coming from different programs have different experience, the level of training, did they take a unique pathway? What are their career goals so you can help them move forward? Maybe status of the training with regard to anesthesiology or surgery milestones or intrustable professional activities. And what the trainee wants to learn in the context of your work together. So all these things would help build a relationship. But what do we get usually? And this is not only in the OR, but it's on the words as well. We get, oh, how do you pronounce your name? And what program are you from? And that's about it. So the question and the original reason I was asked by Andres Navedo was because I was thinking about creating a trainee handoff. Not only should you sign out patients, but maybe we should sign out trainees as well. The problems with that is where is the information kept? Is it complete enough to date? Is everybody filled out their evaluations? Who controls access? When will the handoff take place? We have enough to do without training handoffs and how much time it will take. So I don't think that's really a practical model. But there is one system that's used in the hospital. And it's based on the Toyota production system, the Kanban. Kanban is a card that's used to signal on the assembly line whether there's a need for parts or support. So if you're bolting a seat to the frame of the car, you want the right number of bolts. You don't want too many bolts that are going to take up too much space. You never want to run out of bolts and stop the assembly line. So when you reach, you've used a certain amount. There's a card that goes centrally and says delivered 20 more bolts. And so this is the Kanban. And Stuart Goldman, who is a psychiatrist here, said maybe we need an educational card. And so the educational Kanban for psychiatric trainees is an electric document that's owned and accessed by the trainee, personal log of what they've accomplished and what their learning goals are. It's reviewed with the attending at the start of the rotation. And then the goals are made for the learning collaboratively. And it's regularly updated by the training. Don't think it's practical for use in the OR. So here's my solution. And I think this is in keeping with what you've heard from Ariel. And then the and it's called, I've named this accelerated relation building. And you do this in the briefing. So the training needs to be able to communicate what they want to learn, what their learning goals are. I've never taken, I reviewed the cases for tomorrow. There's a patient with a pacemaker who's going to need to go under anesthesia. And I've never had an experience doing that. Can you help me understand the ramifications of that when we do the case tomorrow? That requires that the trainee have self-directed learning skills ability to self-assess and address knowledge gaps and a learning mindset that I'll talk about in the future. The teacher needs to establish a psychological, safe learning environment, a problem for us at children. And needs to collaborate in setting learning activities. Well, actually there are two cases, two people with pacemakers. One is less sick and we can have more time to talk about that and teach about that case rather than the other one. And is entrust the learner and gives feedback that are really based on the learning goals. And another approach to this really would be to have a coach who's not the primary one teaching but keeps track. So we're starting to do this in GI. The fellows learn about how to do procedures from multiple people but the coach makes sure that their overall progress is moving in the right direction like if you review the simple apps to see the progress. On the teacher side you have to make sure that you don't have any bias either based on the positive or negative impression of the trainee or the fact that they may be like you and your more inclined to be positive. And on the learner side there are a couple things to watch out for. So this green line is performance. So there's a bottom quartile, a lower quartile higher and the top quartile. And this is performance. And you can see that when trainees rate themselves the people perform less well have the least perception of where they are. And this is so if I ask the audience how many people think they're better than average at what they do. Everybody in this room is going to raise their hand. We all have a inflated somewhat inflated self assessment but this can be a problem especially for early learners. And then the other problem is that we have we older folks in this room trained in the era the fixed performance mindset and this is Carol's Dweck's work forgive me for watching but I know you have to get to the OR in two minutes. The fixed is where it's most important for you to demonstrate good performance. So you can't show weakness. You are want to get an anesthesiology position at this hospital and you don't want to show the people you're working with that you don't know something. And we have to move from that to the growth mindset where learning is most important asking questions is not a sign of weakness actually it's a way of learning and assessing people and the first objective is to improve not impress. So what are the practical opportunities I think you could use the evening check-in or first day of a rotation to exchange the information that's relevant and trustment and to jointly plan learning activities the briefing. Consider options to improve feedback deep briefing give the feedback when you're closing give the feedback before the the wake up of the patient so you make sure that it gets done use faculty development to standardize the commitment to teaching maximize teaching efficiency which we really haven't talked about improved trustment and assessment skills and orient the trainees so that they're better able to self assess and set learning goals and that they have a learning mindset and so they're seeking feedback rather than avoiding that. So in summary making and trustment decisions can be made easier when the trainees demonstrate certain behaviors and we elicit them as faculty members the use of simple approaches not upon to building feedback into the OR routine can improve the educational experience both for trainees and supervisors and helping trainees self assess accurately and understand learning needs maybe the solution to the lack of continuity and supervision because I don't think we're going to be able to go back to longer rotations. So I hope we've given you some information that's relevant to all your professions and disciplines and that will provide some theoretical basis but really some practical approaches as well so thank you for your attention. There's not really time for questions so if you have any questions and you're not going to the OR please come up and talk to us.