Outlier experienced surgeon’s performances impact on benchmark for technical surgical skills training

Prospective, randomised and blinded clinical trials in surgery and interventional cardiology demonstrate that proficiency based progression (PBP) training produces a superior intra-operative skill-set.1-5 The PBP approach to training relies on a robust, quantitative assessment of experienced operator performance to establish a benchmark.  Atypical expert performances can lower the benchmark level that trainees must demonstrate in simulation before progressing to in vivo performance.

This study published in the ANZ Journal of Surgery (https://doi.org/10.1111/ans.14474) quantitatively addressed the issue of benchmarking based on the consistent performances of experienced surgeons.  It indicates an approach to dealing with the performances of experienced surgeons that differ significantly from their peers.  The methodology described in two studies quality assures proficiency benchmarks, which trainees must demonstrate in simulation, before progressing to in vivo performance, by ensuring that the competency benchmarks set are based on experienced surgeons who perform unvaryingly to this level of competency.

The statistical approach used in both studies demonstrates an objective, transparent and fair approach data falling > 1.96 standard deviations from the mean differs statistically significantly and as in this case should be excluded from quantitative benchmark definitions. The studies showed that objectively assessed atypical expert performances were few and Z‐score standardization identified them and produced a more robust quantitative definition of proficiency.

  1. Cates CU, Lönn L, Gallagher AG. Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simulation and Technology Enhanced Learning. 2016:bmjstel-2015-000090
  2. Angelo RL, Ryu RK, Pedowitz RA, Beach W, Burns J, Dodds J, et al. A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic bankart skill set. Arthroscopy: The Journal of Arthroscopic & Related Surgery. 2015;31:1854-1871
  1. Van Sickle K, Ritter EM, Baghai M, Goldenberg AE, Huang IP, Gallagher AG, et al. Prospective, randomized, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying. J. Am. Coll. Surg.2008;207:560-568
  2. Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hogman C, McClusky DA, 3rd, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. The American Journal of Surgery. 2007;193:797-804
  1. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, et al. Virtual reality training improves operating room performance: Results of a randomized, double-blinded study. Ann. Surg.2002;236:458-463; discussion 463-454
/