Defining the cardiac surgical learning curve: a longitudinal cumulative analysis of a surgeon’s experience and performance monitoring in the first decade of practice

Abstract

Background

Individual surgeons’ learning curves are a crucial factor impacting patient outcomes. While many studies investigate procedure-specific learning curves, very few carried out a longitudinal analysis of individual cardiac surgeons over the course of their career. Given the evolving landscape of cardiac surgery with the introduction of transcatheter and robotic procedures, a contemporary evaluation of the cardiac surgical learning curve is justified and a method of personal performance monitoring is proposed in this study.

Methods

A retrospective study of 1578 consecutive patients of a cardiac surgeon over ten years was undertaken. Risk adjustment was based on Euroscore. Cumulative risk adjusted morbidity (CRAM) charts of operative mortality, return to theatre and length of stay were constructed. Secondary endpoints included postoperative stroke and deep sternal wound infection. Change-point detection was applied to investigate temporal trends and identify when a significant change in outcome occurred. Multivariate analysis was performed to assess the influence of patient and system factors on operative mortality.

Results

Patient average risk profile was highest in the later years of practice. Cardiopulmonary bypass time remained stable from 86.5 to 92 min across the decade. The frequency of redo operations increased from 4.07% in the first two years of practice to 9.29% in the last two years. The proportion of aortic surgery increased from 6.98 to 10.58% of total cases. There was a significantly reduced operative mortality signalled at case 1220 with the change point identified around case 970.

Conclusion

This prompts training colleges to consider application of sequential performance monitoring in surgical training programs, to confirm the progress of trainees and identify early evolving patterns that suggest support is required or milestones are being achieved.

Introduction

The evaluation and training of cardiac surgery has become increasingly scrutinised in recent years, with the development of databases for monitoring patient outcomes [1]. In Australia, cardiothoracic surgeons undergo a six-year accredited training program and a variable number of fellowship years, prior to the commencement of consultant practice [2]. Full proficiency in many cardiac operations remain yet to be achieved by junior consultant years, due to limited exposure as a first operator during formal training. Thus, junior consultants often remain under the mentorship and guidance of more senior surgeons until proficiency is achieved. As surgeons become more proficient over the course of their careers, the importance of benchmarking has been recognised, with many databases making available surgeon-specific morbidity and mortality rates [3].

Surgical training has been traditionally based on the apprenticeship model of training, with a lack of robust evidence in characterising the learning curve. Bridgewater and colleagues examined the learning curve effect by comparing mortality in patients operated on by newly appointed and established cardiac surgeons, finding that mortality in patients treated during a surgeon’s first year of practice are significantly higher than in the fourth year (p = 0.049) [4]. Baldonado and colleagues reported a small case series, reviewing the first 272 cases of robotic pulmonary lobectomies performed by a single surgeon, finding that there was significantly less blood loss, shorter operative time and length of stay in the last 120 surgeries, compared to the first 120 surgeries [5]. Novick et al. assessed the learning curve in off-pump coronary artery surgery, resulting in largely inconclusive findings [6]. Despite the influx of literature investigating procedure-specific learning curves, very few studies have carried out a longitudinal analysis of individual cardiac surgeons over the course of their career, especially in the most performed procedures [7, 8]. While the volume-outcome association is clearly described in the literature, many external factors such as systems aimed at recognising and responding to adverse events play a significant role in patient outcomes, rather than attributing this solely to the surgeon [9, 10]. This has not been previously addressed in the analysis of the cardiothoracic surgical learning curve. Of note, while many cardiothoracic surgical units undertake monthly or quarterly performance reviews, current practice in Australia is typically to monitor the performance of a unit or group of surgeons, then comparing it to state or national baseline performance. While unit-based performance review is of value, there is no existing routine mechanism in the current Australian practice for surgeons to monitor their individual progress longitudinally. Reflective practice is key to continuous improvement, and we aim to demonstrate a statistically valid and objective method of monitoring a surgeon’s performance. Such monitoring is not only useful to ascertain improvement, but also any decrements in performance that would be amenable to early intervention.

The landscape of cardiac surgery has also evolved significantly in the last two decades, with the introduction of transcatheter and robotic procedures becoming more commonplace. Given this change, a contemporary evaluation of the cardiac surgical learning curve is justified. The objectives of this study are to (1) evaluate the learning curve of an individual Australian cardiac surgeon over the first ten years of practice using statistical process control (SPC) analysis, (2) evaluate risk-adjusted outcomes including mortality, return to theatre (all cause) and prolonged postoperative length of stay, (3) evaluate the association between experience and risk-adjusted outcomes, and (4) demonstrating the implementation of the CUSUM analysis in individual performance monitoring.

Materials and methods

Patient cohort & data This retrospective observational study was reviewed by the Metro South Health Human Research and Ethics Committee, in accordance with the principles set out by the Declaration of Helsinki, with an approved waiver of consent (Reference Number: HREC/2023/QMS/99358). The study cohort consisted of 1578 consecutive and unselected patients who underwent cardiac surgery performed by an individual surgeon (CC) across five hospital environments, from 17th March 2013 to 5th September 2022. Data definitions correspond to that of the Australian and New Zealand Cardiac and Thoracic Surgeons (ANZSCTS) Cardiac Surgery database, to ensure consistency of data definitions. Operative mortality was defined as death within 30 days postoperatively. The primary endpoints for this study are risk-adjusted 30-day mortality, rate of all cause return to theatre (RTT) and prolonged postoperative length of stay greater than 14 days (LOS > 14 days). The secondary endpoints include deep sternal wound infection and postoperative stroke.

Risk adjustment

To account for case complexity variations, a risk adjustment score based on the Euroscore model was employed [11]. Although it was not possible to use the current version of Euroscore (Euroscore II), as some key factors were not available across the entire study timeframe, it has been previously demonstrated that Euroscore has adequate discrimination but poor calibration for the patient cohort used [12]. To address the calibration issue, data published by the Queensland Cardiac Outcomes Registry was used to develop an odds correction of the basic score to recalibrate the model for the contemporary patient cohort risk of death (ROD). Furthermore, as it has been established that there is an association between Euroscore and the risk of a range of post procedural adverse events (including prolonged length of stay) an odds correction was applied to Euroscore to generate a risk estimate for RTT and LOS > 14days.

Degree of supervision/mentorship

Throughout the period of data collection, the surgeon was the primary operating surgeon or the supervising surgeon for a trainee. The first year of practice was of a fellowship position with primary operating rights. There was a consultant on-call with whom cases could be discussed. Thereafter, there was no such supervision provided after the first year of practice. Throughout the whole decade, there were less than 10 cases where another consultant surgeon was the assistant.

Public/private practice

Other than one year of practice in Hong Kong, all subsequent years were undertaken in public and private hospitals situated in Queensland, Australia. Public hospitals in Australia consist of a large team of residents, non-training registrars, accredited registrars, fellows and consultants. These teams are selected by the director of the unit and are subject to centralised rostering systems, such that the team in the operating theatre each day varies. On the contrary, private practice consists of a stable and fixed team selected by the surgeon. A high proportion of preoperative referrals are from cardiologists with routine investigations prior to cardiothoracic surgery. All postoperative cares involve a standard intensive care unit.

Control charts

Cumulative sum (CUSUM) charts are a family of graphical statistical tools that can be used for monitoring the output of a process and although originating in the manufacturing industries, have been used extensively in various forms to monitor surgical outcomes to signal if and when sufficient evidence has accumulated to confirm to a predefined level of confidence that a change in the output has changed [13]. Each form of control chart has its own relative strengths and weaknesses in terms of statistical robustness, speed of response, false alarm rate and ease of interpretation [14]. As such, in this analysis, two forms of control charts have been used.

Risk-adjusted sequential probability ratio test (RA-SPRT)

Sequential statistical analysis of outcomes was performed using the RA-SPRT as proposed by Steiner & Cook (2000) [13]. In designing the chart, the average run length to signal (in control) was set to five years, with the superior-most line signifying a running test that the odds of the event had doubled and inferior-most line signifying a running test that the odds had halved. Upper and lower control limits have been set to flag when the significance level has been achieved.

Cumulative risk adjusted Mortality/Morbidity (CRAM) charts

CRAM charts are a variation of the Variable Life Adjustment Display (VLAD) Chart used by Queensland Health to aid in the monitoring of quality of service provision, and signals when further investigation or performance is warranted [15]. Although CRAM charts are not as statistically robust as the RA-SPRT they are visually more intuitive to read and interpret. The metric charted is the cumulative difference between the expected number of events (derived using the adjusted risk score) and observed number of events with positive inflections indicating a gain in terms of lives or events saved (the observed event rate is less than the expected rate), and a negative inflection indicating that the observed event rate is greater than the expected rate (a net excess of events). To construct the CRAM charts the first 200 cases were used in each instance to establish the baseline calibration of the expected rate of events. A CRAM chart was constructed for operative mortality, length of stay greater than 14 days and return to theatre. A change point detection analysis was performed to identify the point in time when a significant change occurred in the outcome being monitored.

Analysis

All descriptive analysis was conducted using SPSS Statistics software (IBM, SPSS Inc.). Demographic and clinical variables of patients were summarised with rate and 95% confidence interval (Wilson score interval) for binary variables or median and interquartile range for continuous variables. Multivariate regression analysis was also performed to assess the influence of various factors that may contribute to the outcome (e.g. procedure type, elective or emergency status, facility and year of experience). Firstly, a logistic regression included both the risk of death (ROD) estimate, as well as clinically important factors (age, gender, elective status, type of surgery and year of practice). As the ROD estimate includes a number of these clinically important factors, a second logistic regression was performed including the ROD estimate but excluding the individual variables of the risk model.

Results

Demographic and operative characteristics

Patient characteristics are presented in Table 1. There was a steady increase in the average age of patients over the decade, from an average of 65 years in the first two years of practice to 71 years in the last two years of practice. The average unadjusted Euroscore over one decade of patients was 4% (2.10-8.57%). While the average unadjusted Euroscore across the successive years of practice was not linear, the highest unadjusted Euroscores were in the later years of practice, with the highest unadjusted Euroscore at years 5 and 6 (4.37%). Cardiopulmonary bypass time remained stable through the decade within a narrow range, from 86.5 min to 92 min. Notably, previous cardiothoracic intervention or redo operations were more frequent in the later years of practice, from 7.27% in the first two years of practice to 19.26% in the last two years of practice. The frequency of aortic surgery increased from 6.98% of total cases in the first two years to 10.58% in the last two years.

Table 1 Patient demographics and perioperative characteristics. `expressed in n, (range) *expressed in %, (range)

Postoperative outcomes

The crude outcomes are presented in Table 1. Operative mortality remained stable across the decade of practice, from 1.89% in the first two years to 2.06% in the last two years. There was a higher rate of return to theatre in the third and fourth year of practice, at 10.82%, with subsequently lower rates of return to theatre in the later years of practice. Length of stay more than 14 days was found to be steady at 5–6% across the decade. Other than the primary endpoints listed in the objectives, other important postoperative outcomes including deep sternal wound infection and permanent postoperative stroke were also examined. The rate of deep sternal wound infection remained consistently low, from 1.0% in the first 400 cases, 1.3% in cases 401–800 and decreasing markedly to 0.3% in cases 801–1578. The incidence of permanent postoperative stroke was 1.5% in the first 400 cases, reaching a peak of 3.0% in cases 401–800, then a nadir of 1.3% in cases 801–1578. Interestingly, the proportion of elective cases were lower in the case 401–800 group (50.0%), compared to the first 400 cases and the final 777 cases (61–64%). Logistic regression of factors impacting operative mortality revealed that elective procedures had lower odds of operative mortality (OR = 0.17, p = 0.001), while valve surgery or combined valve and CABG surgery had higher odds of operative mortality than coronary artery bypass graft (CABG) alone (OR = 2.76, p = 0.049; OR 5.65, p = 0.002 respectively) (Fig. 1). Accounting for risk of death, each separate year of practice did not show a significant difference in mortality outcomes (Fig. 1a and b). There was no significant impact of private compared to public practice on mortality outcomes (p = 0.86) (Fig. 1b).

Fig. 1

Logistic regression of factors impacting operative mortality. 1a displays the logistic regression including the Risk of Death (ROD) estimates and clinically important variables that may impact on operative mortality. Figure 1b displays the logistic regression including the ROD estimates only

Cumulative risk-adjusted mortality/morbidity (CRAM) analysis

Odds-corrected mortality outcomes are presented in a CRAM chart to illustrate the net events saved across successive cases (Fig. 2). Mortality rate was stable to the 1000 case mark. Across the first 100 cases, there were no cases of operative mortality, reflecting the positive deflections in the CRAM chart (Fig. 2). Then, from case 100 to 350, the observed rate was greater than predicted, accompanying the negative deflection in the graph. From case 350 to 1100, the expected and observed rates were similar. Then, after 1100 cases, there was a marked reduction in the rate of operative mortality, and thus a significant increase in the net events saved, accompanying the sharp upward deflection in the graph.

In the CRAM analysis for return to theatre, there was a net positive of events saved across the first 280 cases, followed by a steady decline in net events saved, or greater rates of return to theatre than expected, across the remainder of cases (Fig. 3).

Regarding LOS > 14 days, there was an increase in net events saved across the first 150 cases, followed by an increased rate of LOS > 14 days until case 1000. A change point was identified at approximately case 1070, where a significant improvement in LOS > 14 days was identified, which was sustained for the remainder of cases (Fig. 4).

Risk-adjusted sequential probability ratio test (RA-SPRT) for all-comers and subgroups

The RA-SPRT analyses for operative mortality was undertaken for all consecutive patients, CABG-only and valve-only subgroups. In consecutive patients included in this study, a reduced mortality event rate was detected at case 970 and sufficient evidence was achieved to confirm the change at case 1220, remaining below the lower limit of significant thenceforward (Fig. 2a). In the subgroup analyses of the CABG-only cohort, no significant change point was identified across cumulative cases. In the valve-only subgroup, a significant change was identified at case 200, with a slightly reduced odds of mortality events across the remainder of cases.

Discussion

When considering surgical outcomes, the contributing factors are multifaceted, including casemix, environment, resources, processes of care and operator/team experience. While the objective of this paper centres around performance monitoring and the learning curve, we first consider each of the aforementioned factors carefully before attributing the outcomes to operator experience. Firstly, when examining casemix, the Euroscore risk algorithm is a composite score calculated with patient factors and comorbidities, widely used to predict the risks of in-hospital mortality after cardiac surgery [16]. In this study, patients with higher Euroscores were operated on more frequently in later years of practice compared to early years of practice. It can be reasonably expected that as the operator gained more experience, they operated on patients in higher risk groups. While the casemix involved more comorbid, high-risk patients in the later years, outcomes including operative mortality, return to theatre and length of stay remained steadily low. This is reassuring that the improved results in the later years were not due to an avoidance of high-risk cases.

Secondly, when examining the environment and resources, outcomes were compared between hospitals that the surgeon operated in over the decade. Overall, there was no significant difference in operative mortality, rates of return to theatre or length of stay between public and private facilities. In the multivariate analysis, the facility was not found to be associated with patient outcomes. The multivariate regression found that elective status was associated with better patient outcomes (p = 0.001) while, as expected, high-risk quartiles were associated with increased operative mortality (p = 0.017). Interestingly, compared to standalone CABG, standalone valve and concomitant CABG and valve surgery carry a 2.7 and 5.7 times higher risk of operative mortality respectively (p = 0.049 and p = 0.0002 respectively).

Most importantly, the risk-adjusted operative mortality was stable until the 1000-case mark and began to show a marked reduction after 1100 cases. This is comparable with the findings of Novick and colleagues (1999), one of the few studies examining a single-surgeon learning curve in an intermediate-volume cardiac centre in America [17]. Interestingly, they reported a higher-than-expected cumulative failure rate in the first year of practice, then a linear reduction in the percentage of patients free from operative mortality over a decade of surgical practice [17]. However, their study did not identify a ‘change-point’ at which there was a consistent reduction in event rate. Using the change-point identification analysis, our study confirmed a consistent reduction in event rate at approximately 970 cases, and sufficient evidence to confirm the change at 1220 cases. In stark contrast, a CUSUM analysis of CABG by a single training cardiac surgeon over four years investigating only the first 50 cases, inferred that 23 cases of CABG and 73 cases of left internal mammary artery harvesting are sufficient to allow independent practice [18]. However, caution should be taken in the interpretation of these results, as the cumulative analysis relies on a proportion of cases to establish base rates. It is therefore not unexpected that there were no significant differences detected in Song and colleagues’ analysis, as there were likely insufficient cases to establish a representative base rate. Our study suggests that the case volumes required to have a consistent reduction in operative mortality lies at approximately the 1000-case mark.

Performance monitoring is central to the purpose of this analysis. The use of the CUSUM metric identifies changepoints which can signal when performance should be reviewed. The uptick in mortality and permanent stroke between the 100th and 350th case suggest opportunities for earlier improvement. While there are intrinsic limitations of a retrospective study, these metrics can be applied prospectively in the same fashion. This type of analysis should also be accompanied by a subjective or objective analysis of other factors impacting the outcome, such as case-mix and environmental factors which have been aforementioned, prior to attributing the outcome directly to surgical proficiency. Together, the utility of this type of analysis is greatly enhanced in assessing and improving surgical performance.

While many extraneous factors were considered and accounted for in this study, there are several limitations to be discussed. Firstly, the single-surgeon nature of this paper is an obvious limitation, as the learning curve of each cardiac surgeon is variable, according to differing exposure and caseload, requirements of training colleges and access to mentorship. However, the task of analysing learning curves is difficult due to the many external factors that impact on patient outcomes. Therefore, the decision to analyse a single surgeon was made to better identify and control for these factors before attributing the observed outcomes to a ‘learning effect’. Analysis of a single surgeon also allowed for consistent data quality over a decade and introducing multi-surgeon analysis may increase the variability and impact of external factors. Additionally, while the total practice of the surgeon involves adult cardiac surgery, adult thoracic surgery and transcatheter valvular interventions, this analysis is limited to cardiac only to enhance applicability of the findings to other adult cardiac surgeons. Secondly, as the surgeon operated in five hospitals throughout the decade, it is prudent to be mindful that the data from multiple hospitals, whilst being collected prospectively, is affected by the input and interpretation of multiple data managers. However, this risk is not unique to this study and applies to any database that utilises multiple data managers. Thirdly, a common limitation of length of stay data is that external factors that could influence this such as administrative processes and bed pressures are difficult to quantify and assess. Of note, the learning curve of an Australian cardiac surgeon begins immediately after completion of the training program without an intervening period of further training or supervision, which may differ from other training programs internationally. The strengths of this study include its long term follow up over a decade, the ability to analyse and control for many external factors, the use of risk-adjusted cumulative analysis and change-point identification to translate the results into a clinically applicable statement.

The findings of this study carry important implications for how training colleges are supporting individual trainees to competently enter independent practice. While an index number of cases are required to be met as a trainee, heavy senior oversight for medicolegal reasons, complexity of surgical pathology and reduced case volumes, most recently exacerbated by the pandemic, have significant impacts on trainees’ autonomy and exposure as the primary operator [19, 20]. Additionally, this study has provided robust methodology for individual monitoring of performance, which can help to identify and signal when intervention may be required, and should be considered by all surgeons throughout their careers [21] While training colleges are responsible for ensuring that a graduating fellow is prepared for independent practice, it is necessary to keep in mind the learning curve faced by junior consultants as they progress through their career and the importance of continued oversight and mentorship.

For the first years of a consultant surgeon, the monitoring of individual performance can be done through the statistical methods used in this paper, and the frequency of monitoring should be tailored to the surgeon’s operative volume. This paper, in essence, recommends a statistically valid method reflective practice through objective means that trainees and surgeons alike should consider in each stage of their career.

Conclusions

This article revealed that a consistent reduction in operative mortality was achieved at the 1000-case mark in the first decade of an adult cardiac surgeon’s practice. This finding prompts colleges to assess how they can support trainees to maximise their operator experience during surgical training and highlights the importance of continued mentorship into postgraduate years and beyond.

Fig. 2

Risk-adjusted sequential probability ratio test (RA-SPRT) (bottom) and cumulative risk-adjusted mortality/morbidity (CRAM) analyses (top) for operative mortality

Fig. 3

Risk-adjusted sequential probability ratio test (RA-SPRT) (bottom) and cumulative risk-adjusted mortality/morbidity (CRAM) analyses (top) for return to theatre

Fig. 4

Risk-adjusted sequential probability ratio test (RA-SPRT) (bottom) and cumulative risk-adjusted mortality/morbidity (CRAM) analyses (top) for operative mortality