Reliable Diagnostics for Coronary Artery Disease

Reliable Diagnostics for Coronary Artery DiseaseMatjaˇz Kukar University of Ljubljana,Faculty of Computer and Information Science,Trˇz aˇs ka25,SI-1001Ljubljana,Sloveniae-mail:matjaz.kukar@fri.uni-lj.siCiril Groˇs eljUniversity Medical Centre Ljubljana,Department for Nuclear Medicine,Zaloˇs ka7,SI-1525Ljubljana,Sloveniae-mail:ciril.groselj@kclj.si AbstractIn the past decades Machine Learning tools have been successfully used in several medical diag-nostic problems.While they often significantly outperform expert physicians(in terms of diagnostic accuracy,sensitivity,and specificity),they are mostly not being used in practice.One reason for this is that it is difficult to obtain an unbiased estimation of diagnosis’reliability.We discuss how reliability of diagnoses is assessed in medical decision making and propose a general framework for reliability estimation in Machine Learning,based on transductive inference.We compare our approach with the usual Machine Learning probabilistic approach as well as with classical step-wise diagnostic process where the reliability of diagnosis is presented as its post-test probability. The proposed transductive approach is evaluated in a practical problem of clinical diagnosis of the coronary artery disease.Significant improvements over existing techniques are achieved.Clinical decision-making is a complicated process based on experience,judgement,and reason-ing that should simultaneously integrate information from the medical literature and a variety of other sources,including quantitative results of clinical trials and,most importantly,diagnostic test results.Since no diagnostic test is perfect,thefinal decision–diagnosis of a condition in question –also carries a certain amount of imperfectness and therefore cannot be considered completely reliable.It is often crucial to know the magnitude of diagnosis’(un)reliability in order to minimize risks for the patient’s health or even life.In recent decades a Bayesian approach to integrating various test results withfindings from the patient’s history and physical examination has become increasingly popular,especially in a con-junction with stepwise diagnostic process.While the Bayesian calculation of post-test probabilities provides a physician with a quantitative assessment of diagnosis’reliability,it suffers from several problems.In this paper we propose a general strategy,a transductive reliability estimation,based on the Machine Learning(ML)paradigm.It can be used for detecting and selecting reliable diagnoses independently of the applied ML algorithms.The reasoning behind transductive approach is that in order to solve a more general task,resources may be wasted or compromises have to be made which would not have been necessary for solving only the problem at hand(i.e.to diagnose a patient vs. to diagnose the whole patient population).This common-sense principle reduces a more generalproblem of inferring a functional dependency on the whole of input space(inductive inference)to the problem of estimating the values of a function only at given points(transductive inference).The goal of our study was to determine whether it could be possible to increase number of re-liable diagnoses(both positive and negative)thus reducing unnecessary further diagnostic testing (rationalization of diagnostic procedure).We have successfully adapted this principle for estima-tion of diagnoses’reliability and compared it with post-test probability calculations of coronary artery disease.We managed to improve the specificity of reliable diagnoses by33%and sensitiv-ity by18%.We performed an extensive comparison with clinical diagnostic process and included enthusiastic physicians’responses to our results.In most clinical institutions the patients’data are stored in a central computer database.With time,more and more records that include confirmed diagnoses appear in the database.Such databases are frequently subject of retrospective studies.The patients in whom the outcome has already occurred are selected and analyzed,thus looking backward to assess potential risk factors and diagnostic principles.Retrospective studies naturallyfit in the ML application framework, which is becoming increasingly popular as a support tool in medical decision making.Having collected a set of patient descriptions with confirmed diagnoses,the task of a ML al-gorithm is to automatically generate a model(a description)of the given data with respect to the correct diagnosis.A set of possible diagnoses is used as a target for classification process.A gen-erated model can subsequently be used for risk factors’assessment and decision-making support. Unfortunately,only little work has been done in ML to assess reliability of individual decisions.2.1.Assessment of post-test probabilityIn past decades a Bayesian approach to integrating various test results withfindings from the patient’s history and physical examination has become increasingly popular.By this approach the post-test diagnostic probability is calculated from the pre-test probability(prevalence of a disease), diagnostic test’s outcome,sensitivity and specificity,thus providing a basis for a quantitative estima-tion of diagnosis’reliability.This approach is often used in a conjunction with stepwise diagnostic process,where diagnostic tests are performed sequentially until the post-test probability of a disease exceeds certain threshold,usually[1].While the Bayesian calculation of post-test probabili-ties provides a physician with a quantitative assessment of diagnosis’reliability,it has from several problems.Those include obtaining exact values for pre-test probabilities and diagnostic tests’sen-sitivity and specificity,as well as an unbiased test result and selection of conditionally independent (in Bayesian sense)diagnostic tests[3,8].2.2.Transductive reliability estimation in Machine LearningWe propose a general strategy,a transductive reliability estimation[4,5],for detecting and selecting reliable diagnoses independently of the applied ML algorithms.Basically,we have a two-step process,featuring an inductive step followed by a transductive step.An inductive step is just like an ordinary inductive learning process in Machine Learning.A Machine Learning algorithm is run on the training set,inducing a classifier.A selectedexample is taken from an independent dataset and classified using the induced classifier.An example,labelled with the classified class is included into the training set.A transductive step is almost a repetition of an inductive step.A Machine Learning al-gorithm is run on the changed training set,transducing a classifier.The same example as before is taken from the independent dataset and and classified using the transduced classi-fier.Both classifications(probability distributions)of the same example are compared and their difference(symmetric Kullback-Leibler divergence)is calculated,thus approximating the randomness deficiency.A classification–or a diagnosis–is considered reliable if the difference in probability distributions between the two above steps is small enough1.The idea behind transductive reliability estimation leverages two approaches.Thefirst one is the iid(identically independently distributed)assumption,usually used in statistical models and Machine Learning,and the second one is randomness deficiency[2,7].Basically,we estimate how much the randomness of the training set(in a Kolmogorov complexity sense)would change if it included a new example.2.3.Clinical diagnostics of coronary artery diseaseCoronary artery disease(CAD)is one of the most important cause of mortality in the world.It is caused by diminished bloodflow through coronary arteries due to stenosis or occlusion.CAD produces impaired function of the heart andfinally the necrosis of the myocardium(infarction). During the exercise the volume of the blood pumped to the body has to be increased,this causing the bloodflow through the coronary arteries to be increased as well.In a(low grade)CAD the bloodflow as perfusion of the myocardium is adequate at rest or during a moderate exercise,but insufficient during a severe exercise.Therefore,signs and symptoms of the disease develop only during the exercise.There are four diagnostic levels of CAD.1.Signs and symptoms of the disease are evaluated clinically and ECG is performed at rest.2.This is followed by sequential ECG testing during controlled exercises by gradually increas-ing the work load of the patient.3.If the previous test is not conclusive,or if additional information regarding the perfusion ofthe myocardium is needed,myocardial scintigraphy is performed at rest and during exercise.4.If an invasive therapy of the disease is contemplated,the diagnosis has to be confirmed byimaging of the coronary vessels(coronary angiography).In our study we used a dataset of327patients(250males,77females)with performed clinical and laboratory examinations,exercise ECG,myocardial scintigraphy and coronary angiography be-cause of suspected CAD.The features from the ECG an scintigraphy data were extracted manually by the clinicians(up to77attributes).In228cases the disease was angiographically confirmed and in99cases it was excluded.162patients had suffered from recent myocardial infarction.The patients were selected from a population of approximately4000patients who were examined at the Nuclear Medicine Department between1991and1994.We selected only the patients with com-plete diagnostic procedures(all four levels)[6].For determining the pre-test probability we applied a tabular calculation,consisting of three attributes(age,sex,type of chest pain),retrieved from[9].We performed all our experiments by using naive Bayesian classifier[4]as our ML tool of choice,since it usually performs very well in medical problems and can comprehensibly explain its classifications to the physicians.However,this does not mean that our approach is limited to this ML paradigm.In fact,many different ML approaches,such as decision trees,K-nearest neighbour and neural networks have been tested within the framework of transductive reliability estimations [5]and all of them performed well.3.1.Experiments in the clinical diagnostics of CADIn the stepwise diagnostic process we measured after each completed step the percentage(share) of reliable diagnoses(with the post-test probability over),and errors made in this process (percentage of incorrectly diagnosed patients with seemingly reliable diagnoses).We evaluated the problem in four different settings:Case1:Physicians.Stepwise diagnostic process is performed by physicians.Pre-test probabilities are obtained from literature.A boundary for reliable diagnoses is(post-test probability). Case2:Stepwise Machine Learning.Stepwise diagnostic process is performed by using a ML classifier(naive Bayesian classifier)for evaluation of each offirst three diagnostic levels.A boundary for reliable diagnoses is(post-test probability).Case3:Naive Bayes.Diagnostics with all attributes available on the third diagnostic level(my-ocardial scintigraphy)using naive Bayes classifier.A boundary for reliable diagnoses is(probability of the most probable class).Case4:Naive Bayes(transductive).Diagnostics with all attributes available on the third level using naive Bayesian classifier.Reliability of diagnosis is estimated with transductive method (Sec. 2.2).A boundary for reliable diagnoses is automatically calculated,using internal leave-one-out testing procedure.The experimental results are presented in Tab.1.In cases1and2(stepwise diagnostic process for physicians and naive Bayesian classifier)the performance in terms of reliable classifications was quite similar.In cases3and4we used all available attributes on the third diagnostic level(myocar-dial scintigraphy).In both cases we significantly outperformed physicians in terms of classification accuracy,sensitivity and specificity.Naive Bayesian classifier’s diagnoses’reliability was assessed by classifier’s probability(case3)and by transductive method(case4).The calculated boundary for reliable estimations was,which is quite close to for post-test probabilities.The overall results in case4are quite impressive.While the number of reliable classifications stays on the same high level as in case3(naive Bayes with all attributes),the total number of errors drops significantly below physicians’levels.In a direct comparison,naive Bayes(transductive)im-proved the number of reliable positive diagnoses by0.18(18%)and the number of reliable negative diagnoses even by0.33(33%)!The number of errors in positive diagnoses is slightly higher than the physicians’,while the number of errors in negative diagnoses dropped below0.04(4%)!For comparison of physicians’and transductive Bayes’performance a reliable ROC curves for positive and negative patients are depicted in Fig.2(a)and Fig.2(b).In both cases the ROC curve for transductive reliable estimations(marked ML)lies above the physician’s ROC curve and closer to the left top corner,that an ideal ROC curve should touch.In the problem of clinical diagnostics of CAD the physicians are mostly interested in increasing numbers of correctly diagnosed negative patients after the third diagnostic level(higher specificityparison between physicians’results and different ML approaches.Remarkable improvements achieved with transductive Bayesian classifier are emphasized.1.Physicians(post-test)0.850.850.860.720.030.460.082.Naive Bayes(post-test)0.890.910.830.790.050.460.033.Naive Bayes(class prob.)0.910.920.870.900.070.810.114.Naive Bayes(transductive)0.910.920.870.900.050.790.04T r u e r e l i a b l e p o s i t i v e r a t e (a)Reliable ROC curve on positive patients.(b)Reliable ROC curve on negative pa-tients.Figure parison of reliability estimations between physicians (post-test probabilities)and transductive reliability estimation.10%-15%less patients needing to perform coronary angiography test seems reasonable.We are aware that the results of the CAD study are obtained on a significantly restricted popula-tion and therefore may not be generally applicable to the normal population,i.e.to all the patients coming to the Nuclear Medicine Department.Further studies are in progress to verify our find-ings on larger population.In particular,on-line data gathering,advocated in our previous papers,has now at least partially entered everyday practice and may result in providing us with a more representative dataset in the near future.[1]G.A.Diamond and J.S.Forester.Analysis of probability as an aid in the clinical diagnosis of coronary arterydisease.New England Journal of Medicine ,300:1350,1979.[2] A.Gammerman,V .V ovk,and V .Vapnik.Learning by transduction.In Proceedings of theConference on Uncertainty in Artificial Intelligence ,pages 148–155,Madison,Wisconsin,1998.[3]L.Goldman and E.Braunwald.Primary Cardiology .W.B.Saunders Company,Philadelphia,Pennsylvania,1998.[4]M.Kukar.Making reliable diagnoses with machine learning:A case study.In Proc.European Conference onArtificial Intelligence in Medicine ,pages 65–74,Porto,Portugal,2001.[5]M.Kukar.Transductive reliability estimation for medical diagnosis.Artificial Intelligence in Medicine ,2002.Toappear.[6]M.Kukar,I.Kononenko,C.Groˇs elj,K.Kralj,and J.Fettich.Analysing and improving the diagnosis of ischaemicheart disease with machine learning.Artificial Intelligence in Medicine ,16(1):25–50,1999.[7]M.Li and P.Vit´a nyi.An introduction to Kolmogorov complexity and its applications .Springer-Verlag,New York,edition,1997.[8]M.Olona-Cabases.The probability of a correct diagnosis.In J.Candell-Riera and D.Ortega-Alcalde,editors,Nuclear Cardiology in Everyday Practice ,pages 348–357.Kluwer,1994.[9] puter-assisted interpretation of noninvasive tests for diagnosis of coronary artery disease.Car-diovasc.Rev.Rep.4,pages 367–375,1983.。