The Area Over a Receiver Operating Characteristic (ROC) Curve as an Index of Diagnostic Inaccuracy

Aristides T. Hatjimihail, MD, PhD
 ath@hcsl.com

Hellenic Complex Systems Laboratory, Kostis Palamas 21, 66131 Drama, Greece.

Initialization Code

auc[μD_,μH_,σD_,σH_,u_]:=CDF[NormalDistribution[0,1],((μD-μH)/(Sqrt[((σD)^2)+((σH)^2)+2(u^2)]))]

rcAUC[μD_,μH_,σD_,σH_,u_]:=-(((E^(-((((μD-μH))^2)/(2 (2 (u^2)+((σD)^2)+((σH)^2)))))) (Sqrt[(2/π)]) u (μD-μH))/(((2 (u^2)+((σD)^2)+((σH)^2)))^(3/2)))

rcAOC[μD_,μH_,σD_,σH_,u_]:=(((E^(-((((μD-μH))^2)/(2 (2 (u^2)+((σD)^2)+((σH)^2)))))) (Sqrt[(2/π)]) u (μD-μH))/(((2 (u^2)+((σD)^2)+((σH)^2)))^(3/2)))

rrcAUC[μD_,μH_,σD_,σH_,u_]:=-((2 (E^(-((((μD-μH))^2)/(2 (2 (u^2)+((σD)^2)+((σH)^2)))))) (Sqrt[(2/π)]) u (μD-μH))/((((2 (u^2)+((σD)^2)+((σH)^2)))^(3/2)) Erfc[-((μD-μH)/((Sqrt[2]) (Sqrt[2 (u^2)+((σD)^2)+((σH)^2)])))]))

rrcAOC[μD_,μH_,σD_,σH_,u_]:=(((E^(-((((μD-μH))^2)/(2 (2 (u^2)+((σD)^2)+((σH)^2)))))) (Sqrt[(2/π)]) u (μD-μH))/((((2 (u^2)+((σD)^2)+((σH)^2)))^(3/2)) (1-(1/2) Erfc[-((μD-μH)/((Sqrt[2]) (Sqrt[2 (u^2)+((σD)^2)+((σH)^2)])))])))

Manipulate[
Which[
plot==0,
Plot[{Tooltip[auc[μD,μH,σD,σH,y]-auc[μD,μH,σD,σH,0],"change of AUC"],Tooltip[auc[μD,μH,σD,σH,0]-auc[μD,μH,σD,σH,y],"change of AOC"]},{y,0,uu},PlotRange->All,AspectRatio->1,AxesLabel->{"uncertainty","change"},AxesOrigin->{0,0},PlotStyle->{ColorData["HTML","SlateBlue"], RGBColor[1,.47,0]} ,Filling->Bottom,PlotLegends->Placed[{Row[{"Δ(AUC(",Style["u",Italic],"))"}],Row[{"Δ(AOC(",Style["u",Italic],"))"}]},Above],ImageSize->{340,340},PlotPoints->25,MaxRecursion->2],
plot==1,
Plot[{Tooltip[(auc[μD,μH,σD,σH,y]-auc[μD,μH,σD,σH,0])/auc[μD,μH,σD,σH,0],"relative change of AUC"],Tooltip[(auc[μD,μH,σD,σH,0]-auc[μD,μH,σD,σH,y])/(1-auc[μD,μH,σD,σH,0]),"relative change of AOC"]},{y,0,uu},PlotRange->All,AspectRatio->1,AxesLabel->{"uncertainty","relative change"},AxesOrigin->{0,0},PlotStyle->{ColorData["HTML","SlateBlue"], RGBColor[1,.47,0]} ,Filling->Bottom,PlotLegends->Placed[{Row[{"Δ(AUC(",Style["u",Italic],")) / AUC(0)"}],Row[{"Δ(AOC(",Style["u",Italic],")) / AOC(0)"}]},Above],ImageSize->{340,340},PlotPoints->25,MaxRecursion->2],
plot==2,
Plot[{Tooltip[rcAUC[μD,μH,σD,σH,y],"rate of change of AUC"],Tooltip[rcAOC[μD,μH,σD,σH,y],"rate of change of AOC"]},{y,0,uu},PlotRange->All,AspectRatio->1,AxesLabel->{"uncertainty","rate of change"},AxesOrigin->{0,0},PlotStyle->{ColorData["HTML","SlateBlue"], RGBColor[1,.47,0]} ,Filling->Bottom,PlotLegends->Placed[{Row[{"(∂_u)AUC(",Style["u",Italic],")"}],Row[{"(∂_u)AOC(",Style["u",Italic],")"}]},Above],ImageSize->{340,340},PlotPoints->25,MaxRecursion->2],
plot==3,
Plot[{Tooltip[rrcAUC[μD,μH,σD,σH,y],"relative rate of change of AUC"],Tooltip[rrcAOC[μD,μH,σD,σH,y],"relative rate of change of AOC"]},{y,0,uu},PlotRange->All,AspectRatio->1,AxesLabel->{"uncertainty","relative rate of change"},AxesOrigin->{0,0},PlotStyle->{ColorData["HTML","SlateBlue"], RGBColor[1,.47,0]} ,Filling->Bottom,PlotLegends->Placed[{Row[{"(∂_u)AUC(",Style["u",Italic],") / AUC(",Style["u",Italic],")"}],Row[{"(∂_u)AOC(",Style["u",Italic],") / AOC(",Style["u",Italic],")"}]},Above],ImageSize->{340,340},PlotPoints->25,MaxRecursion->2]],
{{plot,1,"plot"},{0->"change",1->"relative change",2->"rate of change", 3->"relative rate of change"},ControlType->PopupMenu},
"",
"nondiseased population",
{{μH,0,"mean"},0,6,.01,ImageSize->Tiny,Appearance->"Labeled"},
{{σH,1,"standard deviation"},0.01,6,.01,ImageSize->Tiny,Appearance->"Labeled"},
Delimiter,
"diseased population",
{{μD,3.67,"mean"},0,6,.01,ImageSize->Tiny,Appearance->"Labeled"},
{{σD,1,"standard deviation"},0.01,6,.01,ImageSize->Tiny,Appearance->"Labeled"},
Delimiter,
"upper bound of the uncertainty\nof measurement,",
{{uu,1,"second test"},0,6,.01,ImageSize->Tiny,Appearance->"Labeled"},ControlPlacement->Left,SaveDefinitions->True,AutorunSequencing->{1,2,5,6},SynchronousUpdating->False,TrackedSymbols->True]

Description

This Demonstration plots the change, the relative change, the rate of change and the relative rate of change of the area under (blue plot) and the area over (orange plot) the receiver operating characteristic (ROC) curve of a diagnostic test as the uncertainty of measurement increases from 0 to a user-defined upper bound. Measures are determined for normally distributed nondiseased and diseased populations for various values of the mean and standard deviation of the populations. A normal distribution of the uncertainty is assumed. The type of plot is selected using the "plot" menu. The five parameters that can be varied using the sliders are given in arbitrary units.

Snapshots

Details

The ROC curves are used in the evaluation of the clinical accuracy of a diagnostic test applied to diseased and nondiseased populations. An ROC curve is a plot of the sensitivity of a test versus 1-specificity. Sensitivity is the fraction of the diseased population with a positive test while specificity is the fraction of the nondiseased population with a negative test. Therefore, the ROC plots the true positive fraction versus the false positive fraction. Furthermore, the area under an ROC curve is used as an index of the diagnostic accuracy of the respective test. The area under the curve decreases as the uncertainty of measurement of the diagnostic test increases [1].

Assuming AUC(u) is the area under an ROC curve and AOC(u) is the area over the ROC curve for an uncertainty of measurement u, the changes Δ(AUC(u)) and Δ(AOC(u)) are defined as AUC(u)-AUC(0) and AOC(u)-AOC(0), the relative changes as Δ(AUC(u))/AUC(0) and Δ(AOC(u))/AOC(0), the rates of change as (∂_u)AUC(u) and (∂_u)AOC(u), and the relative rates of change as (∂_u)AUC(u)/AUC(u) and (∂_u)AOC(u)/AOC(u), respectively.

As AOC(u)=1-AUC(u), it can be considered that the area over the ROC curve is an index of diagnostic inaccuracy. In fact, as the plots of this Demonstration show, the relative change, the rate of change and the relative rate of change of the area over an ROC curve versus the uncertainty of measurement are greater than the absolute value of the respective measures of the area under the ROC curve for the same populations.

To the best of the author's knowledge, measures of the area over the ROC curve versus the uncertainty of measurement have not been discussed in the literature except for [2].

The area over the ROC curve could be used in the evaluation of a diagnostic test as a diagnostic inaccuracy index. For example, in the thumbnail and the snapshots, the population data describes a bimodal distribution of serum glucose on a nondiabetic and a diabetic population [3].

References

[1] A. T. Hatjimihail. "Receiver Operating Characteristic Curves and Uncertainty of Measurement" from the Wolfram Demonstrations Project—A Wolfram Web Resource. demonstrations.wolfram.com/ReceiverOperatingCharacteristicCurvesAndUncertaintyOfMeasure.

[2] A. T. Hatjimihail. "Uncertainty of Measurement and Areas Over and Under the ROC Curves" from the Wolfram Demonstrations Project—A Wolfram Web Resource. demonstrations.wolfram.com/UncertaintyOfMeasurementAndAreasOverAndUnderTheROCCurves.

[3] T.-O. Lim, R. Bakri, Z. Morad and M. A. Hamid, "Bimodality in Blood Glucose Distribution—Is It Universal?," Diabetes Care, 25(12), 2002 pp. 2212–2217. doi:10.2337/diacare.25.12.2212.

Keywords

receiver operating characteristic curves
ROC curves
uncertainty of measurement
sensitivity
specificity
diagnostic test
clinical accuracy
diagnostic accuracy
diagnostic inaccuracy
normal distribution
binormal distribution
area under ROC curves
area over ROC curves
AUC
AOC
relative change
rate of change
relative rate of change

Related Links

Receiver Operating Characteristic Curves and Uncertainty of Measurement

Uncertainty of Measurement and Areas Over and Under the ROC Curves

Receiver Operating Characteristic Curves and Uncertainty of Measure

Disclaimer of Warranties

The notebook is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or Hellenic Complex Systems Laboratory or any of its members be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the notebook or the use or other dealings in the notebook.

About

The Area Over a Receiver Operating Characteristic (ROC) Curve as an Index of Diagnostic Inaccuracy
Version Number 1.1.0
Hellenic Complex Systems Laboratory 2021
First published at Wolfram Demonstrations Project at: (HyperlinkURL["demonstrations.wolfram.com/TheAreaOverAReceiverOperatingCharacteristicROCCurveAsAnIndex/", https://demonstrations.wolfram.com/TheAreaOverAReceiverOperatingCharacteristicROCCurveAsAnIndex/])
The notebook is distributed under  an Attribution - NonCommercial - ShareAlike 4.0 International Creative Commons License.