The reasons mentioned above have motivated us to create our own groundtruth set that can be used to evaluate and compare the performance of multiple algorithms. A set of 20 images were chosen from this dataset to cover a wide range of retinopathy as shown in Table 1. The images were all either healthy or suffered from early-stage Diabetic Retinopathy and were good resolution images. There were no abnormalities such as laser scars in the images. The purpose of this selection was to have a dataset of ideal scenario images for assessment. This means that the results of the assessment on this dataset should produce the ideal performance of a given algorithm or technique. In other words, the dataset should identify the ‘peak performance’ of the algorithm being tested.
|Retinopathy Grade||Number of Microaneurysms||Number of images (training)||Number of images (test)|
The images were groundtruthed by an expert grader. During the groundtruthing the grader marked all the microaneurysms that were visible to him. A circular marker was used rather than pixel-based marker . Majority of the literature has relied on object-based metrics to measure the accuracy of detection. This is because it gives a more sensible measure of performance – indicating the amount of MA objects detected in the image relative to the total MA objects present. Furthermore, reliance on pixel-based metrics can be misleading due to the imbalance in proportion between very few MA pixels and a large number of background pixels.
The microaneurysm labeling was performed using imannotate, an open source MATLAB tool that we built to assist annotating images. We used the tool to label circles around each MA candidate that was labeled by the expert. In addition to marking each candidate, we also labeled each MA candidate using one of the following labels : Obvious, Regular or Subtle based on their relative visibility and/or local contrast in the image. Close to Vessel is a label given to MA candidates that lie close to a blood vessel. According to  an MA candidate was marked as close to vessel if it lay around 1 MA diameter away from the vessel. We have stuck to this convention during our labeling process.
The images in the dataset belonged to the same resolution of 1440 x 960 px.
Habib, M. M., et al. “Microaneurysm detection in retinal images using an ensemble classifier.” Image Processing Theory Tools and Applications (IPTA), 2016 6th International Conference on. IEEE, 2016.