Skinive Comparative analysis 2021 – 2022
Authors: Kirill S, Viktor S.
1. Introduction
Skin diseases are the most common cause of all human illnesses. This pathology occurs in almost 900 million people in the world [1]. There are more than 3000 known skin diseases worldwide. Skin cancer is now the most common type of malignant neoplasia in white populations. Especially melanoma is the skin cancer with the worst prognosis [2]. Melanoma and non-melanoma skin cancer show an increasing incidence rate worldwide. Incidence rates of 40-50/100,000 inhabitants/year should be expected in Europe in the next decades [3]. Skin diseases cause financial, socio-economic, and psychological burden to the community. Also, skin pathology is place a strain on general practitioners and specialty doctors. According to statistical studies, the average number of dermatologists in European countries is not enough. It is only 5 per 100,000 inhabitants [4].
Measures aimed at earlier skin diseases detection are necessary. The promising item is the implementation of machine learning technologies. This is confirmed by Chui, who showed that in medicine it is possible to automate 36% of functions. Especially at the levels of data collection and analysis [5]. Of particular importance in the development of such automation is the elaboration of mobile applications. The apps are very convenient tools for routine and widespread use in clinical practice.
Mobile phone applications are widely used in dermatology. Since the visual component of skin diseases can be easily digitized using a mobile phone camera. Then the image can be analyzed by machine learning algorithms to determine the pathology . Based on the results obtained, the user receives recommendations on further actions.
The potential of the apps for skin disease AI-detection is especially relevant during periods of unfavorable epidemiological states. For example, the COVID-19 pandemic when in-person medical consultations are inaccessible and remote health service provision is of particular importance [6].
1.1 The use of artificial intelligence in dermatology
The deep machine learning possibilities in skin disease recognition and differential diagnosis are inspiring. They demonstratethe capability in assisting clinicians in their daily practice. Such solutions triage cases to prioritize clinical care, help non-dermatologists to initiate dermatological care more accurately. Finally, they potentially improve access to necessary care.
The accuracy of diagnosis of skin diseases among general practitioners (GP) and dermatologists is significantly different. For example, the study proved that the accuracy of melanoma recognition among GPs is only 0.49-0.80. Among dermatologists this indicator reaches 0.85-0.89 [7]
Another study compared the accuracy of skin pathology recognition between a machine learning algorithm and medical professionals [8]. The study shows that the accuracy of the neural network diagnostics is 90% (for the first three results). So, this indicates that the accuracy of machine learning algorithms can be comparable to that of dermatologists. It is significantly higher than that of primary care physicians.
Regardless of the obvious advantages, the use of software products for solving such important tasks as skin disease detection is the subject of criticism. The critics’ argument is a doubtful sensitivity and specificity, as well as the lack of a standardized approach to testing among manufacturers, which is a prerequisite for an objective assessment. The introduction of such solutions into clinical practice is possible only if there is a well-trained neural network [9]. This implies a number of activities, including the creation of training and validation datasets, the neural network regular retraining, improvement of the outer shell for convenient use of the mobile application by the end-user, and others [10].
The illustration of performance in improving the network and the results obtained can be seen in the Skinive neural network.
2. Materials and methods to study of the Skinive Accuracy
2.1. Characteristics of the Skinive neural network
The Skinive algorithm is a model of pre-trained full-convolutional dermatological neural networks and application software of the neural network environment, which are located in a secure cloud and are integrated with applications (web, mobile, and other software) using an API service.
The Skinive algorithm is the key technology of similar applications and is a model of pre-trained dermatological neural networks and application software of the neural network environment, which are located in a secure cloud and integrated using an API service.
The accuracy of the Skinive recognition algorithms is constantly improving due to the systematic database replenishment with images of skin pathology, as well as the improvement of the processing algorithm. Each image, before being sent for algorithm training, is clinically tested by a dermatologist. Thanks to close cooperation with medical institutions, some of the cases used to improve the accuracy of the software diagnostic test, have histological confirmation.
2.2. Improvement of the Skinive neural network in 2022
From January 2022 to July 2022, the Skinive neural network has undergone a number of both technical and clinical improvements.
Within the framework of “ELISE” program, to improve the capabilities of the neural network, the following work was carried out: the facial imaging module was added (for the purpose of personal data filtering before processing), and the module determining the suitability for image processing (to improve overall accuracy) and the training dataset was extended to 54,604 images of skin pathologies.
In addition, the list of pathologies recognized by the Skinive neural network has been expanded. Particular attention was paid to such pathology groups as dermatitis and eczema. These pathologies are no less relevant than neoplasms or skin infections. They significantly impair the patient’s quality of life and require a timely diagnosis.
In addition to dermatitis, in 2022, it became possible to differentiate Versicolor among other fungal skin lesions, as well as acral nevus among pigmented skin neoplasms. These conditions require a different approach in clinical practice.
Thus, the total number of images in the Skinive training dataset for Q3 2022 is 164,142 photos of skin pathologies. The number of recognizable skin pathologies has reached 51 (Table 1).
Table 1 – Classes of recognized pathology
The main principle of work in 2022 was not only quantitative, but also qualitative data improvement. All data that was used in the neural network training underwent a multilevel check for compliance with the image quality and unambiguous interpretation by medical experts.
In July 2022, a large-scale work was carried out, which made it possible to obtain new data on the results of the accuracy of the skin pathology recognition by the Skinive 3.0 neural network.
2.3 Formation of the validation dataset to study of the Skinive Accuracy
To determine the accuracy of the Skinive algorithm, a validation dataset was specially prepared, consisting of 25,688 images of skin pathology. But the images included in the validation dataset were not used to train the neural network. Each image was evaluated by a dermatologist and oncologist for the correspondence of the pathology depicted to a certain class, taking into account the results of histological examination.
The structure of the validation dataset is formed in accordance with the structure of the training dataset, with the exception of dermatitis and eczema. The last two classes are not included in the validation dataset, since their recognition by the Skinive neural network became possible only in 2022, and the total number of photographs of these diseases did not allow us to isolate a separate validation dataset in sufficient volume to adequately assess the accuracy.
The validation dataset includes 7 classes of skin diseases. To clarify, a detailed description of the validation dataset is shown in Table 2.
Table 2 – Skinive Validation Dataset
This validation dataset was used in the analysis of accuracy in 2021, which made it possible to determine the dynamics of growth in sensitivity and specificity indicators.
To determine the sensitivity of the algorithm groups of skin diseases were used such as Acne and Rosacea, Papulosquamous, Mycosis, Viral diseases, Herpes, Premalignant + Malignant.
To determine the specificity of the algorithm 20982 images of benign skin neoplasms were used.
2.4. Data security
The hardware and software complex is built on Amazon Web Services solutions and is a series of virtual servers that are inaccessible from the outside, interconnected by solutions based on several software products and solutions, such as Docker, Amazon Elastic Container Services.
Skinive does not store any personally identifiable information that can be used to identify a specific user. User identification is carried out by a randomly generated token that is stored in the device and with which the application is accessed. The stored data is impersonal and has no path to be mapped to the actual user of the system. Graphic images sent to the server by the user undergo an anonymization procedure by removing all the image service information (EXIF headers, etc.)
Infrastructure logging services store information about fatal application errors, the original IP address of the request, the name of the called function, and the version of the user’s software.
3. Results of the Skinive Accuracy
The used validation dataset was formed in 2021 to conduct internal testing of the accuracy of the Skinive neural network. The results were retained and never published.
After sending the generated validation datasets for analysis to the Skinive algorithm in 2022, the following values of sensitivity and specificity were obtained for each of the analyzed classes (Table 3):
Table 3 – Skinive Neural Network Accuracy Results for 2022 and comparison with similar results for 2021
In this case, the sensitivity was determined as the ratio of the number of cases of skin pathology correctly determined by the algorithm (Acne and Rosacea, Papulosquamous, Mycosis, Viral diseases, Herpes, Premalignant + Malignant) to the number of all clinically confirmed cases, respectively. That is, this value shows the proportion of true positive cases, those cases that are correctly identified by the algorithm for risk. The higher the sensitivity, the less likely the algorithm will miss a high-risk skin pathology.
Specificity in the report of Skinive accuracy was defined as the ratio of the number of cases of benign skin neoplasms correctly identified by the algorithm to the number of all clinically confirmed benign neoplasms. This value shows the proportion of true negative cases (those situations where there are no high risks on the photograph). Accordingly, the higher the specificity, the less the likelihood of erroneous prediction of malignant pathology where it does not exist, and fewer “false alarms” will be received.
Error matrix of the Skinive
To visually understand the errors of the algorithm and identify the tendency of incorrect recognition by the neural network, an error matrix was compiled (table 4), which reflects the distribution of correct answers (by sensitivity) and errors between classes of nosologies.
Table 4 – Error Matrix by Disease Class
Designation: Green Cell is Correct Risk Level Prediction.
The overall Sensitivity, Specificity and Miss Rate of the Skinive algorithm is shown in the table 5:
Table 5 – Overall indicators of the Skinive Algorithm for 2021 and 2022
4. Discussion of the Skinive Accuracy
Thanks to the whole complex of work carried out on the Skinive neural network, it was possible to obtain an increase in sensitivity and specificity indicators for almost all studied classes of neoplasms and skin diseases.
For example, the sensitivity to the detection of precancerous and malignant neoplasms in 2021 was 82.6%, while in 2022 it was 96.3% (+13,7%), which shows a significant increase in sensitivity.
The results obtained allow us to assert that the set of measures aimed at improving the Skinive neural network is reflected in improving the accuracy of the algorithm in recognizing skin cancer. Timely and accurate recognition allows more effective treatment of this pathology. Obviously, it is this pathology that has the greatest relevance and significance from the point of view of preserving the life and health of the population.
A significant increase in sensitivity is also noted for other classes of skin pathology (Acne and Rosacea, Papulosquamous, Mycosis, Viral Diseases and Herpes) from +3 up to +10%.
The results show that Skinive’s neural network is more sensitive to skin pathology and less likely to miss situations where the patient actually needs medical attention or more detailed examination using other tools (dermatoscopy, biopsy followed by histological examination).
Although the general specificity of the algorithm has decreased, it is worth noting a significant increase in specificity in the recognition of benign diseases: in 2021 it was 93.0%, in 2022 – 98.2% (+5.2%). Based on this dynamic, Skinive’s neural network has become less prone to “false alarms”. Thanks to this feature, the use of Skinive’s neural network will reduce the number of unwarranted doctors visits and, at the same time, reduce the undue burden on medical professionals.
Analysis of the error matrix
A detailed analysis of the error matrix shows among which diseases and with what frequency the greatest difficulties arise for the Skinive algorithm in the differential diagnosis of neoplasms and skin diseases. These values can be considered a key result for planning further work on improving the Skinive neural network and increasing the accuracy of the algorithm for individual nosologies.
It is obvious from the error matrix that the highest percentage of errors in the recognition of skin pathology is observed in the pair “Benign” and “Premalignant + Malignant”: 5.9%. Error analysis showed that the neural network erroneously attributed the dysplastic nevus to benign nevus. With a visual comparison (Figure 1), indeed, difficulties may arise in the differential diagnosis of such cases.
Figure 1: Comparison of macro photography of a pigmented and dysplastic nevus
These clinical situations require more accurate differential diagnosis, including dermatoscopy or biopsy with histological examination.
The Miss Rate indicator
The Miss Rate indicator confirms the decrease in the number of missed skin pathologies. So, in 2021 it was 7.0%, in 2022 – 1.8% (-5.2%). That is, the number of erroneous conclusions (“false hopes”) has been significantly reduced. In absolute terms, the number of false negatives decreased from 328 to 86. A more detailed analysis of false negatives for skin diseases is presented in Table 6.
Table 6: Comparative Analysis of the Miss Rate for 2021 and 2022
Сomparing the Miss Rate for 2021 and 2022 in the report of Skinive accuracy, a global tendency towards a decrease in the number of misinterpretations in the recognition of skin pathology becomes obvious. In addition, this comparison allows us to note the dynamics and correctness of the vector of ongoing work on improving the neural network.
The most positive changes occurred in the Premalignant + Malignant class. The total number of errors in 2021 was 17.4%, while in 2022 it was only 3.7% (-13.7%). Among other classes, this indicator decreased from -3.4 up to -10,4%.
Given the relevance of timely and reliable recognition of skin pathology risks, eventually, this trend indicates a real possibility of using the Skinive neural network in clinical practice.
5. Conclusions
In 2022, the dataset on which the Skinive algorithm was trained was increased by more than 50,000 photographs of skin pathology. A key feature of all these photos was that the vast majority of them (90-95%) were made on a smartphone camera without the use of external optics or other special means (for example, dermatoscopes).
At the same time, the results of the analysis of the accuracy of the Skinive 3.0 neural network (trained in 2022 on a total of more than 160,000 images) show a significant increase in sensitivity (correctly identifying skin pathology when seeking medical attention is required) Neural networks and specificity (proper determination of benign neoplasms when medical care is not needed) compared to 2021.
The addition of the image quality module allowed for the early screening of poor-quality images, thus improving the quality of the images collected for datasets, which directly affected the algorithm’s ultimate accuracy.
Given the focus of the Skinive algorithm on mobile photos in 2022, it can be argued that this neural network has become more adapted to real clinical practice using a smartphone camera. Certainly, mobile devices with an installed Skinive app can be used by primary care physicians to screen skin pathology with great confidence. This is confirmed by the significant decrease in the number of faulty passages of skin pathology when recognized by Skinive neural network.
References:
- Andrews’ diseases of the skin: clinical dermatology. James WD, Berger TG, Elston DM, Odom RB. Saunders Elsevier; 2006.
- Current state of melanoma diagnosis and treatment. Davis, L.E.; Shalin, S.C.; Tackett, A.J. Cancer Biol. Ther. 2019, 20, 1366–1379. https://doi.org/10.1080/15384047.2019.1640032
- Epidemiology of skin cancer. Ulrike Leiter, Thomas Eigentler, Claus Garbe. PMID: 25207363. DOI: 10.1007/978-1-4939-0437-2_7
- Augustin M, Reusch M, Augustin J, Wagner T, Kämpfe S. European dermatology health care survey. Short report. 2013 (https://www.dermasurvey.eu/wp-content/uploads/eu-derma-health-care-survey-2013-short.pdf)
- Chui M, Manyika J, Miremadi M. Where machines could replace humans – and where they can’t (yet) (https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/where-machines-could-replace-humans-and-where-they-cant-yet)
- Chatterjee, P., et al. The 2019 novel coronavirus disease (COVID-19) pandemic: A review of the current evidence. Indian J Med Res. 2020 Feb & Mar;151(2 & 3):147-159;]), as well as in remote areas [[04]. Health service provision in rural and remote areas: a needs analysis. P Dunne, C Patterson, M Kilmartin, M Sladden. PMID: 8028543. DOI: 10.5694/j.1326-5377.1994.tb127349.x
- Michael DC, Judy W. Agreement between dermatologists and primary care practitioners in the diagnosis of malignant melanoma: review of the literature. J Cutan Med Surg. Sep-Oct 2012;16(5):306-10
- Liu, Y., et al. A deep learning system for differential diagnosis of skin diseases. Nat Med 26, 900–908 (2020). https://doi.org/10.1038/s41591-020-0842-3
- Deep Learning Approaches for Prognosis of Automated Skin Disease. Pravin R. Kshirsagar, Hariprasath Manoharan, S. Shitharth, Abdulrhman M. Alshareef, Nabeel Albishry, and Praveen Kumar Balachandran. doi: 10.3390/life12030426. PMCID: PMC8951408. PMID: 35330177
- Machine Learning and Its Application in Skin Cancer. Kinnor Das, Clay J. Cockerell, Anant Patil, Paweł Pietkiewicz, Mario Giulini, Stephan Grabbe and Mohamad Goldust. PMCID: PMC8705277. DOI: 10.3390/ijerph182413409
