Regular foot examinations are part of a program to reduce the risk of serious diabetes complications^1,2 and the assessment of compliance with this guideline is central to quality assurance reporting in diabetes management.^3–5 Assessing the evidence of foot examinations is among five process quality measures for diabetes management used by the Agency for Healthcare Research and Quality (AHRQ) to report on national health care effectiveness.⁶ Such quality assurance (QA) measures are important for the assessment of health care provider performance and for improving patient safety.⁷ Clinical performance measures based on administrative structured data in the electronic medical record (EMR) are currently available but may be suboptimal for certain measures as compared to the information reported in the unstructured, free-text part of the EMR.^8,9 Manual audit of the medical record that relies on the free text of the EMR for many quality indicators including foot examinations is subject to lack of internal consistency and poor inter-rater reliability.^10–13 Furthermore, it is expensive, time consuming¹¹ and therefore is not feasible in large ambulatory patient populations. Thus QA measurements are performed on small samples of patient populations. While this sampling strategy is informative at the aggregate level of a health care organization, it does not provide actionable data that can be used to improve the health and safety of individual patients or assess the performance of individual physicians. The increasing adoption of the EMR¹⁴ and advances in Natural Language Processing (NLP) and machine learning make it possible to analyze the unstructured text of clinical reports automatically or semi-automatically in ways which were historically impossible or cost prohibitive using paper based records.^6,15,16

This study uses clinical documentation of in- and out-patient visits to the Mayo Clinic, Rochester, MN. Figure 1 shows an example of the text contained in a physical examination section of a clinical note that documents a foot examination. According to the clinical care guidelines for diabetes management, a complete foot examination should document the following three components: structural, neurological, and vascular.¹⁷ The evidence of documentation is determined by examining the medical record by a licensed health care provider and must include at least two of the three components.¹⁷ More recently, manual administrative coding systems tailored to National Committee for Quality Assurance (NCQA) requirements are being used as well (e.g., Kaiser Permanente).

Figure 1 An example of the text of a physical examination report.

Several systems using NLP and machine learning have been developed to process unstructured text of clinical reports.^6,16,18–22 The system described in this article relies on Support Vector Machines (SVMs) which represent a set of automatic classification algorithms widely used in medical text categorization.^23–27

We trained three SVMs (one for each foot examination component) to associate a set of predictive covariates with the correct classification (normal, abnormal, not assessed) of the text in physical examination sections of clinical notes. The overall architecture of the system is illustrated in Figure 2. In Phase I, the text of the physical examination section is searched electronically to determine if it contains evidence of a foot examination.²⁸ If such evidence is found, then the text of the note is converted to a vector of predictive covariates in Phase 2 and presented to three SVM classifiers.

Figure 2 Architecture and process-flow of the foot exam identification system.

Figure 3 Keywords used as predictive covariates.

Figure 4 Schematic representation of the contextual scope for qualifiers.

• (1) … General: She is a AGE woman in no apparent distress. … Vessels: No abdominal aortic, femoral, renal, or carotid bruits. Dorsalis pedis is 0. Heart: … . Lungs: …. Abdomen: … Extremities: Venostasis and skin discoloration. Bilateral lower extremity +1 edema. No ulcers. No skin breakdown. Bilateral changes of Onychomycosis.…
• (2) {skin 1,lower 1,extremit 1,change 1,no 1,ulcer 1,onychomycosis 1, fe_outcome abnormal}

In the example (1), the qualifier covariate “no” is set to “1” because it occurs in the context of the word “ulcers” which represents a structural component covariate. The contextual scope here is defined as the span of text within five words to the left or right of the target. The optimal size of five words was determined empirically by iterating through several rounds of validation ranging from three to ten words.

Regular expressions for covariates were defined in terms of the stemmed as well as unstemmed variants, and since the stemmed form is typically shorter it would match before the unstemmed form. Thus, while the example in (1) contains the plural form “ulcers”, it is represented as stemmed “ulcer” in the covariate vector in (2). The words “dorsalis,” “pedis,” and “edema” are not represented in the vector in (2) because they are defined as part of the vascular component, while the example in (2) represents the covariate extraction for the structural component only.

We used WEKA implementation of the sequential minimal optimization algorithm (SMO) classifier³¹ with the default parameter settings—attribute normalization, polynomial kernel with exponent of 1.0, and complexity of 1.0. No parameter optimization was performed for this study.

The proposed methodology is scalable to any number of patients or providers and thus offers a low-cost, timely and reliable strategy for investigating diabetes risk factors in large populations as well as longitudinally for individual patients. This methodology will also enable more accurate quality assurance measurements and may be used at the point of care to provide actionable information to physicians about patient profiles at high risk for diabetes complications.

Table 3 Table 3 Distribution of Misclassification Errors

Informal observation of the erroneous samples also indicates that some of the misclassification errors resulted because our approach only loosely associates qualifiers with the appropriated term in the text of the report. In some instances this strategy may have resulted in associating the wrong qualifier with a given term or failing to associate a qualifier due to fixed scope boundaries. A more sophisticated strategy based on syntactic phrase boundaries identified by an automatic parser with a rule-base capable of processing semi-structured clinical reports may improve the results.

While the keywords may be extracted from available corpora automatically, domain expertise is still required to classify the clinical records for training data and to suggest additional keywords that may not have been captured by the statistical methods. The latter may be accomplished with a general purpose NLP system. One advantage of the techniques proposed in this paper over general-purpose NLP applications is that they are simple and may be focused on a specific type of a physical exam at the system design stage allowing for greater flexibility. Also, physical examination sections of clinical reports tend to contain telegraphic and sometimes semi-structured speech that varies by clinic and provider. While a general-purpose NLP system is more versatile than the methods described in this paper, to be most effective, such system’s rule-base may need to be modified to account for the idiosyncrasies of the language used in physical examination sections. It may be beneficial to use a general purpose NLP system, however, to aid in extracting optimal covariates for machine learning.

This study demonstrates an application of machine learning for identification of foot examination findings from clinical reports. This application may improve quality and safety of patient care by providing inexpensive and scalable methodology for conducting quality assessments as well as for assessing potential risk factors at the point of care.