Enormous amounts of textual biomedical information are becoming available electronically on a daily basis, making it difficult for researchers or indexers to keep current. There is a pressing need to develop automated methods to index, codify, and manage the growing body of information in order to increase its accessibility and use for applications such as information retrieval, knowledge discovery, and database curation. Natural Language Processing (NLP) techniques have been playing an increasing role in extracting and managing biomedical entities and relations from text.^1–4 However, the development of NLP methods is costly and time-consuming, largely due to the formidably large number of biomedical terms that NLP systems must recognize. Fortunately, much effort has gone into creating biomedical knowledge resources and tools usable by NLP systems, for instance, the Gene Ontology (GO),⁵ UniProt,⁶ HUGO,⁷ JCBN,⁸ NCBI Taxonomy,⁹ and the Unified Medical Language System (UMLS),^10,11 thereby substantially reducing the overhead of the development process. These resources serve as huge repositories of biomedical terms. The UMLS also provides associations from the terms to normalized concepts, which are assigned semantic categories.

High-quality semantic classification is generally critical for NLP and for other knowledge-based systems using concepts and/or reasoning. To facilitate the development of ontologies, it is worth exploring an automated method to semantically classify ontological concepts. Ideally the method could also be used to improve the ontology itself and the performance of the systems depending on it. For this study we focus on use of corpus-based distributional properties for such a classification task. According to Harris’s sublanguage theory, the syntactic dependence of words on other words in general language and, especially in specialized languages, exhibits different likelihoods.¹² For example, in the biomedical domain the object of the verb “prevent” is more likely to be a disorder than a body part. NLP researchers have explored use of such distributional differences to semantically categorize words.^13–15 Such an approach differs significantly from the manner by which ontologies are developed, because experts generally use their knowledge of the domain to manually classify ontological concepts. An automatic tool that assists experts in developing and maintaining ontologies for NLP as well as other knowledge-based applications would have substantial benefits of being consistent, reproducible, and high throughput, and when used in conjunction with experts, should help speed the overall process.

In this paper, we focus on semantic classification within the UMLS, because it is a comprehensive knowledge representation framework with a growing user population and regular maintenance. Many NLP researchers have used UMLS concepts and their associated terms as a source of lexical knowledge. For example, the UMLS has been used in biomedical named entity recognition tasks¹⁶ and in providing mappings to normalized conceptual representations (e.g., for indexing of medical images).¹⁷ The UMLS has also been used by some NLP systems to determine relations among the extracted terms through specified semantic patterns.^18,19 Systems using the UMLS and linguistic knowledge to determine relations generally depend on recognizing UMLS concepts and associating them with the appropriate semantic categories to function properly. Therefore, semantic classification is critical to relation determination that follows entity recognition. However, the UMLS semantic classifications have been reported to contain inconsistencies,²⁰ with a number of concepts assigned questionable semantic types. For example, concepts are semantically inconsistent under T169 “Functional Concept” (2006AD), which contains procedure-related concepts such as “Intramuscular Injection,” function-related concepts such as “Regulatory Pathway,” disorder-related concepts such as “Tumor-like lesion,” and very high-level concepts such as the verb “Increase.”

The corpus-based method we propose is based on distributional properties of terms and is domain-independent. Therefore, it is general and can be used for other ontologies as well, provided that the ontological concepts can be found in text. The requirement of having the concepts occur in text is not a limitation for our work because one of the primary goals of this automated semantic classification method is to increase the utility of an ontology for NLP purposes. Additionally, rather than using the existing UMLS semantic classes, we focus on reclassifying UMLS concepts into broader semantic classes that are more compatible with NLP applications. Our main consideration involves well-defined and clinically relevant classes, and therefore disregards classes such as T066 “Machine Activity,” T070 “Natural Phenomenon or Process,” and T170 “Intellectual Product.” A secondary consideration concerns the granularity of the classification, which is elaborated on in the Background section.

Thus, in this paper we describe the development and evaluation of an automated method for reclassification of UMLS concepts that is based on a distributional similarity approach. We try to address the issues of semantic granularity, objectivity, and consistency. To the best of our knowledge, this is the first time an automated, corpus-based method has been developed for the semantic classification of UMLS concepts.

In the following sections we will first review related research on reorganizing the UMLS semantic classification and associated issues, different approaches for performing semantic classification (with an emphasis on the corpus-based approach), and background about the specific resources we used in building our semantic classifiers. Then we describe the details of the implementation and experiments, followed by the results, our observations, interpretations, and conclusions.

The appropriateness of semantic granularity is another open issue and is actually application-dependent. The semantic types of the SN are geared for biomedical knowledge-based applications but sometimes are too fine-grained for a general-purpose biomedical NLP system because the semantic patterns occurring in biomedical text are coarser than the types of semantic classes needed for knowledge-based applications. For example, text may have the same semantic patterns involving Fungus, Virus, Rickettsia or Chlamydia, Bacterium, and Archaeon as those involving microorganisms in general, and therefore these classes would not be important to differentiate for NLP purposes. However, such differentiation could be important for other biomedical applications. Considerable research has been performed on simplifying the UMLS semantic classification as well as auditing the errors in it. The motivation for reducing the complexity of the SN is to make it easier for human comprehension and for system integration (including NLP systems). Several approaches have been proposed for aggregating the SN types into fewer semantic groups, such as rule-based regrouping,²² regrouping based on SN relations,²³ and regrouping by string matching of SN type definitions.²⁴ Some SN auditing approaches were derived directly from a coarser-grained classification (i.e., metaschema).^25,26 Others performed auditing by checking concepts assigned to mutually exclusive SN types,²⁷ by checking redundant SN assignments,²⁸ or by applying principles to enforce ontological soundness.²⁹

Basically the above approaches can be considered “ontological” and thus are still susceptible to the limitations of the existing SN structure. For example, we consider the concept “Intramuscular Injection” should be more appropriately assigned a procedure-related class instead of the broad “Functional Concept.” However, this can not be achieved by a re-grouping approach that always aggregates “Functional Concept” with other broad types such as “Qualitative concept.” Also, none of the above auditing methods involve automating the process of determining the most appropriate classification(s) for each concept. Our work differs from the above related work in that we reclassify the concepts directly and automatically by statistical models.

Although the implementation varies with different similarity formulae, the basic idea is that the similarity of two words w₁, w₂ can be calculated through the two distributions P(C|w₁) and P(C|w₂), where C is the union of the w₁, w₂ context features, which in the simplest form would be words that co-occur in a corpus with w₁ and w₂. That is, the more overlap there is between the contexts that co-occur with both w₁ and w₂, the more similar w₁ and w₂ are semantically. A practical issue of implementing distributional similarity is the probability estimation for unseen (c_i, w_j) pairs, where c_i is an element of C that may co-occur with w₁ but not w₂ or vice versa. Recently, a widely cited solution has been Lee’s α-skew divergence,³⁹ which was derived from the Kullback-Leibler (KL) divergence (or relative entropy):⁴⁰

(1)

The KL divergence quantifies the coding inefficiency of assuming the distribution is Q while the true distribution is P.⁴¹ It is preferably called “divergence” because it does not satisfy the symmetric property (i.e., D(P ‖ Q) = D(Q ‖ P)) or the triangle inequality (i.e., D(P ‖ Q) ≤ D(P ‖ R) + D(R ‖ Q)) of Euclidean distance measures. The α-skew divergence is an asymmetric generalization of the KL divergence (here we substitute the above example into the following general formula):

(2)

where 0 ≤ α ≤ 1, and D is the KL divergence. The linear combination in Sα helps estimate the conditional probabilities of unseen {c_i, w₁} pairs in P(C|w₁) by a magnitude of (1 − α) · P(c_i|w₂) and thus smoothes the model.

Another important issue about the syntactic dependencies is feature-weighting. Assigning a weight to each syntactic dependency with respect to a particular word estimates the importance of that dependency in characterizing the word. For example, if the adjective “malignant” is frequently used to modify disorder terms such as “tumor,” then “malignant” should be assigned a relatively high weight with respect to “tumor.” In contrast, “malignant” rarely modifies a procedure type of term such as “cryotherapy” and thus should be weighed less correspondingly. In Cimiano and Völker’s work mentioned above, they reported that α-skew divergence performed best with conditional probability as the feature-weighting function, and therefore we adopted it for the α-skew divergence implemented in this paper (as described in detail in the Materials and Methods section).

The methods proposed in this paper differ from the above related work because to the best of our knowledge, this is the first time a distribution-based method has been proposed for automatically classifying UMLS concepts. The proposed distributional approach uses syntactic dependencies from shallow parsing to classify UMLS concepts. In addition, we used existing knowledge resources provided by the National Library of Medicine (NLM) to build our models for reclassifying and validating UMLS concepts.

• 1 The UMLS Semantic Network
  As discussed above, some high-level SN types (e.g., “Health Care Activity”) are known to subsume semantically heterogeneous concepts. These types are not suitable to be used in training distributional models. However, there are also well-defined SN types (e.g., “Diagnostic Procedure”) that generally contain semantically homogenous concepts. These well-defined SN types can be grouped to form broader semantic classes for which distributional classifiers can be built, as will be described in Material & Methods. The SN type assignments of the CUIs are also subject to manual curation and changes over different UMLS releases. These updates were made to improve the UMLS and thus should be relatively accurate. Therefore, it should be possible to use the updates as an economical way to obtain a test set (and gold standard) of a moderate sample size.
• 2 MetaMap and the MEDLINE/PubMed Baseline Repository
  Associated with the UMLS, the MetaMap⁴⁹ program was developed by the NLM to map terms in free text to UMLS concepts. The NLM also maintains the MEDLINE/PubMed Baseline Repository (MBR)⁵⁰ annual databases that represent a static view of all completed citations in MEDLINE up to a particular year, with 14,792,864 citations of the whole 2005 database processed by MetaMap into human- and machine-readable formats. The MetaMap machine-readable outputs can serve an important role for implementing concept-based distributional similarity in two ways: 1) the machine-readable outputs contain detailed information about the shallow parsing and concept mapping (see Supplement 1, available as an online data supplement at www.jamia.org), which is valuable for obtaining the syntactic dependencies; 2) MetaMap facilitates the aggregation of synonymous terms along with their syntactic dependencies into concepts (e.g., both “cryotherapy” and “cold therapy” are mapped to C0010412).

The two major components of the material were: 1) the UMLS 2006AA and 2005AA MRSTY.RRF files, from which we obtained an initial test set containing 5,996 CUIs that had been assigned different SN type(s); 2) two disjoint training corpora requested from the 2005 MBR database in MetaMap machine-readable format: one consisted of 100,000 title/abstracts and the other consisted of 99,313 (we will refer to them as the 100K and the 99K set hereafter).

Figure 1 provides an overview of different steps of the classification method, which can be summarized as: 1a) the MBR corpus was used to determine syntactic dependencies of the CUIs; 1b) seven training classes were determined, the CUIs belonging to each class were identified, their syntactic dependencies collected (however, test CUIs were excluded), weights were calculated and normalized, and a distribution of the dependencies was formed for each class; 1c) for each test CUI, the syntactic dependencies of the CUI were collected, weights of the corresponding features were calculated and normalized, and a distribution of the dependencies was formed for the CUI. Finally, as shown on the right-hand side between Figure 1b and 1c, the α-skew divergence was used to compute the distributional similarities, and then the class with the closest similarity score was proposed as the appropriate class. In the following subsections we describe each step of the methods in more detail.

Figure 1 (a) Preparing the syntactic dependencies. We post-processed the MBR corpora and applied the context-searching rules to extract syntactic dependencies of the CUIs in the corpora. The extractions then provided the syntactic dependencies of the CUIs required (more ...)

Figure 2 A sentence with POS tags, phrase types, and CUI mappings identified from the MetaMap machine-readable format. Notations: S: sentence, P: phrase and phrase type, W: word and POS, asterisk marks the head noun, E: CUI, mapped concept, SN type.

Figure 3 (a) The rules applied to extract the syntactic dependencies from Figure 2. (b) The syntactic dependencies extracted from Figure 2.

The syntactic dependencies associated with each broad class were pooled to form a distribution profile of the class, so that there were seven such training distributions (in Figure 1b we only show the procedure distribution but the process for the others was the same). We adopted the conditional probability function P(class|syntactic dependency) to compute the weights, and each conditional probability was estimated by frequency(class ∩ syntactic dependency)/frequency(syntactic dependency), according to the frequencies in the corpus. For each class, the weighted syntactic dependencies were then normalized to form a probability distribution, e.g., in Figure 1b, the X-axis of the procedure distribution represents the union of all the syntactic dependencies extracted from the corpus, with their probabilities summing to 1.

The syntactic dependencies associated with each test CUI (e.g., that of CUI_k in Figure 1c) were pooled to form the testing distribution of that CUI, i.e., each test CUI was associated with its own distribution profile (in Figure 1c we only show the distribution for one test CUI). For the test CUIs, the weights of the syntactic dependencies were calculated using P(CUI|syntactic dependencies) = frequency(CUI ∩ syntactic dependency)/frequency(syntactic dependency), and the they were normalized in the same way as the class distributions.

• 1 Evaluation of the gold standard
  In order to verify the quality of the gold standard, 50 CUIs in the gold standard set were randomly selected and shown to an expert with a biomedical background. The CUIs along with all their associated strings (with parenthesized annotations removed) from the MRCONSO.RRF table were provided to the expert. He was asked to select one or more of the seven classes that he considered to be most appropriate for each CUI. A disagreement was counted when the expert’s classification disagreed with the gold standard; an agreement was counted when the expert’s classification overlapped totally with the gold standard, and a partial agreement was counted when the two partially overlapped. For the C1442395 “Small T cell lymphoma” example above, if the expert considered it to be both an anatomy and a disorder class, then it was counted as a partial agreement with the gold standard, but if the expert considered it to be disorder only, it was considered as a complete agreement. After the first evaluation round, the expert was shown the partial agreements and disagreements, and was asked if he would change his mind on some of his previous decisions. Then a second round evaluation was made accordingly.
• 2 Evaluation of the classification method
  The main outcome measure was error rate (see formula 3 below), which was calculated by comparing differences in classification between the automated method and the gold standard. We experimented with three main factors in the evaluation: 1) the number of syntactic dependencies was varied, 2) two different training corpora (i.e., the 100K and 99K) were used, and 3) two sets of error rates were calculated so that one was based on the best prediction only and the other was based on consideration of the top two predictions. When counting the correct/incorrect classifications, the count for a single test CUI was divided by the number of classes it was associated with in the gold standard. For example, if in the gold standard a CUI belonged to both classes gene protein and substance, and in the top 2 predictions of the classifier only substance was obtained, it was counted as a 0.5 correct classification and a 0.5 misclassification. A tie was defined when the correct/incorrect predictions received equal similarity scores from the classifier. The error rates were calculated by the formula:

  (3)

  where M is the number of misclassifications, T is the number of ties, and N is the total number of CUIs tested. The error rate can be understood as complementary to classification accuracy. We chose the misclassifications made by our optimal model (trained on the 99K corpus and having at least 10 syntactic dependencies) and summarized those for the misclassified classes. In addition, for each misclassified CUI we examined the ranking that was determined by our method to see where in the ranking the correct class occurred.
• 3 Estimation of the classification coverage
  We estimated the potential number of CUIs that can be classified by our method. First we estimated the recall of MetaMap by counting the number of unique CUIs identified in the whole 2005 MBR corpus, as well as how many belonged to the seven classes (according to Supplement 2, available as an online data supplement at www.jamia.org). Then within our 199K training corpus (i.e., obtained by combining the 100k and 99k corpora), we calculated the number of unique CUIs identified by MetaMap and the number of these unique CUIs that have at least one syntactic dependency extracted by our context-searching rules, which provided estimation of the recall of our context-searching rules. Within the CUIs having at least one syntactic dependency, we created a histogram of the number of syntactic dependencies and normalized the frequencies, which provided estimation of the proportion of the CUIs with varying abundance of features, assuming the proportions remain consistent with the expansion of the training corpus. Through the cascade described above, we could approximate the number of CUIs that would be classified under varying accuracies.

Our goal was to reclassify UMLS concepts into broader classes that would be more suitable for the needs of NLP applications, and therefore we did not intend to apply the method to classifying the concepts into fine-grained classes (e.g., the original 135 SN types). However, the differentiation by our method of gene protein from substance revealed a potential for coping with a finer granularity. The feasibility study based on the seven broad classes showed promising results, and more importantly, demonstrated how NLP and a knowledge base may reciprocally support each other. In the following subsections we discuss various issues related to our results and methods.

Another issue involves the information that was made available to the expert for the classification task. The expert was shown only concept strings in order to avoid having him see existing ontological knowledge encoded in the UMLS, since this could influence his judgment. This was also the reason why the parenthesized annotations were removed from the strings because they contained classifications. Although in this approach, the information available to the expert was more limited, the expert made his own judgments under a more objective condition. For example, the expert’s diverse assignments for “Nephrolithiasis” above implied his domain knowledge sufficed in spite of the limited information. In addition, had the expert been given more information such as the contextual relations with other concepts, his agreement with the gold standard would likely have been even higher. For example, the parent concept “Kidney Diseases” could have influenced him to classify “Nephrolithiasis” only as disorder, which would then result in complete agreement with the gold standard.

There were also issues in using the MBR 2005 database as the training data. First, the behavior of different MetaMap versions reflects the changing configurations of each UMLS release. For example, in one of the 2005 MBR training abstracts “posterior cricoarytenoideus” was correctly identified as C0448337, but all later occurrences of its abbreviation “PCA” were uniquely mapped to C0030625 “passive cutaneous anaphylaxis,” which contaminated the features of the procedure class with that of the anatomy class. By examining the error on the Web-based MetaMap version, we found it was due to the 2005 Knowledge Sources. Second, the old SN classification used in the previous MetaMap version was more error-prone (e.g., C0333343 “Body cavities” was incorrectly assigned to T190 “Anatomical Abnormality”), but this could be solved by synchronizing it with the up-to-date SN classification. Third, we found that in the MBR 2005 database the POS tagger used by the MetaMap at that time had a bug in processing all the “do”- and “have”-derived auxiliary verbs. We dealt with the known errors by discarding all sentences with the affected auxiliary verbs from our training corpora. Estimated from the 199K corpora, the discarded sentences constituted 10.5% (100,855/960,491) of all the sentences which were associated with CUIs. This also resulted in underestimation of the recall of our context-searching rules, which should be higher than the 88% reported in Results. We believe that the performance of our method will gain improvement from the future improved releases of the MBR databases and that MetaMap will continue to be a valuable resource. Fourth, using directly the pre-processed MBR corpus might not be optimal for building the distributional models because some potentially better parameters of MetaMap were not explored.

We estimated in Results that only 22% of UMLS concepts were identified in the entire 2005 MBR database. However, a recall of 22% underestimates the coverage of usable concepts with respect to text processing tasks. Many UMLS concepts never appear in text because they are too specific or artificial to occur in natural language (e.g., C0232298 “Left axis deviation greater than −90 degrees by EKG”). This phenomenon correlates well with the estimation by McCray et al.⁵³ that only about 10% of the UMLS strings could be found in MEDLINE. Therefore, many UMLS concepts are not applicable for text processing applications. In addition, many multi-term concepts are identified by MetaMap as separate constituent concepts. For example, although the term “Hypertensive spasm of cardiac sphincter” corresponding to C0232593 cannot be mapped fully, the component concepts C0857121 “Hypertensive,” C0037763 “Spasm,” and C0227192 “Cardiac sphincter” can be mapped appropriately.

We approximated syntactic dependencies from the shallow-parsed sentences by manually created rules, since it was an economical way to use the MBR corpus. Our approach was similar to Cimiano and Völker’s, but we used richer parse information obtained from MetaMap and applied more comprehensive context-searching rules. However, there were some errors caused by using an incomplete parse. For example, the rule for searching the active verb of a head noun, i.e., rule 1c of Figure 3a, would fail to differentiate verbs as the past tense or as the perfect participle. In the sentence “normal peripheral blood B cells expressed high levels of CDw75…” from a training abstract, the rule captured “expressed” correctly as the active verb of “B cells,” but incorrectly chose “expressed” as the active verb of “cell-surface antigen” in the sentence “… CDw75 is a sialylated cell-surface antigen expressed in a number of tissue-specific isoforms.” The wrong syntactic dependencies could have introduced some noise into the features of the class distributions as well as into some test CUI distributions. In this paper we have not explored variations of our rules, but in the future we will experiment with stricter rules and use of a larger corpus to address the potential sparseness issue.

The results related to the number of syntactic dependencies agreed with our general intuition that having more features increases the classification accuracy. In practice we will request a larger training corpus to achieve higher accuracy and to cover more concepts. However, this should not adversely affect the use of an automated method because although the corpus-based approach will require large training data, the classifiers are not subject to frequent changes once they are built. Moreover, by studying the number of syntactic dependencies (per concept) extracted from the training corpora (both had median of 13), the result that the 99K outperformed the 100K on the same 157 test CUIs implied that the quality of the corpus is more important than the quantity. Therefore, finding methods to obtain a more compact but qualified training corpus is another topic worth further study.

The error rates when considering the top two predictions were much lower than when considering the top prediction, which implied the ranking mechanism of the distributional classification worked reasonably. In addition, we observed that many of the CUIs that did not belong to the seven classes and were excluded from training could be classified properly. We also foresee the method can offer the flexibility to create different semantic perspectives by grouping the semantic classes differently, which enables generating different semantically-oriented versions of the ontology for different applications. These derived topics are left for our future work.

In this paper we propose the use of a distributional similarity approach to classify and/or validate the semantic classification of UMLS concepts for NLP applications. This was achieved by utilizing a corpus processed by MetaMap, which proved to be a valuable resource for this work. The initial results of classifying CUIs according to seven broad semantic classes showed that the method is indeed feasible, with an estimated lowest error rate of 0.198. The performance was even more promising when considering the correct classification to be covered by the top two predictions, where the estimated lowest error rate was 0.116. We believe that the method can be further improved by implementation refinements and that the performance currently would be effective as an advising system.