Image Text Integration project

 

Table of Contents

Introduction

The goala of the ITI project are 1) to automatically annotate biomedical images extracted from scientific publications with respect to their modality, content, usefulness for clinical decision support and instructional purposes; and 2) project the annotations onto unannotated images stored in databases or obtained in a clinical setting by linking images through content-based image similarity.

Image annotation granularity and facets

  • coarse; addresses:
    • image modality
    • relation to a specific clinical task (image utility)
    • body location
  • medium; provides a more detailed description of the image using existing biomedical domain ontologies
  • specific; provides very detailed descriptions of clinical entities and events in an image using terms that are not included in existing ontologies and often are familiar only to clinicians specializing in a narrow area of medicine.

Publications

  1. Simpson M, Demner-Fushman D, Sneiderman C, Antani SK Thoma GR.Using Non-lexical Features to Identify Effective Indexing Terms for Biomedical Illustrations
    Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL-09),Athens, Greece, April 2009
  2. Sneiderman C, Demner-Fushman D, Fung KW, Bray B. UMLS-based Automatic Image Indexing.
    Proceedings of the 2008 Annual Symposium of the American Medical Information Association (AMIA 2008), Washington, DC, November 2008
  3. Demner-Fushman D, Antani S, Simpson M, Thoma GR. Combining Medical Domain Ontological Knowledge and Low-level Image Features for Multimedia Indexing
    LREC 2008 (Sixth International Conference on Language Resources and Evaluation), OntoImage Workshop, Marrakech, Morocco, May 2008
  4. Antani S, Demner-Fushman D, Li J, Srinivasan BV, Thoma GR. Exploring use of images in clinical articles for decision support in Evidence-Based Medicine.
    Proc. SPIE-IS&T Electronic Imaging. San Jose, CA. January 2008;6815:68150Q(1-10)
  5. Demner-Fushman D, Antani SK, Thoma GR. Automatically Finding Images for Clinical Decision Support.
    Proceedings of IEEE International Workshop on Data Mining in Medicine (DM-Med 2007). Omaha, NE. October 2007;:139-44
  6. Li J, Demner-Fushman D, Antani SK, Thoma GR. Localizing Text and Symbols in Images from Biomedical Journal Articles.
    Poster at 20th NIH Research Festival. National Institutes of Health September 2007
  7. Srinivasan B, Antani SK, Demner-Fushman D, Thoma GR. Identification and Segmentation of Multi-Panel Images in Biomedical Journal Articles
    Poster at 20th NIH Research Festival. National Institutes of Health. September 2007

ITI Processes

  • Text processing:
    • Caption extraction and segmentation
    • Mention extraction
    • Biomedical terminology extraction
  • Image processing:
    • multi-panel figure segmentation
    • text and symbols localization
    • color and texture features computation
  • Image classification using supervised machine learning
  • Image annotation:
    • manual coarse level annotation
    • automatic UMLS-based medium-level annotation (collaboration with Dr. Charles Sneiderman)
    • manual specific annotation (collaboration with Dr. Bruce Bray)

Evaluation and Annotation interface

The user interface developed and maintained by Matthew Simpson is available to registered users here

The user interface tutorial developed by Michael Chung is available here