Computer Assisted Retrieval of Biomedical Image Features from Spine X-rays: Progress and Prospects

Skip navigation

Subject Areas
Technical Reports

Computer Assisted Retrieval of Biomedical Image Features from Spine X-rays: Progress and Prospects

L. Rodney Long
G. R. Thoma
National Library of Medicine
Bethesda, MD 20894

Abstract

Image indexing for biomedical content is a prohibitively expensive task if done manually. This leads to the demand for effective automated or computer assisted indexing methods. We are doing research on this problem for a set of digitized x-ray films from the second National Health and Nutrition Examination Survey (NHANES II). The 10,000 cervical spine and 7,000 lumbar spine images from this survey are retrievable through a resource we have developed, the Web-based Medical Information Retrieval System (WebMIRS). Using WebMIRS, the user may retrieve x-rays based on text data, such as demographics, health questionnaire responses, or physican's examination results. Two National Institutes of Health workshops have identified visual features of the images specifically related to osteoarthritis, but the images have never been indexed for these features, which include anterior osteophytes, disc space narrowing, and subluxation. We are investigating the methods by which the indexing and retrieval of the images using these particular features, may be achieved, in a validated manner acceptable to the biomedical community. In this paper we review our work to date, plus the work of collaborating researchers, and present some possible outcomes that appear to be realistic

1. Introduction

The image data for our research consists of 10,000 cervical spine and 7,000 lumbar spine images digitized from films collected in the second National Health and Nutrition Examination Survey (NHANES II). The capability of retrieving x-ray images by important biomedical features has been a long-standing research goal within the biomedical image processing community. No comprehensive approach has yet been applied to a large image collection of x-rays in any practical system, to the best of our knowledge. In this paper we present a brief review of the relevant work that we and other researchers have done, and outline the remaining problems and potential solutions.

The creation of a database of x-ray images that are retrievable by image content requires the development of practical methods for both indexing and retrieval. To date we have concentrated on indexing. As part of our work, we have developed the Web-based Medical Information Retrieval System (WebMIRS) [1-2], a database system that allows retrieval of the images by associated text, such as demographics, health questionnaire responses, or physician's examination data. The images are viewable through WebMIRS at one-fourth their original spatial resolution. The full resolution images are available through a separate FTP site also operated by our branch of the National Library of Medicine [3]. In addition, we have developed a Digital Atlas of the Cervical and Lumbar Spine [4] that contains validated examples of the biomedical image features that could be used as indices to the images.

2. Specific goals for the x-ray database

The question of what biomedical image information is available and significant in our images was addressed by two workshops [5] held under the sponsorship of the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS). These workshops concluded that the following features could be determined from the images in repeatable fashion by human observers: for cervical spine images, anterior osteophytes, disc space narrowing, and subluxation; for lumbar spine images, anterior osteophytes, disc space narrowing, and spondylolisthesis. We have taken these recommendations as defining the basic features to be indexed in our image set. Along with these basic biomedical features, we have the goal of indexing and retrieving the images by basic geometric features, such as vertebra dimensions and spacing distances. Such measures are used in vertebral morphometry, where the ratio of anterior to posterior vertebral height may be used as a measure of fracture [6].

3. Accomplishments and published research in the field

3.1 Creation of reference data for vertebral structures

Figure 1. Vert. boundary points We have acquired a set of coordinate data for the cervical and lumbar spine vertebra that includes the key boundary points collected in vertebral morphometry. The collected points are illustrated in Figure 1. The data collected includes the standard six points of vertebral morphometry (points 1-6), plus the anterior midpoint, and points marking the extremities of any anterior osteophytes that are present. Data was also collected denoting the presence of fused vertebrae in the lumbar spine. The data collection was carried out under supervision of a board-certified radiologist with expertise in bone radiology, on 550 of the cervical and lumbar spine images. The data contains a large number of vertebrae showing the presence of anterior osteophytes, one of the conditions of interest for indexing, and is therefore a potential source of training data for algorithms to classify vertebrae according to the presence or absence of this biomedical feature. The significance of this data is that it provides a reference for segmentation algorithm development and validation, and for training of classification algorithms, at least for the anterior osteophyte classification. We have made the collected data available for research use through the WebMIRS system and on our FTP site.

3.2 High-level analysis and basic orientation in the images

Since, as we note in Section 3.3, the segmentation algorithms operate in a local region of the image assumed a priori to contain the structures of interest, it is necessary to determine this local region of interest (ROI) either interactively or with a preliminary high-level analysis of the image. Recent work toward this goal was reported by Zamora [7], who used a method based on line integrals of image grayscale to determine approximate spine axis locations on a test set of 40 cervical spine images. Using the reference data of Section 3.1 to independently calculate a "truth" spine axis, Zamora reported an orientation error in his algorithm of less than 15 degrees, for 34 of the cases. Zamora extended his method to estimate a rectangle conservatively containing the spine ROI, and used the Section 3.1 reference data to evaluate performance. For vertebrae C2-C6, the Zamora rectangle was found to contain the vertebral boundaries in over 90% of the cases.

We have previously shown [8-9] that basic landmark information appears to be preserved in the cervical spine images after heavy smearing and subsampling. In a test case of 48 cervical spine images, we successfully derived landmarks for skull, shoulder, and back-of-skull background in 46 cases. For the 46 successful cases, we further used the skull/shoulder landmarks to find the spine ROI by detecting characteristic curves assumed to lie in the spine ROI. (One of these is the curve of brightest grayscale along the length of the spine.) We estimated this curve for the 46 cases, then used our curve estimate in turn to estimate the spine orientation. We compared the orientations computed by this algorithm to the spine orientations computed using the data of Section 3.1 and found that in 45 of the cases, the orientation error was less than 15 degrees.

We have recently done work in direct search for spine region anatomy without initial low resolution analysis, by direct search of two of the most consistently prominent cervical spine image features: the back-of-skull contour, and the spinous process on the C1 vertebra (SPC1). Implementation of simple raster scan methods for skull boundary detection, followed by curve analysis of the resulting boundary, have yielded good initial results: in a test of 136 images, a conservative bounding box for SPC1 was found in 88% of the cases.

Global analysis of such radiological images for localizing regions of interest does not seem to be commonly reported in the literature, although it does exist. For example, Chwialkowki [10] reported "reasonably accurate and reliable" results in analyzing lateral MRI images of the lower spine for localizing such anatomy as the spinal canal. He used 26 MR studies with an average of 14 images per study. Interactive methods of ROI localization appear to predominate, however.

3.3 Segmentation of the vertebrae with local segmentation algorithms

In recent years researchers have published results using various forms of deformable template approaches to segment vertebrae. Without exception, these methods rely on user interaction for initialization, or the researchers report that lack of automated initialization methods is a signficant performance problem. A fundamental and comprehensive treatment of the whole field of Active Shape Modeling (ASM) has been provided by Cootes [11]. A semi-automated implementation of this approach has been achieved for the segmentation of lumbar spine dual x-ray absorptiometry (DEXA) images [12]; here, the user manually identifies two "anchor points" for placing a template; the template then deforms by ASM, maintaining invariance of the anchor points, which are placed at the top and bottom of a column of vertebrae. Gardner [13-14] developed a semi-automated system, based on active contour (snake) modeling of the vertebrae, which operated on digitized lumbar spine x-ray film. In this system, points on the vertebral boundaries are specified by the user, with assistance from the system in point placement. The selected boundary points then become constraints on the active contour that is automatically fit to the vertebra boundary. This process is carried out vertebra by vertebra. The same author has pointed out [15] the potential problems in taking dimensional measurements from vertebrae on x-rays, due to the projective nature of the imaging modality, which results in overlapping edges and concealed boundaries.

In [16] we reported work using ASM models of cervical spine vertebrae to segment C2/C3 vertebrae in a test set of 40 images. The segmentation performance was evaluated against a set of "truth" boundaries derived manually from the images, and mean point-to-point boundary errors were calculated. A number of erroneous results were observed, including convergence to tissue boundaries near the vertebrae, and convergence to lines on boundary regions interior to vertebrae. For 16 of the cases, the results showed a mean point-to-point error of less than 1/10 inch. Strong sensitivity of convergence to initial template positioning was observed in some cases. Work done independently by Sari-Sarraf [17] using similar techniques and tools also appeared to give successful results to a first-order level in a significant number of cases.

3.4 Identification (classification) of anatomy in the images

To date the only anatomical labeling we have investigated is gross level labeling of the shoulder and skull, although the spine curves we have calculated [9] admit the possibility of labeling the general spine region. Labeling of the individual vertebrae will follow vertebral segmentation and is expected to use skull anatomy (because of its relative ease and robustness of detection) as a beginning reference point.

3.5 Indexing (classification) by biomedical feature

The final indexing and classification step for our images requires taking the segmented, labeled images along with expert training data, and dividing the labeled structures into classes of normal or abnormal for the biomedical conditions of interest. It also generates all of the geometry measures desired from the data. Work toward this step that applies radius of curvature criteria to segmented vertebrae boundaries has been reported by Stanley [18].

4. Problems

The remaining technical problems in building a practical system for the retrieval of data by image content from this set of x-ray images include the need to solve the segmentation problem with methods that are robust, of fine granularity, and biomedically validated. In spite of the progress made, the test sets used to date are too small relative to the image collection. In addition, most of the work has focussed on the basic task of getting algorithms to produce boundaries that appear reasonable, in a first order sense, with little attention paid to the resolution quality of the segmentation achieved. This issue is closely related to the lack of critical review of the results by biomedical experts in the area. Each of these issues must be addressed for such systems to become accepted tools in the biomedical research community. It should be noted that both in our work and the work of Sari-Sarraf a number of problems were observed, and technical issues that require resolution were raised. Among the most outstanding of these are the need for a good method for initializing ASM to segment the vertebrae , the need to investigate the possiblity of improved modeling of the image grayscale distribution in the neighborhood of vertebrae boundary points (perhaps with a mixed Gaussian pdf), and the need to understand nonconvergence of the algorithm for certain images even when the template initial conditions are set near known truth. A final significant outstanding problem is automated failure detection for the cases of nonconvergence or convergence to erroneous solutions.

5. Prospects and conclusion

As judged by the work of Gardner and Smyth, the prospects for achieving an interactive system for basic-level segmenting of our images under user guidance appear quite good, although it is not certain that the quality of segmentation achievable will allow classification of the biomedical features of interest with an accuracy comparable to a human expert. This remains to be determined experimentally. The possibility of fully automating the segmentation and indexing remains a more problematic and distant goal; some of the imposing problems appear tractable with time and effort, such as automatic detection of "problem" images by histogram analysis, or initialization of template positioning by grayscale line integral analysis or low resolution image analysis; but other problems, such as automatic failure detection, appear more fundamental, and no clear approach has been proposed.

6. References

1. Long LR, Pillemer SR, Lawrence RC, Goh G-H, Neve L, Thoma GR. World Wide Web platform-independent access to biomedical text/image databases. Proceedings of SPIE Medical Imaging 1998: PACS Design and Evaluation: Engineering and Clinical Issues, SPIE Vol. 3339, San Diego, CA, February 21-26, 1998, pp. 52-63.
2. Long LR, Pillemer SR, Lawrence RC, Goh G-H, Neve L, Thoma GR. WebMIRS: Web-based Medical Information Retrieval System. Proceedings of SPIE Storage and Retrieval for Image and Video Databases VI, SPIE vol. 3312, San Jose, CA, January 24-30, 1998, pp. 392-403.
3. See the Web site of the NLM Communications Engineering Branch: http://archive.nlm.nih.gov
4. Long LR, Pillemer S, Goh G-H, Berman LE, Neve L,Thoma GR, Premkumar A, Ostchega Y, Lawrence R, Altman RD, Lane NE, Scott WW, Jr. A digital atlas for spinal x-rays. Proceedings of SPIE Medical Imaging 1997: PACS Design and Evaluation: Engineering and Clinical Issues, SPIE vol. 3035, Newport Beach, CA, February 22-28, 1997, pp. 586-594.
5. Digitized Radiographic Images: Challenges and Opportunities. Chaired by: Stanley R. Pillemer, M.D. June 2-3, 1993, Bethesda, MD, sponsored by the National Institute of Arthristis and Musculoskeletal and Skin Diseases (NIAMS), the National Center for Health Statistics, and the National Library of Medicine. Digitized Radiographic Images. Chaired by Stanley R. Pillemer, M.D., Reva C. Lawrence, M.P.H. May 9, 1995, Bethesda, MD, sponsored by NIAMS.
6. Hedlund LR, Gallagher JC. Vertebral morphometry in diagnosis of spinal fractures. Bone and Mineral, Vol. 5, 1988, pp. 59-67.
7. Zamora G, Sari-Sarraf H, Mitra S. Estimation of orientation of cervical vertebrae for segmentation with active shape models. Proceedings of SPIE Medical Imaging 2001: Image Processing, San Diego, CA, February 17-23, 2001.
8. Long LR, Thoma GR. Feature indexing in a database of digitized x-rays. Proceedings of SPIE Electronic Imaging 2001: Storage and Retrieval for Media Databases. Vol. 4315, San Jose, CA, January 21-26, 2001, pp. 393-403.
9. Long LR, Thoma GR. Identification and classification of spine vertebrae by automated methods. Proceedings of SPIE Medical Imaging 2001: Image Processing. Vol. 4322, San Diego, CA, February 17-22, 2001.
10. Chwialkowski MP, Shile PE, Pfeifer D, Parkey RW, Peshock RM. Automated localization and identification of lower spinal anatomy in magnetic resonance images. Computers and Biomedical Research, vol. 24, 1991, pp. 99-117.
11. Cootes TF, Taylor CJ. Statistical models of appearance for computer vision. Wolfson Image Analysis Unit, Imaging Science and Biomedical Engineering, University of Manchester, September 13, 1999. Draft report, available at http://www.wiau.man.ac.uk.
12. Smyth PP, Taylor CJ, Adams JE. Vertebral shape: automatic measurements with active shape models. Radiology, May 1999, vol. 211, no. 2, pp. 571-578.
13. Gardner JC, Heyano SL, Yaffe LG, von Ingersleben G, Chesnut CH III. A semi-automated computerized system for fracture assessment of spinal x-ray films. Proceedings of SPIE Medical Imaging 1996: Image Processing, SPIE vol. 2710, February 12-15, 1996, Newport Beach, CA, pp. 996-1008.
14. Gardner JC, von Ingersleben G, Heyano SL, Chesnut III, CH. An interactive tutorial-based training technique for vertebral morphometry. Osteoporosis International, vol. 12, 2001, pp. 63-70.
15. Gardner JC, Yaffe LG, Johansen JM, von Ingersleben G, Chesnut CH III. Problems with six point vertebral morphometry. Proceedings of SPIE Medical Imaging 1998: Image Processing, SPIE vol. 3338, San Diego, CA, February 23-26, 1998, pp. 107-117.
16. Long LR, Thoma GR. Use of shape models to search digitized spine x-rays. Proceedings of IEEE Computer Based Medical Systems 2000, Houston, TX, June 23-24, 2000, pp. 255-260.
17. Sari-Sarraf H, Mitra S, Zamora G, Tezmol A. Customized active shape models for segmentation of cervical and lumbar spine vertebrae, Texas Tech University College of Electrical Engineering technical report, available at http://www.cvial.ttu.edu/~sarraf
18. Stanley RJ, Long R. A radius of curvature approach to cervical spine vertebra image analysis. Proceedings of the 38th Annual Rocky Mountain Bioengineering Symposium, Copper Mountain, Colorado, April 20-22, 2001, pp. 385-90.

URL: http://archive.nlm.nih.gov/pubs/long/cbms2001/cbms2001.php
Last updated April 23, 2003

Send questions or comments about this site to