Structured

1Corpora for Information Retrieval ResearchWilliam Hersh, MD Professor and ChairDepartment of Medical Informatics & Clinical EpidemiologyOregon Health & Science Universityhersh@ 2Overview•Information retrieval (IR) use case •Corpora developed for IR research–Clinical document retrieval –Genomics document retrieval –Image retrieval3The information retrieval (IR) process (Hersh, 2003)MetadataMetadata Queries Queries ContentContent SearchengineSearch engine Retrieval-Boolean-Natural languageIndexing-Words-Terms -Attributes4IR in science: from literatureto knowledgeAll literature Possibly relevantliterature Definitely relevantliteratureStructured knowledgeInformation retrievalInformationextraction,text mining5Evaluation of IR systems•System-oriented –how well system performs–Historically focused on relevance-based measures•Recall –# relevant retrieved / # relevant in collection •Precision –# relevant retrieved / # retrieved by search–When documents ranked, can aggregate both using mean average precision (MAP)•User-oriented –how well user performs with system–e.g., performing task, user satisfaction, etc.6System-oriented IR evaluation•Historically assessed with test collections, which consist of–Content –fixed yet realistic collections of documents, images, etc.–Topics –statements of information need that can be fashioned into queries entered into retrieval systems –Relevance judgments –by expert humans for which content items should be retrieved for which topics•Evaluation consists of “runs”with output for each topic measured and averaged7Some well-known system-orientedevaluation forums•Text Retrieval Conference (TREC, ; Voorhees, 2005)–Many “tracks”of interest, such as Web searching, question-answering, etc.–Non-medical, with exception of Genomics Track (Hersh, 2006)•Cross-Language Evaluation Forum (CLEF, )–Focus on retrieval across languages, European-based–Additional focus on image retrieval, which includes medical image retrieval tasks (Hersh, 2006)•Both operate on annual cycle of test collection release, experiments, and analysis of results8Corpora for IR evaluation•Clinical document retrieval –OHSUMED •Genomics document retrieval and categorization –The TREC Genomics Track•Image retrieval –ImageCLEFmed9General information•Needs and tasks–Information retrieval, i.e., search•Availability–Via registration and signing usage agreement (required by publishers)•Annotation–Use annotation from source–Topics and relevance judgments added10OHSUMEDftp:///pub/OHSUMED•Good early test collection with substantial use–My highest ranking citation in Google Scholar is ACM SIGIR conference proceedings paper (Hersh, 1994) about collection(!)•Stats–348,566 MEDLINE records–106 topics from general medicine–Relevance judgments at two levels, definitely and possibly relevant•Caveats–Old –documents from 1987-1991–“Clustered”collection –from general medical journals only, not all of MEDLINE11TREC Genomics Track•New biotechnologies have revolutionized data collection, management, and analysis •Prototype use case: gene microarray experiment generating dozens of genes previously unknown to be involved in a physiological or disease process•Scientists must find information about these genes efficiently12TREC Genomics Track /genomics/•Overall focus on retrieval of documents about genomics by biomedical researchers •2004-2005 –focus on two user tasks–Ad hoc retrieval –finding documents based on information needs (Hersh, 2006)–Article “triage”–does a paper have data that would warrant inclusion in the Mouse Genome Informatics database? (Cohen, 2006)•2006 –passage retrieval (Hersh, 2006)–Users approach the literature with questions and want•Passages of text that indicate answer in document •Aspects that passages on same topic•Linkage to documents that contain these passages13TREC 2006 Genomics –from passages to aspects to documentsPassagesAspectsDocuments14Image retrieval•Biomedical professionals increasingly use images for research, clinical care, andeducation, yet we know very little about how they find them•Image retrieval is a “poor stepchild”to textretrieval, with less understanding of how people use systems and how well they work•Most image retrieval work has focused on either text annotation retrieval or image processing, but–Not both in combination –Not from an IR perspective15Image retrieval approaches(Müller, 2004)•Semantic or textual –by text annotation, e.g.,–Narrative description–Controlled terminology assignment –Other types of textual metadata, e.g., modality, location•Visual–Identification of features, e.g., colors, texture, shape, segmentation16Example of visual image retrievalinterfacehttp://www.sim.hcuge.ch/medgift/w01_Presentation_EN.htm17ImageCLEFmed/image/•Aims to simulate general searching over wide variety of medical images•Uses standard IR approach with test collection consisting of–Content –Topics–Relevance judgments•Has operated through two cycles of CLEF (2005-2006)–Both used same image collection–Developed new topics and performed relevance judgments for each cycle18ImageCLEFmed 2005-2006test collectionGerman –7805English –780578057805PathologyPathoPICEnglish –323193231932319Pathology Pathology Education Instructional Resource (PEIR)English –4071177407Nuclear medicineMallinckrodt Institute of Radiology (MIR)English –177French –189987252076Mixed CasimageAnnotations Images Cases Predominant images Collection19Example case from CasimageID : 4272Description : A large hypoechoic mass is seen in the spleen. CDFI reveals it to be hypovascular and distorts the intrasplenic blood vessels. This lesion is consistent with a metastatic lesion. Urinary obstruction is present on the right with pelvo-caliceal and uretreal dilatation secondary to a soft tissue lesion at the junction of the ureter and baldder. This is another secondary lesion of the malignant melanoma. Surprisingly, these lesions are not hypervascular on doppler nor on CT. Metastasis are also visible in the liver.Diagnosis : Metastasis of spleen and ureter, malignant melanomaClinical Presentation : Workup in a patient with malignant melanoma. Intravenous pyelography showed no excretion of contrast on the right.ImagesCaseannotation20For more information•/•ftp:///pub/OHSUMED/•/genomics •/image•Questions?21References•Hersh WR,Information Retrieval: A Health and Biomedical Perspective , New York: Springer-Verlag, 2003.•Hersh WR, Buckley C, Leone TJ, Hickam DH, OHSUMED: an interactive retrieval evaluation and new large test collection for research, Proceedings of the 17th Annual ACM SIGIR Conference on Research and Development in Information Retrieval , 1994, 192-201.•Hersh WR, Bhupatiraju RT, Ross L, Johnson P, Cohen AM, Kraemer DF, Enhancing access to the bibliome: the TREC 2004 Genomics Track, Journal of Biomedical Discovery and Collaboration , 2006, 1:3, //content/1/1/3.•Cohen AM, Hersh WR, The TREC 2004 Genomics Track categorization task: classifying full text biomedical documents, Journal of Biomedical Discovery and Collaboration , 2006, 1:4, /content/1/1/4.•Hersh WR, Müller H, Jensen J, Yang J, Gorman P, Ruch P, Advancing biomedical image retrieval: development and analysis of a test collection, Journal of the American Medical Informatics Association , 2006, 13: 488-496.•Hersh W, Cohen AM, Roberts P, Rekapalli HP, TREC 2006 Genomics Track overview, The Fifteenth Text Retrieval Conference -TREC 2006, /~hersh/trec-06-genomics.pdf.。