| Foreword | 6 |
---|
| Preface | 8 |
---|
| Acknowledgements | 10 |
---|
| Contents | 12 |
---|
| List of Contributors | 24 |
---|
| Introduction | 28 |
---|
| Seven Years of Image Retrieval Evaluation | 30 |
| Introduction | 30 |
| Evaluation of IR Systems | 32 |
| IR Test Collections | 33 |
| Cross--Language Evaluation Forum (CLEF) | 36 |
| ImageCLEF | 36 |
| Aim and Objectives | 36 |
| Tasks and Participants | 38 |
| Data sets | 39 |
| Contributions | 39 |
| Organisational Challenges | 41 |
| Conclusions | 42 |
| References | 43 |
| Data Sets Created in ImageCLEF | 46 |
| Introduction | 46 |
| Collection Creation | 47 |
| Requirements and Specification | 48 |
| Collection Overview | 50 |
| Image Collections for Photographic Retrieval | 51 |
| The St. Andrews Collection of Historic Photographs | 51 |
| The IAPR TC--12 Database | 53 |
| The Belga News Agency Photographic Collection | 55 |
| Image Collections for Medical Retrieval | 56 |
| The ImageCLEFmed Teaching Files | 57 |
| The RSNA Database | 61 |
| Automatic Image Annotation and Object Recognition | 62 |
| The IRMA Database | 62 |
| The LookThatUp (LTU) Data set | 63 |
| The PASCAL Object Recognition Database | 64 |
| The MIR Flickr Image Data Set | 65 |
| Image Collections in Other Tasks | 65 |
| The INEX MM Wikipedia Collection | 66 |
| The KTH--IDOL2 Database | 67 |
| Conclusions | 68 |
| References | 69 |
| Creating Realistic Topics for Image Retrieval Evaluation | 71 |
| Introduction | 71 |
| User Models and Information Sources | 74 |
| Machine--Oriented Evaluation | 74 |
| User Models | 75 |
| Information Sources for Topic Creation | 76 |
| Concrete Examples for Generated Visual Topics in Several Domains | 79 |
| Photographic Retrieval | 79 |
| Medical Retrieval | 80 |
| The Influence of Topics on the Results of Evaluation | 81 |
| Classifying Topics Into Categories | 82 |
| Links Between Topics and the Relevance Judgments | 83 |
| What Can Be Evaluated and What Can Not? | 83 |
| Conclusions | 84 |
| References | 85 |
| Relevance Judgments for Image Retrieval Evaluation | 88 |
| Introduction | 88 |
| Overview of Relevance Judgments in Information Retrieval | 89 |
| Test Collections | 89 |
| Relevance Judgments | 90 |
| Relevance Judging for the ImageCLEF Medical Retrieval Task | 97 |
| Topics and Collection | 97 |
| Judges | 98 |
| Relevance Judgment Systems and the Process of Judging | 99 |
| Conclusions and Future Work | 103 |
| References | 104 |
| Performance Measures Used in Image Information Retrieval | 106 |
| Evaluation Measures Used in ImageCLEF | 106 |
| Measures for Retrieval | 107 |
| Measuring at Fixed Recall | 108 |
| Measuring at Fixed Rank | 110 |
| Measures for Diversity | 112 |
| Collating Two Measures Into One | 113 |
| Miscellaneous Measures | 113 |
| Considering Multiple Measures | 114 |
| Measures for Image Annotation and Concept Detection | 115 |
| Use of Measures in ImageCLEF | 116 |
| Conclusions | 117 |
| References | 117 |
| Fusion Techniques for Combining Textual and Visual Information Retrieval | 120 |
| Introduction | 120 |
| Information Fusion and Orthogonality | 122 |
| Methods | 123 |
| Results | 123 |
| Early Fusion Approaches | 123 |
| Late Fusion Approaches | 124 |
| Inter--media Feedback with Query Expansion | 129 |
| Other Approaches | 130 |
| Overview of the Methods from 2004--2009 | 130 |
| Justification for the Approaches and Generally Known Problems | 130 |
| Conclusions | 133 |
| References | 133 |
| Track Reports | 140 |
---|
| Interactive Image Retrieval | 142 |
| Interactive Studies in Information Retrieval | 142 |
| iCLEF Experiments on Interactive Image Retrieval | 144 |
| iCLEF Image Retrieval Experiments: The Latin Square Phase | 145 |
| iCLEF Experiments with Flickr | 148 |
| The Target Collection: Flickr | 149 |
| Annotations | 149 |
| The Task | 150 |
| Experiments | 152 |
| Task Space, Technology and Research Questions | 159 |
| Use Cases for Interactive Image Retrieval | 159 |
| Challenges: Technology and Interaction | 160 |
| References | 162 |
| Photographic Image Retrieval | 165 |
| Introduction | 165 |
| Ad hoc Retrieval of Historic Photographs: ImageCLEF 2003--2005 | 166 |
| Test Collection and Distribution | 167 |
| Query Topics | 168 |
| Relevance Judgments and Performance Measures | 171 |
| Results and Analysis | 171 |
| Ad hoc Retrieval of Generic Photographs: ImageCLEFphoto 2006-2007 | 173 |
| Test Collection and Distribution | 174 |
| Query Topics | 175 |
| Relevance Judgments and Performance Measures | 176 |
| Results and Analysis | 177 |
| Visual Sub--task | 178 |
| Ad hoc Retrieval and Result Diversity: ImageCLEFphoto 2008--2009 | 179 |
| Test Collection and Distribution | 179 |
| Query Topics | 180 |
| Relevance Judgments and Performance Measures | 182 |
| Results and Analysis | 182 |
| Conclusion and Future Prospects | 184 |
| References | 185 |
| The Wikipedia Image Retrieval Task | 187 |
| Introduction | 187 |
|