Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 1 DESIGN CONSIDERATIONS FOR DIGITAL IMAGE LIBRARIES Anthony J. Maeder School of Computing, Engineering and Mathematics University of Western Sydney Sydney, Australia a.maeder@uws.edu.au Presented at the 2nd International Conference on African Digital Libraries and Archives (ICADLA-2), University of Witwatersrand, Johannesburg, South Africa, 14th – 18th November, 2011 Abstract Design of digital image libraries requires choices for numerous configuration aspects, such as resolution and display settings. These aspects can be categorized into different types of design criteria based on whether they are a human viewing and usage factor, or a stage in the image library management process. The criteria can also be applied in a hierarchy of nested versions of access to the library to suit different usage circumstances. Here we present a framework for design criteria using this approach, and apply it to some example cases. Keywords: digital image, digitization, human visual system Introduction Digital image libraries may be constructed to serve different purposes for usage, ranging from human viewing and understanding (e.g. physical artefacts or historical images) to computational analysis and automated interpretation (e.g. satellite or medical images). Our interest here is for libraries of images of the first type (which are loosely termed “natural scene images”), as frequently held by museums or archives. The extent of user utility for such libraries will be directly affected by choices in design criteria used for the digital image representation. Often choices are made for these criteria on an ad hoc or default basis, and so some confusion over the soundness and reasoning behind such choices may result. Here we will discuss categorization of the design criteria and recommend reasonable approaches for making the related choices, based on simple properties of human visual perception. A scheme for applying the criteria hierarchically will be described, to enable a succession of images with graded properties to be produced for different purposes. To limit the scope of this work to a manageable range we will concentrate on digital image libraries of scanned photographic images, rather than special purpose content such as documents or artworks. Similar principles are relevant also to directly acquired digital images of similar content. Examples of applying these criteria will be provided for some typical image library situations. Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 2 Design Criteria Digital image libraries typically provide access to natural scene images in high quality digitally processed formats expressly selected for human viewing satisfaction and for utilization of the images in other media. Choices of design criteria values used for representing images in these formats are made with the intention of allowing ease of assimilation or recognition of the scene contents, and high quality rendition or reproduction capabilities respectively. Some examples of standard approaches to digital image processing of these images are intensity transformations (e.g. to match human visual system response) or colour table mapping (e.g. to provide consistency with scenes of similar content) as provided by “auto-correction” functions in digital image manipulation software. Images can also be cropped or airbrushed to remove extraneous detail, and subjected to appearance improvement operations such as de-blurring, de-noising, or edge enhancement. Images may be stored in various formats which translate the input data into another mode of representation (e.g. intensities may be mapped in a lookup table rather than as raw values). Compression and structured storage protocols may be applied. When the image is presented to the user, it may be within a window as part of the screen layout, or as full screen, or printed to a page. All of these aspects must influence value choices. Appropriate design criteria for digital image libraries therefore depend on a range of contextual considerations including complexity of the images, viewer recognition of content, ease of search and access, and protection of commercial interests. Previous work (e.g. Frey & Susstrunk 1996) has suggested a wide range of such criteria, which may be interrelated. Here we suggest a high level systematic approach to group such criteria together, in order to allow the interrelationships to be made explicit. It is helpful in developing this structure, to consider design criteria in two different dimensions: psycho-physical factors (related to visual perception behaviour) and usage stages (related to digital image library management functions). Factors are based on fundamental aspects of the human visual system, such as visual acuity, contrast sensitivity, colour constancy, visual attention. The “early vision” aspects of these are widely characterized in literature, but the effects of image content and quality are less conclusively understood. The concept of image “quality” (Frey & Susstrunk 1996) denotes the apparent noise in the image content which would disrupt pure visual information, while the concept of “attention” (Parkhurst, Law & Niebuhr 2002) indicates where information of interest to the user is located in an image. A method for combining these aspects together with low level vision components to map image “importance” has been developed (Maeder 2005) and applied to a range of visual characterization tasks. Recent work on “interestingness” (Halonen, Westman & Oittinen 2009) supports this approach. Stages are based on the operations performed on digital images to support the use of the library. The basic concepts of Frey & Susstrunk (1996) are informative for this. It is necessary to use different forms of representation for images at the different stages, so that the different image management tasks are conducted with appropriate accuracy and ease. For instance, if choices for spatial and intensity factors are made at the time of the original digitization, these need not necessarily be preserved for image retention and presentation purposes. Nevertheless they need to be chosen to give Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 3 sufficient data for any required image enhancement or content analysis operations to be performed before the image is incorporated in the library. We define the specific high level classes of interest for digital image library design criteria within these two dimensions as follows: Factors Spatial: pixel density, spacing, aspect ratio, shape, size; Intensity: pixel brightness, contrast, colour values, gamut; Quality: visual appearance sharpness, clarity, aliasing; Inform: visual information density, localization, spread. Stages Input: image capture by digital photography or scanning; Process: image enhancement, feature detection or emphasis; Store: image formatting or compression to retain long term; Display: image reconstruction, presentation on screen/print. For a given digital image library situation, design choices must be made for each class of factors, at each stage. For instance, choice of spatial and intensity settings for the digitization at the input stage will be informed by the underlying quality of the original photographs, and how much information of interest the viewer would be able to see in them. Image sources such as amateur photographs or holiday postcards will be lower in quality and information than studio photographs of handmade household objects or large format landscape/industrial photographs. Once digitized, the image may be subject to some spatial and intensity improvements such as sharpening, using pixel decimation filtering, and contrast enhancement, using histogram redistribution. The image quality would be raised at this stage but the information content for viewer attention would be unchanged. For storage purposes, a highly compressed format may be selected to meet space and time requirements of the user environment, maintaining quality and information levels. When displayed on a screen, the spatial and intensity values may need to be further reduced to match the resolution and hardware characteristics available. The user may perceive this as lower quality and higher information content. Hierarchical Application It is usual for digital image libraries to be configured so as to allow different forms of usage, depending on the needs and sophistication of the user. For example, small thumbnail image versions are used to allow for browsing or refinement of search. Larger image versions such as postcard size are used for inspection of individual images in the library, with the intention of analysis of the contents or decision to obtain a higher grade version by rescanning or restricted access mode or purchase through commercial interaction. Coupled, the set of versions may facilitate more complex operations such as content based image retrieval, or query by example. Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 4 Consequently it may be necessary to apply the same design criteria successively for several different performance regimes in a digital image library, in order to determine the best choices of values to use for each of the usage situations. These values can lead to a nested sequence of implementation version, allowing easy generalization or specialization as the hierarchy is traversed. Depending on the limitations of hardware and software in the library environment, and communications bandwidths, different variants may coexist at each level in the hierarchy to produce similar but specifically tuned performance depending on these circumstances. Examples We consider two common situations for digital image libraries: conventional “general purpose” public interest type images (such as historical scenes, museum artefacts), and high detail “specialized content” expert use type images (such as industrial or architectural scenes, botanical or zoological specimens). The former are typically available on small standard photographic prints (postcard size, e.g. 6x4 inches or 150x100 mm) and the latter as enlargements (e.g. 12x8 inches or 300x200 mm). Recent collaboration with the Morija Museum and Archives in Lesotho has provided examples of each of these two cases. General purpose type images there consist of photographs taken by early settlers, pictures of individuals and families, and some general views of items from the museum anthropological and paleontological collections. Special purpose type images there consist of professional photographs of the landscape and buildings, and detailed close-ups of specific cultural artifacts such as ceremonial clothing. We applied the framework presented above to these cases. Table 1 shows sample values for a general purpose digital image library. As the quality of photographs is generally quite modest, and the image content is quite simple in structure, the related quality and information factors suggest that spatial and intensity choices can be fairly coarse for the digitization without compromising the available detail. Different kinds of processing operations would be needed to improve some images and as a result the scanned resolution would be further reduced. Storage would be compressed at a high quality lossy rate, nearly “visually lossless”. Display would be as an image on only part of the screen, as the software would need to allow active search and browse facilities with side text annotations for each image. Printed versions of the images would be similar to originals, and no more than postcard size. Table 2 shows sample values for a special purpose digital image library. The photographs are of high quality and available as enlargements of variously 5x7 inches and 8x10 inches. This dictates that a high quality initial digitization is appropriate, to ensure as much of the inherent detail can be captured. Little processing is needed once the images are digitized because of the high quality and controlled nature of the photographs. Storage would use a lossless compression method which enabled the full size digitized image data to be available always. These images are accessed individually and are intended to be displayed on full screen or printed at enlargement size, so the display values also need to be aligned with the stored image resolution. Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 5 Table 3 shows a hierarchical extension of the example considered in Table 2. We have added capabilities for wide range search, requiring multiple thumbnail images to be presented on the screen simultaneously, and the same type of partial screen display for viewing with search and annotation ability. The choices shown here would be most likely implemented as three separate versions of each image for ease and speed of access, rather than resizing on the fly. However it is quite conceivable that the implementation could be based around only the special purpose image criteria, if efficient resizing software was available. Conclusion This work has concentrated on describing design considerations for digital image libraries based on photographs. The wide variety of image acquisition devices and settings used for digitizing such images can result in very different intrinsic image characteristics for choices of these design criteria. Variations in user performance (and satisfaction) will be experienced if their visual attention patterns are affected by these differences. The framework provided here allows these impacts to be considered when constructing the library. Different modes of access can also be catered for within the framework by catering for a hierarchy of values of the design criteria. References Frey F. and Susstrunk S. 1996. Image quality requirements for the digitization of photographic collections. SPIE Proceedings, 2663:2-7. Halonen R., Westman S. and Oittinen P. 2009. Naturalness and interestingness of test images for visual quality evaluation. SPIE Proceedings, 7867:1-12. Maeder A.J. 2005. The Image Importance approach to human vision-based image quality characterization. Pattern recognition letters, 26:347-354. Parkhurst, D., Law, K. and Niebur, E. 2002. Modeling the role of salience in the allocation of overt visual attention. Vision Res. 42(1): 107-123. Wang Z., Bovik A.C., Sheikh H.R. and Simoncelli E.P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc., 13(4):1-14. Table 1: Design criteria stages and factors for typical general purpose usage (based on 6x4 inch monochrome originals) Input Stage Process Stage Storage Stage Display Stage Spatial Factor 200-300 dpi (1200x800 – 1800x1200 pixels) 2:1 downsize (600x400 – 900x600 pixels) 10:1 lossy (24K-36K) 640 x 480 pixels Intensity Factor 8-bit greyscale 8-bit greyscale warp 8-bit lookup 8-bit lookup Quality Factor Mid Mid Low Low Inform Factor Low Low Mid Mid Table 2: Design criteria stages and factors for typical high detail usage (based on 12x8 inch colour originals) Input Stage Process Stage Storage Stage Display Stage Spatial Factor 300-600 dpi (3600x2400 - 7200x4800 pixels) 1:1 fullsize (3600x2400 – 7200x4800 pixels) 4:1 lossless (4M-16M) 3Kx2K pixels Intensity Factor 3x16-bit RGB 3x8-bit RGB warp 16-bit lookup 8-bit lookup Quality Factor High High High High Inform Factor Mid Mid High High Second International Conference on African Digital Libraries and Archives (ICADLA-2) Maeder - 6 Table 3: Hierarchical application of design criteria to provide 3 levels of usage, for the example of Table 2. Input Process Storage (Thumb) Storage (View) Storage (Repro) Display (Thumb) Display (View) Display (Repro) Spatial 300-600 dpi 1:1 fullsize 100:1 lossy 10:1 lossy 4:1 lossless 60x40 pixels 640x480 pixels 3Kx2K pixels Intensity 3 x 16-bit RGB 3 x 8-bit RGB 8-bit RGB 8-bit lookup 16-bit lookup 8-bit RGB 8-bit lookup 8-bit lookup Quality High High Low Mid High Low Mid High Inform Mid Mid Mid Mid High Mid High High