Anselm Grundhöfer

We propose a method to generate a display with enhanced color appearance by projecting monochromatic 365 nm ultraviolet (UV) light onto an optimized pattern surface containing red, green, and blue fluorescent pigments. Our method is based on the Paxel framework proposed by Pjanic and Grundhöfer [1] describing the color formulation process when projecting spatially varying illumination onto a printed surface containing pigments with spatially varying properties, in our case fluorescent emission.… Show more

We propose a method to generate a display with enhanced color appearance by projecting monochromatic 365 nm ultraviolet (UV) light onto an optimized pattern surface containing red, green, and blue fluorescent pigments. Our method is based on the Paxel framework proposed by Pjanic and Grundhöfer [1] describing the color formulation process when projecting spatially varying illumination onto a printed surface containing pigments with spatially varying properties, in our case fluorescent emission. In this article, we extend this approach by optimizing the spatial arrangement and surface ratios of the used red, green, and blue transparent fluorescent pigments within the patterns. These ratios are optimized in order to obtain a color gamut with maximized volume. Furthermore, we propose a novel setup where the fluorescent inks are printed onto a transparent film which is then placed onto a black, light-absorbing material. This projection setup, consisting of an emissive and a black absorbing layer, enables us to produce high-contrast imagery even if a high amount of ambient light is present. Finally, to reduce the visually unpleasant appearance of the printed patterns, we introduce an alternative hexagon-shaped pattern structure. Show less

In this paper we propose a novel layered-printing method consisting of superposed visible cmy and invisible fluorescent ultraviolet (UV) rgb inks. Our approach can be used to generate a variety of visual color-alteration effects such as revealing two completely distinct images when the print is illuminated with either standard visible or 365 nm ultraviolet (UV) light (Figure 1). This is achieved by computing the maximum achievable color gamuts for both illumination conditions, generating… Show more

In this paper we propose a novel layered-printing method consisting of superposed visible cmy and invisible fluorescent ultraviolet (UV) rgb inks. Our approach can be used to generate a variety of visual color-alteration effects such as revealing two completely distinct images when the print is illuminated with either standard visible or 365 nm ultraviolet (UV) light (Figure 1). This is achieved by computing the maximum achievable color gamuts for both illumination conditions, generating accurate estimates, and applying a spatial-varying gamut mapping to minimize potential ghosting artifacts and calculate the optimal ink surface coverages that, when printed, generate the desired image-alteration effect. Our method uses invisible UV-rgb fluorescent inks which are printed onto a transparent film. It is placed on top of a visible print consisting of standard cmy inks. By separating the UV and the visible inks using the transparent film, physical mixing of the two different ink types is avoided. This significantly increases the intensity of the fluorescent emission resulting in stronger and more vivid color-alteration effects. Besides the revealing of two different images, the same method can be applied for other use cases as well, such as enhancing or adding specific parts to an image under one illumination condition, generating personalized document security features, or aiding color-blind people in color distinction. Show less

This paper introduces a novel photometric compensation technique for inter-projector luminance and chrominance variations. Although it sounds as a classical technical issue, to the best of our knowledge there is no existing solution to alleviate the spatial non-uniformity among strongly heterogeneous projectors at perceptually acceptable quality. Primary goal of our method is increasing the perceived seamlessness of the projection system by automatically generating an improved and consistent… Show more

This paper introduces a novel photometric compensation technique for inter-projector luminance and chrominance variations. Although it sounds as a classical technical issue, to the best of our knowledge there is no existing solution to alleviate the spatial non-uniformity among strongly heterogeneous projectors at perceptually acceptable quality. Primary goal of our method is increasing the perceived seamlessness of the projection system by automatically generating an improved and consistent visual quality. It builds upon the existing research of multi-projection systems, but instead of working with perceptually non-uniform color spaces such as CIEXYZ, the overall computation is carried out using the RLab [10, pp. 243-254] color appearance model which models the color processing in an adaptive, perceptual manner. Besides, we propose an adaptive color gamut acquisition, spatially varying gamut mapping, and optimization framework for edge blending. The paper describes the overall workflow and detailed algorithm of each component, followed by an evaluation validating the proposed method. The experimental results both qualitatively and quantitatively show the proposed method significant improved the visual quality of projected results of a multi-projection display with projectors with severely heterogeneous color processing. Show less

This State-of-the-Art-Report covers the recent advances in research fields related to projection mapping applications. We summarize the novel enhancements to simplify the 3D geometric calibration task, which can now be reliably carried out either interactively or automatically using self-calibration methods. Furthermore, improvements regarding radiometric calibration and compensation as well as the neutralization of global illumination effects are summarized. We then introduce computational… Show more

This State-of-the-Art-Report covers the recent advances in research fields related to projection mapping applications. We summarize the novel enhancements to simplify the 3D geometric calibration task, which can now be reliably carried out either interactively or automatically using self-calibration methods. Furthermore, improvements regarding radiometric calibration and compensation as well as the neutralization of global illumination effects are summarized. We then introduce computational display approaches to overcome technical limitations of current projection hardware in terms of dynamic range, refresh rate, spatial resolution, depth-of-field, view dependency, and color space. These technologies contribute towards creating new application domains related to projection-based spatial augmentations. We summarize these emerging applications, and discuss new directions for industries Show less

In this paper, we propose Paxel, a generic framework for modeling the interaction between a projector and a high-frequency pattern surface. Using this framework, we present two different application setups: a novel colorchanging effect, created with a single projected image and only when the projection surface is changed from a pattern surface to a uniform white surface. The observed effect relies on the spatially different reflectance properties of these two surfaces. Using this approach, one… Show more

In this paper, we propose Paxel, a generic framework for modeling the interaction between a projector and a high-frequency pattern surface. Using this framework, we present two different application setups: a novel colorchanging effect, created with a single projected image and only when the projection surface is changed from a pattern surface to a uniform white surface. The observed effect relies on the spatially different reflectance properties of these two surfaces. Using this approach, one can alter color proprieties of the projected image such as hue or chroma. Furthermore, for a specific color range, defined by an full color-changing sub-gamut, one can embed two completely different images, within a single static projection, from which either one will be revealed depending on the surface. The second application allows the creation of color images using a single channel projector. For this application, we present a full color projection created using a 365 nm ultraviolet (UV) projector in combination with fluorescent pigments (cf. Fig. 1b), enabling new display possibilities, such as projection through participating media, e.g. fog, while hiding the scattering of the projection light outside of the visible spectrum. Both presented approaches create effects that might be striking to the observer, making this framework useful for art exhibitions, advertisements, entertainment and visual cryptography. Finally, in Sec. VI, we provide an in-depth analysis of the reproducible colors based on input parameters, used in the presented algorithm, such as: pattern layout, dot size of the pattern and the number of the clusters formed by k-means algorithm. Show less

We propose a simple method to enhance the image quality of modern Bayer pattern based cameras that are offering an optional sub-pixel accurate sensor shift to capture full rgb images of static scenes. By capturing a series of image pairs of the same, unaltered scene, once captured with the Bayer pattern and once with the full rgb image data, a database of corresponding images can be generated in which the former contains artifacts resulting from the spatial interpolation which is absent in the… Show more

We propose a simple method to enhance the image quality of modern Bayer pattern based cameras that are offering an optional sub-pixel accurate sensor shift to capture full rgb images of static scenes. By capturing a series of image pairs of the same, unaltered scene, once captured with the Bayer pattern and once with the full rgb image data, a database of corresponding images can be generated in which the former contains artifacts resulting from the spatial interpolation which is absent in the latter. Using this data, we train a convolutional neural network (CNN) to generate a camera dependent image processing operation which reduces the image artifacts and enhances the image quality to approximate the quality of the full rgb image within a single exposure, even if moving scenes are captured. We present a simple, do-it-yourself method to capture and pre-process the data, train the network, and to enhance the images. An evaluation using several image quality assessment methods shows the effectiveness of the proposed method.
Show less

In this paper, we propose a fully automated selfcalibration method for arbitrary complex multi-projectorcamera systems (MPCS). Combining the advantages of known, sub-pixel accurate correspondences using robust projected structured light patterns with state-of-the-art camera self-calibration algorithms enables a reliable and accurate intrinsic and extrinsic calibration without any required human parameter tuning. We evaluated the proposed methods using more than ten multi-projection datasets… Show more

In this paper, we propose a fully automated selfcalibration method for arbitrary complex multi-projectorcamera systems (MPCS). Combining the advantages of known, sub-pixel accurate correspondences using robust projected structured light patterns with state-of-the-art camera self-calibration algorithms enables a reliable and accurate intrinsic and extrinsic calibration without any required human parameter tuning. We evaluated the proposed methods using more than ten multi-projection datasets ranging from a toy castle set up consisting of three cameras and one projector up to a half dome display system with more than 30 devices. Comparisons to reference calibrations, which were generated using the standard checkerboard calibration approach, show the reliability of our proposed pipeline. Besides being fully automatic without the necessity of parameter fine tuning, the proposed method also significantly reduces the installation time of MPCS compared to checkerboard-based methods and makes it more suitable for real-world applications. Show less

We present a geometric calibration method to accurately register a galvanoscopic scanning laser projection system (GLP) based on 2D vector input data onto an arbitrarily complex 3D-shaped projection surface. This method allows for accurate merging of 3D vertex data displayed on the laser projector with geometrically calibrated standard rasterization-based video projectors that are registered to the same geometry. Because laser projectors send out a laser light beam via galvanoscopic mirrors, a… Show more

We present a geometric calibration method to accurately register a galvanoscopic scanning laser projection system (GLP) based on 2D vector input data onto an arbitrarily complex 3D-shaped projection surface. This method allows for accurate merging of 3D vertex data displayed on the laser projector with geometrically calibrated standard rasterization-based video projectors that are registered to the same geometry. Because laser projectors send out a laser light beam via galvanoscopic mirrors, a standard pinhole model calibration procedure that is normally used for pixel raster displays projecting structured light patterns, such as Gray codes, cannot be carried out directly with sufficient accuracy as the rays do not converge into a single point. To overcome the complications of accurately registering the GLP while still enabling a treatment equivalent to a standard pinhole device, an adapted version is applied to enable straightforward content generation. Besides the geometrical calibration, we also present a photometric calibration to unify the color appearance of GLPs and standard video projectors maximizing the advantages of the large color gamut of the GLP and optimizing its color appearance to smoothly fade into the significantly smaller gamut of the video projector. The proposed algorithms were evaluated on a prototypical mixed video projector and GLP projection mapping setup. Show less

Calibrating the intrinsic properties of a camera is one of the fundamental tasks required for a variety of computer vision and image processing tasks. The precise measurement of focal length, location of the principal point as well as distortion parameters of the lens is crucial, for example, for 3D reconstruction. Although a variety of methods exist to achieve this goal, they are often cumbersome to carry out, require substantial manual interaction, expert knowledge, and a significant… Show more

Calibrating the intrinsic properties of a camera is one of the fundamental tasks required for a variety of computer vision and image processing tasks. The precise measurement of focal length, location of the principal point as well as distortion parameters of the lens is crucial, for example, for 3D reconstruction. Although a variety of methods exist to achieve this goal, they are often cumbersome to carry out, require substantial manual interaction, expert knowledge, and a significant operating volume. We propose a novel calibration method based on the usage of directionally encoded light rays for estimating the intrinsic parameters. It enables a fully automatic calibration with a small device mounted close to the front lens element and still enables an accuracy comparable to standard methods even when the lens is focused up to infinity. Our method overcomes the mentioned limitations since it guarantees an accurate calibration without any human intervention while requiring only a limited amount of space. Besides that, the approach also allows to estimate the distance of the focal plane as well as the size of the aperture. We demonstrate the advantages of the proposed method by evaluating several camera/lens configurations using prototypical devices. Show less

We propose the first system for live dynamic augmentation of human faces. Using projector-based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection… Show more

We propose the first system for live dynamic augmentation of human faces. Using projector-based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high-speed camera detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non-rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions.

Video:

https://youtu.be/Ilgu3aFCphs

Show less

Galvanoscopic scanning laser projectors are powerful vector graphic devices offering a tremendous local brightness advantage compared to standard video projection systems. However, such devices have inherent problems, such as temporal flicker and spatially inaccurate rendering. We propose a method to generate an accurate point-based projection with such devices. To overcome the mentioned problems, we present a camera-based method to automatically analyze the laser projector's motion behavior… Show more

Galvanoscopic scanning laser projectors are powerful vector graphic devices offering a tremendous local brightness advantage compared to standard video projection systems. However, such devices have inherent problems, such as temporal flicker and spatially inaccurate rendering. We propose a method to generate an accurate point-based projection with such devices. To overcome the mentioned problems, we present a camera-based method to automatically analyze the laser projector's motion behavior. With this information, a model database is generated that is used to optimize the scanning path of projected point sequences. The optimization considers the overall path length, its angular shape, acceleration behavior, and the spatio-temporal point neighborhood. The method minimizes perceived visual flickering while guaranteeing an accurate spatial point projection at the same time. Comparisons and timing measurements prove the effectiveness of our method. An informal user evaluation shows substantial visual quality improvement as well. Show less

Achieving a minimal latency within augmented reality (AR) systems is one of the most important factors to achieve a convincing visual impression. It is even more crucial for non-video augmentations such as dynamic projection mappings because in that case the superimposed imagery has to exactly match the dynamic real surface, which obviously cannot be directly influenced or delayed in its movement. In those cases, the inevitable latency is usually compensated for using prediction and… Show more

Achieving a minimal latency within augmented reality (AR) systems is one of the most important factors to achieve a convincing visual impression. It is even more crucial for non-video augmentations such as dynamic projection mappings because in that case the superimposed imagery has to exactly match the dynamic real surface, which obviously cannot be directly influenced or delayed in its movement. In those cases, the inevitable latency is usually compensated for using prediction and extrapolation operations, which require accurate information about the occurring overall latency to exactly predict to the right time frame for the augmentation. Different strategies have been applied to accurately compute this latency. Since some of these AR systems operate within different spectral bands for input and output, it is not possible to apply latency measurement methods encoding time stamps directly into the presented output images as these might not be sensed by used input device.We present a generic latency measurement device which can be used to accurately measure the overall end-to-end latency of camera-based AR systems with an accuracy below one millisecond. It comprises a LED-based time stamp generator displaying the time as a gray code on spatially and spectrally multiple locations. It is controlled by a micro-controller and sensed by an external camera device observing the output display as well as the LED device at the same time. Show less

We propose a method to fabricate textured 3D models using thermoforming. Differently from industrial techniques, which target mass production of a specific shape, we propose a combined hardware and software solution to manufacture customized, unique objects. Our method simulates the forming process and converts the texture of a given digital 3D model into a pre-distorted image that we transfer onto a plastic sheet. During thermoforming, the sheet deforms to create a faithful physical replica of… Show more

We propose a method to fabricate textured 3D models using thermoforming. Differently from industrial techniques, which target mass production of a specific shape, we propose a combined hardware and software solution to manufacture customized, unique objects. Our method simulates the forming process and converts the texture of a given digital 3D model into a pre-distorted image that we transfer onto a plastic sheet. During thermoforming, the sheet deforms to create a faithful physical replica of the digital model. Our hardware setup uses off-the-shelf components and can be calibrated with an automatic algorithm that extracts the simulation parameters from a single calibration object produced by the same process. Show less

We propose a novel error tolerant optimization approach to generate a high-quality photometric compensated projection. The application of a non-linear color mapping function does not require radiometric pre-calibration of cameras or projectors. This characteristic improves the compensation quality compared with related linear methods if this approach is used with devices that apply complex color processing, such as single-chip digital light processing projectors. Our approach consists of a… Show more

We propose a novel error tolerant optimization approach to generate a high-quality photometric compensated projection. The application of a non-linear color mapping function does not require radiometric pre-calibration of cameras or projectors. This characteristic improves the compensation quality compared with related linear methods if this approach is used with devices that apply complex color processing, such as single-chip digital light processing projectors. Our approach consists of a sparse sampling of the projector's color gamut and non-linear scattered data interpolation to generate the per-pixel mapping from the projector to camera colors in real time. To avoid out-of-gamut artifacts, the input image's luminance is automatically adjusted locally in an optional offline optimization step that maximizes the achievable contrast while preserving smooth input gradients without significant clipping errors. To minimize the appearance of color artifacts at high-frequency reflectance changes of the surface due to usually unavoidable slight projector vibrations and movement (drift), we show that a drift measurement and analysis step, when combined with per-pixel compensation image optimization, significantly decreases the visibility of such artifacts. Show less

High dynamic range (HDR) display prototypes have been built and used for scientific studies for nearly a decade, and they are now on the verge of entering consumer market. However, problems exist regarding the accurate color reproduction capabilities on these displays. In this paper, we first characterize the image reproduction capability of a state-of-art HDR display through a set of measurements, and present a novel calibration method that takes into account the variation of the chrominance… Show more

High dynamic range (HDR) display prototypes have been built and used for scientific studies for nearly a decade, and they are now on the verge of entering consumer market. However, problems exist regarding the accurate color reproduction capabilities on these displays. In this paper, we first characterize the image reproduction capability of a state-of-art HDR display through a set of measurements, and present a novel calibration method that takes into account the variation of the chrominance error over HDR display’s wide luminance range. Our proposed 3D octree forest data structure for representing and querying the calibration function successfully addresses the challenges in calibrating HDR displays: (i) high computational complexity due to nonlinear chromatic distortions; (ii) huge storage space demand for a look-up table. We show that our method achieves high color reproduction accuracy through both objective metrics and a controlled subjective study Show less

The last decade has witnessed tremendous improvements in display technology. Today's smartphones and laptops offer screen resolutions far beyond what the human eye is able to resolve. TV systems are also in the process of changing to resolutions far beyond "full HD", and cinemas have almost completely transitioned from analog film to digital projection. The high-quality projectors of today's cinema are also well suited for non-standard setups such as, for example, in theme park environments… Show more

The last decade has witnessed tremendous improvements in display technology. Today's smartphones and laptops offer screen resolutions far beyond what the human eye is able to resolve. TV systems are also in the process of changing to resolutions far beyond "full HD", and cinemas have almost completely transitioned from analog film to digital projection. The high-quality projectors of today's cinema are also well suited for non-standard setups such as, for example, in theme park environments. Generating a scenic lighting environment is one of the most important factors in the overall immersion quality of such attractions, and the usage of digital projectors can help to raise the bar in this regard. It is to be expected that more and more projection systems will be used in the future not only in theme parks, but also for general entertainment, advertisements, and art installations. While the tools have already been developed to enable sufficiently accurate geometric and photometric calibration for content generation of such installations, the question still remains as to whether non-experts will be able to successfully apply and judge calibration, as well as know how to maintain such a complex system without specific knowledge as to its underlying algorithms. Show less

In this project, we propose a complete process for augmenting physical avatars using projector based illumination, significantly increasing their expressiveness. Given an input animation, the system decomposes the motion into low-frequency motion that can be physically reproduced by the animatronic head and high-frequency details that are added using projected shading. At the core is a spatio-temporal optimization process that compresses the motion in gradient space, ensuring faithful motion… Show more

In this project, we propose a complete process for augmenting physical avatars using projector based illumination, significantly increasing their expressiveness. Given an input animation, the system decomposes the motion into low-frequency motion that can be physically reproduced by the animatronic head and high-frequency details that are added using projected shading. At the core is a spatio-temporal optimization process that compresses the motion in gradient space, ensuring faithful motion replay while respecting the physical limitations of the system. We also propose a complete multi-camera and projection system, including a novel defocused projection and subsurface scattering compensation scheme. The result of our system is a highly expressive physical avatar that features facial details and motion otherwise unreachable due to physical constraints.

Video:

https://www.youtube.com/watch?v=O1qV6glZkOQ&feature=youtu.be

Show less

We propose a novel approach to generate a high quality photometric compensated projection which, to our knowledge, is the first one, which does not require a radiometrical pre-calibration of cameras or projectors. This improves the compensation quality using devices which cannot be easily linearized, such as single chip DLP projectors with complex color processing. In addition, the simple workflow significantly simplifies the compensation image generation. Our approach consists of a sparse… Show more

We propose a novel approach to generate a high quality photometric compensated projection which, to our knowledge, is the first one, which does not require a radiometrical pre-calibration of cameras or projectors. This improves the compensation quality using devices which cannot be easily linearized, such as single chip DLP projectors with complex color processing. In addition, the simple workflow significantly simplifies the compensation image generation. Our approach consists of a sparse sampling of the projector's color gamut and a scattered data interpolation to generate the per-pixel mapping from projector to camera colors in real-time. To avoid out-of-gamut artifacts, the input image is automatically scaled locally in an optional off-line optimization step maximizing the achievable luminance and contrast while still preserving smooth input gradients without significant clipping errors Show less

"Walt Disney Imagineering and Disney Research Zürich are building a projector- camera toolbox to help create spatially augmented 3D objects and dynamic, interactive spaces that enhance the theme park experience by immersing guests in magical worlds."

We show how the intrinsically performed JPEG compression of many digital still cameras leaves margin for deriving and applying image-adapted coded apertures that support retention of the most important frequencies after compression. These coded apertures, together with subsequently applied image processing, enable a higher light throughput than corresponding circular apertures, while preserving adjusted focus, depth of field, and bokeh. Higher light throughput leads to proportionally higher… Show more

We show how the intrinsically performed JPEG compression of many digital still cameras leaves margin for deriving and applying image-adapted coded apertures that support retention of the most important frequencies after compression. These coded apertures, together with subsequently applied image processing, enable a higher light throughput than corresponding circular apertures, while preserving adjusted focus, depth of field, and bokeh. Higher light throughput leads to proportionally higher signal-to-noise ratios and reduced compression noise, or –alternatively– to lower shutter times. We explain how adaptive coded apertures can be computed quickly, how they can be applied in lenses by using binary spatial light modulators, and how a resulting coded bokeh can be transformed into a common radial one. Show less

We present a new video mode for television sets that we refer to as display pixel caching (DPC). It fills empty borders with spatially and temporally consistent information while preserving the original video format. Unlike related video modes, such as stretching, zooming, and video retargeting, DPC does not scale or stretch individual frames. Instead, it merges the motion information from many subsequent frames to generate screen-filling panoramas in a consistent manner. In contrast to… Show more

We present a new video mode for television sets that we refer to as display pixel caching (DPC). It fills empty borders with spatially and temporally consistent information while preserving the original video format. Unlike related video modes, such as stretching, zooming, and video retargeting, DPC does not scale or stretch individual frames. Instead, it merges the motion information from many subsequent frames to generate screen-filling panoramas in a consistent manner. In contrast to state-of-the-art video mosaicing, DPC achieves real-time rates for high-resolution video content while processing more complex motion patterns fully automatically. We compare DPC to related video modes in the context of a user evaluation. Show less

In this paper, we show that optical inverse tone-mapping (OITM) in light microscopy can improve the visibility of specimens, both when observed directly through the oculars and when imaged with a camera. In contrast to previous microscopy techniques, we premodulate the illumination based on the local modulation properties of the specimen itself. We explain how the modulation of uniform white light by a specimen can be estimated in real time, even though the specimen is continuously but not… Show more

In this paper, we show that optical inverse tone-mapping (OITM) in light microscopy can improve the visibility of specimens, both when observed directly through the oculars and when imaged with a camera. In contrast to previous microscopy techniques, we premodulate the illumination based on the local modulation properties of the specimen itself. We explain how the modulation of uniform white light by a specimen can be estimated in real time, even though the specimen is continuously but not uniformly illuminated. This information is processed and back-projected constantly, allowing the illumination to be adjusted on the fly if the specimen is moved or the focus or magnification of the microscope is changed. The contrast of the specimen's optical image can be enhanced, and high-intensity highlights can be suppressed. A formal pilot study with users indicates that this optimizes the visibility of spatial structures when observed through the oculars. We also demonstrate that the signal-to-noise (S/N) ratio in digital images of the specimen is higher if captured under an optimized rather than a uniform illumination. In contrast to advanced scanning techniques that maximize the S/N ratio using multiple measurements, our approach is fast because it requires only two images. This can improve image analysis in digital microscopy applications with real-time capturing requirements. Show less

In this article we show how temporal backdrops that alternately change their color rapidly at recording rate can aid chroma keying by transforming color spill into a neutral background illumination. Since the chosen colors sum up to white, the chromatic (color) spill component is neutralized when integrating over both backdrop states. The ability to separate both states additionally allows to compute high-quality alpha mattes. Besides the neutralization of color spill, our method is invariant… Show more

In this article we show how temporal backdrops that alternately change their color rapidly at recording rate can aid chroma keying by transforming color spill into a neutral background illumination. Since the chosen colors sum up to white, the chromatic (color) spill component is neutralized when integrating over both backdrop states. The ability to separate both states additionally allows to compute high-quality alpha mattes. Besides the neutralization of color spill, our method is invariant to foreground colors and supports applications with real-time demands. In this article, we explain different realizations of temporal backdrops and describe how keying and color spill neutralization are carried out, how artifacts resulting from rapid motion can be reduced, and how our approach can be implemented to be compatible with common real-time post-production pipelines. Show less

Coding a projector's aperture plane with adaptive patterns together with inverse filtering allow the depth-of-field of projected imagery to be increased. We present two prototypes and corresponding algorithms for static and programmable apertures. We also explain how these patterns can be computed at interactive rates, by taking into account the image content and limitations of the human visual system. Applications such as projector defocus compensation, high-quality projector depixelation, and… Show more

Coding a projector's aperture plane with adaptive patterns together with inverse filtering allow the depth-of-field of projected imagery to be increased. We present two prototypes and corresponding algorithms for static and programmable apertures. We also explain how these patterns can be computed at interactive rates, by taking into account the image content and limitations of the human visual system. Applications such as projector defocus compensation, high-quality projector depixelation, and increased temporal contrast of projected video sequences can be supported. Coded apertures are a step towards next-generation auto-iris projector lenses. Show less

CAMShift is a well-established and fundamental algorithm for kernel-based visual object tracking. While it performs well with objects that have a simple and constant appearance, it is not robust in more complex cases. As it solely relies on back projected probabilities it can fail in cases when the object's appearance changes (e.g., due to object or camera movement, or due to lighting changes), when similarly colored objects have to be re-detected or when they cross their trajectories. We… Show more

CAMShift is a well-established and fundamental algorithm for kernel-based visual object tracking. While it performs well with objects that have a simple and constant appearance, it is not robust in more complex cases. As it solely relies on back projected probabilities it can fail in cases when the object's appearance changes (e.g., due to object or camera movement, or due to lighting changes), when similarly colored objects have to be re-detected or when they cross their trajectories. We propose low-cost extensions to CAMShift that address and resolve all of these problems. They allow the accumulation of multiple histograms to model more complex object appearances and the continuous monitoring of object identities to handle ambiguous cases of partial or full occlusion. Most steps of our method are carried out on the GPU for achieving real-time tracking of multiple targets simultaneously. We explain efficient GPU implementations of histogram generation, probability back projection, computation of image moments, and histogram intersection. All of these techniques make full use of a GPU's high parallelization capabilities. Show less

We synchronize film cameras and LED lighting with off-the-shelf video projectors. Radiometric compensation allows displaying keying patterns and other spatial codes on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables professional keying, environment matting, displaying moderator information, scene reconstruction, and camera tracking for non-studio film sets without being limited to the constraints of a virtual studio. This makes… Show more

We synchronize film cameras and LED lighting with off-the-shelf video projectors. Radiometric compensation allows displaying keying patterns and other spatial codes on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables professional keying, environment matting, displaying moderator information, scene reconstruction, and camera tracking for non-studio film sets without being limited to the constraints of a virtual studio. This makes digital video composition more flexible, since static studio equipment, such as blue screens, teleprompters, or tracking devices, is not required. Authentic film locations can be supported with our portable system without causing a lot of installation effort. Show less

Recent radiometric compensation techniques make it possible to project images onto colored and textured surfaces. This is realized with projector-camera systems by scanning the projection surface on a per-pixel basis. Using the captured information, a compensation image is calculated that neutralizes geometric distortions and color blending caused by the underlying surface. As a result, the brightness and the contrast of the input image is reduced compared to a conventional projection onto a… Show more

Recent radiometric compensation techniques make it possible to project images onto colored and textured surfaces. This is realized with projector-camera systems by scanning the projection surface on a per-pixel basis. Using the captured information, a compensation image is calculated that neutralizes geometric distortions and color blending caused by the underlying surface. As a result, the brightness and the contrast of the input image is reduced compared to a conventional projection onto a white canvas. If the input image is not manipulated in its intensities, the compensation image can contain values that are outside the dynamic range of the projector. These will lead to clipping errors and to visible artifacts on the surface. In this article, we present an innovative algorithm that dynamically adjusts the content of the input images before radiometric compensation is carried out. This reduces the perceived visual artifacts while simultaneously preserving a maximum of luminance and contrast. The algorithm is implemented entirely on the GPU and is the first of its kind to run in real time. Show less

At present, more than half of all building activity in the German building sector is undertaken within existing built contexts. The development of a conceptual and technological basis for the digital support of design directly on site, within an existing building context is the focus of the research project "Spatial Augmented Reality for Architecture" (SAR). This paper describes the goals achieved in one aspect of the project: the sampling of colors and materials at a scale of 1:1 using… Show more

At present, more than half of all building activity in the German building sector is undertaken within existing built contexts. The development of a conceptual and technological basis for the digital support of design directly on site, within an existing building context is the focus of the research project "Spatial Augmented Reality for Architecture" (SAR). This paper describes the goals achieved in one aspect of the project: the sampling of colors and materials at a scale of 1:1 using Augmented Reality (AR) technologies. We present initial results from the project; the development of an ad-hoc visualization of interactive data on arbitrary surfaces in real-world indoor environments using a mobile hardware setup. With this, it was possible to project the color and material qualities of a design directly onto almost all surfaces within a geometrically corrected, existing building. Initially, a software prototype "Spatial Augmented Reality for Architecture-Colored Architecture" (SAR-CA) was developed and then assessed based on evaluation results from a user study. Show less

This article focuses on real-time image correction techniques that enable projector-camera systems to display images onto screens that are not optimized for projections, such as geometrically complex, coloured and textured surfaces. It reviews hardware-accelerated methods like pixel-precise geometric warping, radiometric compensation, multi-focal projection and the correction of general light modulation effects. Online and offline calibration as well as invisible coding methods are explained… Show more

This article focuses on real-time image correction techniques that enable projector-camera systems to display images onto screens that are not optimized for projections, such as geometrically complex, coloured and textured surfaces. It reviews hardware-accelerated methods like pixel-precise geometric warping, radiometric compensation, multi-focal projection and the correction of general light modulation effects. Online and offline calibration as well as invisible coding methods are explained. Novel attempts in super-resolution, high-dynamic range and high-speed projection are discussed. These techniques open a variety of new applications for projection displays. Some of them will also be presented in this report. Show less

In this paper we present an innovative adaptive imperceptible pattern projection technique that takes into account parameters of human visual perception. A coded image is temporally integrated into the projected image, which is invisible to the human eye but can be reconstructed by a synchronized camera. The embedded code is dynamically adjusted on the fly to guarantee its imperceptibility and to adapt it to the current camera pose. Linked with real-time flash keying, for instance, this enables… Show more

In this paper we present an innovative adaptive imperceptible pattern projection technique that takes into account parameters of human visual perception. A coded image is temporally integrated into the projected image, which is invisible to the human eye but can be reconstructed by a synchronized camera. The embedded code is dynamically adjusted on the fly to guarantee its imperceptibility and to adapt it to the current camera pose. Linked with real-time flash keying, for instance, this enables in-shot optical tracking using a dynamic multi-resolution marker technique. A sample prototype has been realized that demonstrates the application of our method in the context of augmentations in television studios. Show less

We present a real-time reverse radiosity method for compensating indirect scattering effects that occur with immersive and semi-immersive projection displays. It computes a numerical solution directly on the GPU and is implemented with pixel shading and multi-pass rendering which together realizes a Jacobi solver for sparse matrix linear equation systems. Our method is validated and evaluated based on a stereoscopic two-sided wall display. The images appear more brilliant and uniform when… Show more

We present a real-time reverse radiosity method for compensating indirect scattering effects that occur with immersive and semi-immersive projection displays. It computes a numerical solution directly on the GPU and is implemented with pixel shading and multi-pass rendering which together realizes a Jacobi solver for sparse matrix linear equation systems. Our method is validated and evaluated based on a stereoscopic two-sided wall display. The images appear more brilliant and uniform when compensating the scattering contribution. Show less

Holography and computer graphics are being used as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. Our intention is to combine both technologies to create a powerful tool for science, industry and education. We are currently investigating the possibility of integrating computer generated graphics and holograms. This paper gives an overview over our latest results. It presents… Show more

Holography and computer graphics are being used as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. Our intention is to combine both technologies to create a powerful tool for science, industry and education. We are currently investigating the possibility of integrating computer generated graphics and holograms. This paper gives an overview over our latest results. It presents several applications of interaction techniques to graphically enhanced holograms and gives a first glance on a novel method that reconstructs depth from optical holograms. Show less

This paper presents a non-conservative occlusion culling technique for dynamic scenes with animated or user-manipulated objects. We use a multi-pass algorithm, which decides the visibility based on low level of detail representations of the geometric models. Our approach makes efficient use of hardware support for occlusion queries and avoids stalling the graphics pipeline. We have tested our approach for large real-world models from different areas. Our results show that the algorithm performs… Show more

This paper presents a non-conservative occlusion culling technique for dynamic scenes with animated or user-manipulated objects. We use a multi-pass algorithm, which decides the visibility based on low level of detail representations of the geometric models. Our approach makes efficient use of hardware support for occlusion queries and avoids stalling the graphics pipeline. We have tested our approach for large real-world models from different areas. Our results show that the algorithm performs well for medium complex scenes with 5 to 20 million triangles, with a very low number of hardly noticeable pixel errors, typically in the range of 0.02 percent of the total number of visible pixels. Show less

Modern graphics cards support occlusion culling in hardware. We present a three pass algorithm, which makes efficient use of this feature. Our geo-scientific sub-surface data sets consist typically of a set of high resolution height fields, polygonal objects, and volume slices and lenses. For each height field, we compute a low and high resolution version in a pre-process and divide both into sets of corresponding tiles. For each tile and for the polygonal objects, the first rendering pass… Show more

Modern graphics cards support occlusion culling in hardware. We present a three pass algorithm, which makes efficient use of this feature. Our geo-scientific sub-surface data sets consist typically of a set of high resolution height fields, polygonal objects, and volume slices and lenses. For each height field, we compute a low and high resolution version in a pre-process and divide both into sets of corresponding tiles. For each tile and for the polygonal objects, the first rendering pass computes a z-buffer image using the low resolution tiles, the polygonal objects and the non-transparent volume objects. During the second pass, we render the same objects against the z-buffer of the first pass while submitting an occlusion query with each object. The third pass reads this occlusion information back from the graphics hardware and renders only those high resolution objects, for which the corresponding low resolution objects were not completely occluded. To avoid fill rate bottle necks, the first two passes may be rendered to a low resolution window. Our implementation shows frame rate improvements for all test cases while introducing only a small overhead and no or hardly noticeable errors. Our non-conservative approach does not require front to back sorting and it works for dynamic scenes. Show less

We present techniques which create a consistent illumination between real and virtual objects inside an application specific optical see-through display: the Virtual Showcase. We use projectors and cameras to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. Matching direct and indirect lighting effects,such as shading, shadows, reflections and color bleedingcan be approximated at interactive rates in such a controlled… Show more

We present techniques which create a consistent illumination between real and virtual objects inside an application specific optical see-through display: the Virtual Showcase. We use projectors and cameras to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. Matching direct and indirect lighting effects,such as shading, shadows, reflections and color bleedingcan be approximated at interactive rates in such a controlled mixed environment. Show less