This lecture was presented at the 3D Digital Documentation Summit held July 10-12, 2012 at the Presidio, San Francisco, CA

Advances in Computational Photograph Techniques for Cultural, Historic, and Natural History Materials

This talk presents advances in robust new imaging tools from the emerging science known as Computational Photography. The common feature of the computational photography imaging family is the purpose-driven, selective extraction of information from sequences of digital photographs. The information is extracted from the photographic sequences by computer algorithms. The extracted information is then integrated into new digital representations containing information not present in the original photographs, examined either alone or sequentially.

We will describe robust photography-based digital techniques for use with a wide range of cultural and natural history materials and associated research. We will show the use of these tools in a range of contexts including fine art conservation, research of museum and library collections, and documentation of rock art.

Examples of existing and cutting-edge uses of photography-based imaging will be presented, including Reflectance Transformation Imaging (RTI), Algorithmic Rendering (AR), camera calibration, and methods of image-based generation of textured 3D geometry.

The most mature and widely adopted technique for cultural heritage research is Reflectance Transformation Imaging. RTI creates digital representations from image sequences where light is projected from different directions. The lighting information from this image set is mathematically synthesized into an RTI image, enabling a user to interactively re-light and enhance the subject’s surface in incredible detail.  An Institute of Museum and Library Services (IMLS) sponsored training program is bringing a four day RTI training to all six masters programs in art conservation in North America, as well as four regional museum trainings open to museum professionals.  As a result of this program over 150 museum professionals and pre-professionals will be fully trained in RTI technology, in addition to the many institutions that are adopting RTI outside of this program.

This talk will present the latest developments in RTI. We will examine multi-spectral RTI and the hidden topological landscapes disclosing under-painting and drawings in the infra-red and the fine surface information disclosed in ultra-violet wavelengths.  We will discuss RTI of subjects under magnification using macro and microscopic optics as well as updates in viewing technology.

New developments in the related technology Algorithmic Rendering (AR), which uses the same data sets as RTI, will also be presented. The development of new AR technology by Princeton University and Cultural Heritage Imaging is supported by a significant grant from the National Science Foundation.  The end-product will be an open source tool which will extract and merge visual information available only under certain lighting conditions, certain wavelengths, or certain imaging modalities. Cultural heritage professionals will be able to generate high quality, comprehensible illustrations for documentation, scientific study, and sharing with colleagues, collection visitors, and the public.

New software tools to better collect and manage the metadata surrounding the creation of RTI and AR will also be discussed.  This “digital lab notebook” is a critical element in the generation of scientifically reliable digital representations that enable future reuse for novel purposes, assist the long-term digital preservation of the virtual representations, and aid the physical conservation of the digitally represented museum materials.

Computational photography is a rapidly expanding field generating new tools and methods that can aide conservators in the documentation, study, and widespread understanding of the art works under their care.



Church:    From the new College of Florida, he has worked as a professional bronze sculptor and has been involved in 3D imaging for over twenty years. He’s also the co-inventor with Tom Malzbender of the Computational Computer Technique High Reflectance Transformation Imaging. Mark serves on several international committees including the International Council Museums Documentation Committee. I would now like to now turn it over to Carla.

Schroer:    I’d like to start by just taking a moment and telling you a little bit about Cultural Heritage Imaging, and we are based here in San Francisco and we are a non-profit and we have a mission to drive both the development and adoption of digital imaging solutions. We have this really big broad vision, but I think given the conversation over the last couple of days, I want to point out some things that may make us a little different from a lot of the things we’ve seen, which is that our core philosophy is about really getting tools in the hands of people so they can integrate them in their day to day work and use them so that lots of material can get imaged as opposed to a service provider model where you have to pay specialists to come in and do things. So our approach is to look for things that people can do inexpensively with a minimum amount of training and with off the shelf equipment. We also want to look at making sure the results that are produced are high quality and we want to be thinking about issues like archiving and data reuse over time.


We’re a small organization and we function through collaboration. We have a number of technical groups and research labs that we work with. Here are some of the folks that we’re working with currently, and we also work with a lot of people on the cultural heritage side, including a number of museums and most recently fine art conservation.
So, I’m going to talk a little bit about two primary techniques here but I just want to set the stage for what we mean when we say computational photography and really what we’re talking about is taking a sequence of images with a camera and then in the computer, extracting information from different images in that sequence to create a new type of representation that has information that’s not available in any one image from that sequence. And there are a number of examples of this and there’s a lot of work happening in this field. The one’s we’re going to be talking about here are the reflectance transformation imaging and algorithmic rendering.
So, something we have as a big idea is that we know today, right now, how to with a digital camera, take image sequences that give us an awful lot of information about the objects that we’re trying to document or research, including RTI and AR but also as we’ve seen full 3D models, and we’re also very interested in this idea of keeping track of the process history and what’s happened to make the data more valuable and more reusable.
So, just quickly, we’ve seen a couple of things about RTI recently in the last couple of talks, but let me show you a quick example. This is a 15th century manuscript page from the Bancroft Library and when we have an RTI we can relight the surface also we can apply some mathematical enhancements to the surface.  In this case, what I’ve done is taken all the color out so we’re just looking at surface shape information, and we can see on this very dark projector, some of the surface for this piece of velum is warped, and I want to zoom in here and look at some details. Right here we can see where a letter was scraped off of the velum. We can also see information about ink that’s spalling off the individual characters here and some of the really fine line details.
Another really quick example, this is a Japanese wood block print from the Fine Arts Museums of San Francisco and here is just a screen shot. On the bottom we’re showing the color information from the RTI and at the top we’re showing the surface shape information without any color and you can see the really fine details around the hair and the brow line and so forth.
Okay, so what’s going on with an RTI is that we have a fixed camera position and a fixed object position and we take a sequence of images with light in different positions around the object and then that set of images is synthesized into a new file format, an RTI and this is based on work from Tom Malzbender of HP Labs that was presented at Siggraph in 2001, so we’re over ten years old with this approach at this point. What we have is 2D information or a 2D image that carries also some 3D information. So to explain this a little more, and Roy touched on this and so did Rick a little while ago, but in computer graphics we have a notion of a surface normal. In this graphic here which is a cross section of a surface, the red arrows depict the surface normal and the surface normal is the vector that’s perpendicular at any point along the surface. When we throw light into the equation, light has a physical property of bouncing off of the surface where the incident angle and the exident are equal angles to the surface normal. So if we know where the light was coming from, we know where we are collecting the data, which is the camera, and we have sequenced images from a number of light positions, then we can calculate that surface normal per pixel. So what we have in an RTI is for each pixel, we have the color data like a regular photograph, the RGB data, and we have a mathematical description of the surface normal per pixel. One thing that is cool about this technique is that we can work with shiny material, so in the picture on the left of this gold coin, we have some blowouts on the surface from one of the input images but because we’ve sampled all the way around, when we put it together we have complete surface information.
There are two main ways that we collect RTI data. Down here on the right was the original dome developed by Tom Malzbender. This is an early dome system. We developed, we built a dome for the Worcester Art Museum Conservation Lab and also for the Museum of Modern Art Conservation Lab, but what has really allowed this technique to take off and get more widely adopted was the development in 2006 of highlight RTI and this is an approach that we co-invented with Tom Malzbender. In this case what we do is replacing reflective spheres in the image and then we get a highlight on the sphere and that allows us to figure out where the light was after the fact. So now we can do RTI with a very small amount of equipment. It’s portable. Its equipment most people would already have. Here’s a basic set-up. I’ve got a camera pointing down. We use a string to keep the radius, so we’re basically recreating a virtual dome with a flash. Here’s a similar set-up for a vertical object we can use copy stands, camera stands, a variety of set-ups.
This is just a screen shot from the software showing the highlights on the sphere. There is software that can detect that and that determines the light positions. This is a little map showing all of it and I’ll put in a plug for our demo tomorrow. If you come by we’ll talk a little more, and we’re going to demo how to capture this and we’ll talk a little bit about the software pipeline in more detail. I want to note that all the software is open source. It’s available from our website along with user guides and videos and things.
Another quick example, something that happens in rock art a lot is people want to figure out if one line is on top of another and you can see from this Paleolithic petroglyph in the little call out that it’s really quite easy to tell which line is on top of which other line. We can do this under the microscope as the last paper was showing. This is an example from the Metropolitan Museum and they’re very interested in looking at tool marks on these saddle fragments. We’re seeing a lot of adoption with RTI a number of major museums, primarily driven by conservation, are starting to use this now and this has been really aided by a grant we received from the Institute of Museum & Library Services. That has allowed us to deliver a four-day training in RTI and we’re delivering it ten times and that’s occurring over a broad range of museums and graduate programs.
So at this point I’m going to turn it over to Mark.



Mudge:    Thanks, Carla and I’d like to thank the NCPTT and the National Park Service for putting together this really terrific forum.
I’m going to talk about algorithmic rendering, which is another form of computational photography and in this case, we’re using the same data that you collect in an RTI but taking the shape information and the color information and applying single processing filters to it to generate scientific illustrations. Here we have an illustration of a pine cone taken from a normal and color thing and that’s just not going to laser scan very well. Here are examples of different types of signal processing filters. There’s all sorts that we could show you but we don’t have a lot of time so I’m going to move through here quickly. This would be, I think, of interest to the National Parks people. Here we have a petroglyph at the Legend Rock State Park in Wyoming and you can see down in the bottom area there is a very heavily patinated section that’s very difficult to see and it’s actually not just the projector but it’s difficult to see in the real world. However, if we do a signal processing run on this, you’ll see that you can see all sorts of little zoomorphs that were hidden under the patina, and the result was that the Wyoming State Archeologist was able to uncover two new zoomorphs from this information and this panel had been studied heavily for over thirty years.
So, there are actually hundreds and hundreds of signal processing routines that can be brought to bear for new types of illustrations. Now we’ve received a grant from the National Science Foundation and we are partnering with Princeton to develop what we call the collaborative algorithmic rendering engine and it’s a three-year collaboration and we hope to have some results in about eighteen months. Of course, like RTI, all of the software we’re producing is going to be open source. Now the thing about the care engine is that it creates an extensible framework for anyone who wants to design signal processing routines to plug them into this system, and it will allow the user to bring in RTI data and select the algorithms that most effectively represent their material. You can mix and match, change parameters and we’ll keep a complete process history of everything they do such that at the end of the day, they’ll not only have a scientific illustration with complete provenance information of how it was done but we’ll also have an expanding recipe book of how to represent different types of subject matter and that can grow as people add more and more variations to the operations.
I’d like to say that we also frequently shoot RTI with photogrametry, which gives us the ability to distortion correct the RTI input images or to rectify them and provide that kind of input for the algorithmic rendering process but when we’re talking about close range photogrametry, I just want to show us a little example that we shot a few years ago. It’s a piece of a cuneiform cone that’s about this big and we got a really good camera calibration and from two overlapping stereo images, we developed this you see on the lower left that we’ve got texture. In the upper right we have the surface mesh and as we get closer and closer, we can finally start seeing some of the vertices and the mesh.  So you can take your photogrametry and get it as precise and sub-millimeter as your optics.
Let’s take a brief second right now and jump around because I’m going to jump into the philosophy of science for just a second. We think that it’s useful to think of digital representations in the three ways. The first is fine art and entertainment, which we all understand. The second are visualizations and we’ve all seen visualizations like the comet that came down and hit the Yucatan and wiped out the dinosaurs. Well there’s some scientific data in this visualization but a lot of speculation. We have no idea what the shape and color and craters on the asteroid or the comet looked like. But we’re just putting in speculative stuff and visualizations are part scientific content and part speculation. But finally we have something called digital surrogates and digital surrogates are digital representations that enable the scientific study of the subject without the physical presence of that subject and digital surrogates are built scientifically. They’re built along scientific principles and you have data that can presumably be confirmed by somebody else and everything you do to the data after that, all the processes run on it and the final result are all captured in a scientific lab notebook. This is a way that permits other people to evaluate the quality of your work and to permit replication. The quality and the evaluation of quality of one person’s work by another is the key to scientific imaging and it’s critical that you have this account or either have a nice visualization or entertainment. So at CHI, we have a concept of a digital lab notebook and that’s to both collect information about the capture and then everything that happened afterwards and we want to both collect this metadata and manage it in an organized way throughout its lifecycle. It’s really important that we’re collecting this lab notebook metadata, and the process for collecting this can be built into the imaging tools if you structure the imaging tools properly. It can, in fact, with computational photography tools, be done automatically so that you yourself as the user need worry very little about the metadata, only at the beginning when you’re saying what you’re shooting, whose there and so forth. So the goal is management of the digital lab notebook metadata in a way that includes the relationships among that metadata and hopefully, in a form that can be linked semantically and allow for excellent query and other forms of search.
So if you have the capacity to evaluate the quality of an image, the digital representation, this enables distributed scholarship because you can have people pooling digital surrogates from around the world. You can have other people using other people’s work and this is exactly the idea behind how the human genome project worked. If they all had processed accounts of how they’re data was collected and that enabled them to determine how to trust that data that was there and it also allows for future reuse of the data we collect today because in the future if someone wants to be sure that they can trust a piece of data, they’ve got to be able to see how it was done for them to be able to do this. That allows for growth and maturation of investments in digitization.
So we have the digital knowledge lifecycle for capture, use, regeneration, and we want to track the metadata all the way through and both RTI and algorithmic rendering are designed to use this digital lab notebook process and it’s not just us at CHI talking. Exactly the process history in metadata management I just discussed had been accepted by the top computer scientists working in cultural heritage in Europe in the 3D collection formation framework 7 project of the European Union and they bought into all of this stuff including laser scanning, so reuse and repurposing are fundamental for digital imaging and the test is passed if the information permits the evaluation of its quality by contemporary and future scholars and digital information that does not permit qualitative evaluation is of little value to science and scholarship. So we know now how to capture many sequences of images that give us rich raw data and metadata notebooks and this can democratize the capture of information around the world. It breaks the reliance on a client service bureau method. It allows quality evaluation and almost anyone can create scientifically reliable documentation and thank you for listening.


Speaker Biography

Carla Schroer is co-founder and director of Cultural Heritage Imaging (CHI) a California non-profit corporation, incorporated in 2002. Carla leads the training programs at CHI, along with working on field capture projects with Reflectance Transformation Imaging and related computational photography techniques. Carla also directs the software development and testing activities at CHI. She spent 20 years in the commercial software industry, directing a wide range of software development projects including Sun Microsystems’ Java technology, object oriented development tools, and desktop publishing software. She has extensive experience in software licensing and open source projects in both the commercial and non-profit sectors.