Image Enhancement for Three-Dimensional Analysis
The ability to determine the position of hidden objects within an assembly is highly desirable but can be elusive. While several types of nondestructive evaluation (e.g., impulse radar, impact echo, electromagnetic detection) are somewhat capable of discriminating between masses of differing density or composition, none provides an intuitively legible pictorial representation.1 The promise of radiography lies in the fact that unlike the highly mediated data readouts of sonic methods (Figure 79) and the imprecise feedback of electromagnetic devices, radioscopic data “displays” spatial relationships between objects of differing density.
It is this property of naturalistic imagery that motivates the desire to better identify and understand the relative position of the masses of differing density. Optical and cognitive abilities allow humans to rapidly and successfully distinguish discrete forms and to perceive depth of field and relative distance.
1 In a study of the use of various NDE technologies in the investigation of the New York State Capital researchers concluded “Of the NDE techniques employed on this heavy masonry building, radar proved to be the most successful for imaging hidden structure and conditions. Next, in order of success, were impact echo, ultrasonic pulse velocity, spectral analysis of surface waves, and infrared thermography. Electromagnetic detection was very useful, but its scope is limited to buildings that contain some iron or steel and to locations where framing members are isolated from pipes, conduits, and other metal features. Fiber optics was found to be of minimal use in this type of building because of the limited areas where voids were present.” Radioscopy was not part of their study. The case was documented by the U.S. National Park Service Technical Preservation Services, and published as Preservation Tech Note #4 (http://www.cr.nps.gov/hps/tps/technotes/PTN40/NYStateCap.htm).
The question the authors asked was how can radiographs allow for, and support, this innate ability?”
Determining spatial relations between areas of varying density can be achieved optically or by calculation. Preliminary investigation into both approaches was undertaken for this study. Stereoscopic photo-interpretation is based on geometrically constructing camera angles so as to allow for stereo-viewing of the combined image. The technique is widely applied to visual media other than radiographs in applications as diverse as the child’s hand-held “View-Master” stereoscope to equipment for the interpretation of aerial photography. Photogrammetry – “the art and science of making measurements from photographs” – when based on multiple images, can further the geometrical analysis through the application of mathematical algorithms.
In the scope of this study, the research team concluded that gross spatial relationships of objects of varying density can be distinguished through both stereo-optics and algorithmic modeling. However, given the difficulty of geometrically precise field setup of equipment, and the imprecision and error inherent in measuring and inputting indistinct radiographic “shadows” into a model, the human eye and brain appear to be the most efficient way of making judgments about spatial relations at this time. Improvements in either field setup or post-processing may tip the scale. This possibility warrants additional research.
Stereo-optics (Three-dimensional Radioscopy)
The simplest way to determine the three-dimensional (3-D) structure of the interior of a wall is to use some type of stereo viewing. This can be accomplished with two radiographs taken of the same area from different angles. The equipment necessary ranges from nothing (you can train your eyes to view two adjacent stereo photos in 3-D) to expensive computer programs. The most basic type of equipment is a pocket stereoscope (which can be purchased at companies like Forestry Suppliers, ASC Scientific or Ward’s for about $30).
Depth perception and 3-D viewing occur when your brain integrates and interprets the images received from each eye. The visual cues that allow this to occur include relative size, linear perspective, shade, shadows, and increasing fuzziness with distance. But physiological processes are even more important. Binocular divergence, which is the difference between the images that each eye receives of a particular object is closely tied to convergence, which is the angle made by the two lines-of-site of the eyes. Binocular disparity and convergence are the most important features used in stereo viewing of photographs, aerial photos, and radiographs.
The first stereoscopic viewer was invented by Sir Charles Wheatstone in 1838, shortly after the invention of photography. Simple devices to view stereo photos of vacation spots or exotic locals were common in the early 20th century, and these can still be purchased at toy stores today.
Equipment has been developed for use in mapping, interpreting aerial photographs, and viewing remote images produced by satellites. The most direct approach to simple viewing ranges from the inexpensive pocket stereoscope mentioned above, which uses only two convex lenses on a collapsible stand to much more complex models that use lenses, prisms and/or mirrors. With increasing complexity comes increasing cost, but greater ease of stereo viewing. In the last several decades, stereo plotting systems have been developed, and have become increasingly automated (Heipke, 1995). But these systems are primarily used for photography and are probably “overkill” for radioscopy .
Numerous sites are available online that contain additional information on the basics of stereo viewing. The Canada Centre for Remote Sensing has a site that provides an excellent overview of stereoscopic viewing: http://www.ccrs.nrcan.gc.ca/ccrs/learn/tutorials/stereosc/chap1/chapter1_1_e.html (no longer online).
Gale Rhodes, of the Department of Chemistry, University of Southern Maine has created a site that discusses stereo viewing of photographs and computer screens using no equipment, only the unaided eye: http://www.usm.maine.edu/~rhodes/0Help/StereoView.html
Other helpful sites affiliated with either universities or government organizations give some details on viewing stereo photographs, usually aerial photographs and their interpretation. However, the same principles can be applied to the viewing of stereo pairs of radiographs. These sites have helpful information on the use of pocket stereoscopes and 3-D visualization. They include:
http://rst.gsfc.nasa.gov/Sect11/Sect11_3.html (link no longer available)
http://www.fes.uwaterloo.ca/crs/geog165/api.htm (link no longer available)
The radiographs taken in this investigation were typically taken in a plane 24 inches from the object and offset 3 to 6 inches to the left and right of the target area, as see in Figure 80. The images are then taken of the same target area, but from different angles. Sequential radiographs can also be taken along the length of a wall and viewed as stereo pairs, but the area of stereo viewing will be limited to the area of overlap of subsequent radiographs.
When using a pocket stereoscope to view stereo pairs of radiographs, it is easiest to have printed copies of the radiographs to work with. They should be lined up adjacent to one another, with the radiograph taken to the left of the target site on the left side, and the one taken to the right on the right, so the radiographs have the same relationship as do your eyes to the target area. They should also have the same orientation (in the viewing plane, one radiograph should not be “crooked” or tilted at an angle relative to the other). If they are tilted because of the geometry of the x-rayed object, simply rotate one radiograph so that the tilt is eliminated. Find an area in the center of the radiograph that is the same location, and line this point up on both radiographs so that the points have the same separation as the distance between your pupils (it should be about 3 inches). Set the pocket stereoscope up above these two points, with the center of the lenses separated by the same distance. Look down through the stereoscope as if looking off to infinity, and you should see the image in three dimensions.
When looking at stereo pairs, whether of aerial photographs, normal photographs, or radiographs, some vertical exaggeration can occur. This means that the dimension perpendicular to the viewing plane is exaggerated. It is termed vertical exaggeration because it is usually an issue when viewing aerial photos. This exaggeration occurs because the ratio of the distance between successive photographs and the distance above which they are taken is different that the ratio of the distance between the viewers eyes and the distance to the photos or radiographs.
Normally, when viewing aerial photos, this vertical exaggeration creates the illusion of steeper mountains, taller trees, etc. While this does help with viewing in 3-D, it must be accounted for in mapping exercises. With a radiograph, the dimension which is exaggerated is the depth of the wall. Again, this helps to see the relative position of nails, screws and other fasteners, but it must be taken into consideration when used to accurately locate items in a wall.
Below are two stereo pairs that can be viewed with a pocket stereoscope. Figure 81 shows two radiographs of three screws in a 4-inch by 4-inch wood block. These were taken with the source 20inches from the object, offset three inches to the right and left (see Figure 80 above). When viewed in 3-D, the bottom of the screw on the right is angled sharply toward the viewer, the screw in the middle is relatively vertical, and the bottom of the one on the left is angled to the right and away from the viewer. Make sure you can see this in 3-D before trying Figure 82, which is more complex.
Figure 82 shows two radiographs of the complex wall. They were taken with the source 23 inches from the object, with a 3 inch offset. In this figure you should see the bottom nails tilted, the center two away from the viewer and the outer two toward the viewer, the large rebar is roughly in the center of the wall, in front of the horizontal blocking, and the loop of wire is behind two large nails, but in front of the blocking.
Determination of Size and Position Using a Single Radiograph
Occasionally the size of wall components and their location within a wall can be determined by using some simple geometric principles (discussed in the section on Geometry above). Because a radiograph is essentially a shadow cast on the imager, the closer the object is to the imager, the smaller it is and the closer it approximates its actual size. When viewing a radiograph, the view always appears as if from the source side of the object, unless you mirror image the radiograph. If the image has multiple nails of the same size and orientation, the smaller nails are to the “back” of the wall (closer to the imager), while the larger ones are closer to the front (the source side). A large threaded rod with bolts on both ends at an angle through the wall would have its thickest end towards the source. These observations sometimes allow an appreciation of the relative locations of wall components without calculations or stereo viewing.
When this simple approach is not sufficient, the relationship between object size and image size, which is a function of the relative distance between the source and imager and the source and object can be used. It is defined as:
SO/SI = DSO/DSI
SO is the size of the object,
SI is the size of the image,
DSO is the distance from the source to the imager, and
DSI is the distance from the source to the object
If the object size is known (a standard size nail or screw, for example), its location within the wall can be determined by calculating DSO, and subtracting it from DSI to obtain the distance inside the wall from the imager:
If DX is equal to DSI (the object is adjacent to the imager) the size of object and image are identical. This is the largest possible actual size for the object. If DX is equal to DSF, this establishes the smallest possible actual size for the object.
These relationships work best if the object and imager are parallel to one another and perpendicular to the face of the object. With small components such as nails, using the long axis as a reference size will only work if the nail is parallel to the sides of the wall and the imager. For equidimensional objects (the maximum diameter of a round bolt head, for example), this works well.
More complex mathematical relationships of multiple wall components can be incorporated using two adjacent radiographs taken with the same source-to- image and source-to-object distances. This requires establishing common reference points.
Calculations in Three Dimensions Using Two Radiographs
While fairly simple estimates of location and/or size can be made using one radiograph (if the setup distances are known), more accurate calculations can be made using two radiographs that are taken by moving the source parallel to the plane of the object while keeping the imager stationary. These trigonometric calculations are well established both for radiographs and photography (and have been widely used for things such as topographic mapping, using aerial photography). The information in this section has been taken from an excellent reference on x-ray photogrammetry by Bertil Hallert (1970).
The optimum setup to collect radiographs that can be easily used to make these calculations is to set the source so that it shoots perpendicular to the plane of the object. If the first image is taken on the left of the target area, and then the source is moved to the right for the second image (without moving the imager), the conditions are met for using the relationships defined in Figure 83. Other, more complex calculations for different geometric positioning can be found in Hallert (1970).
The following parameters can then be measured and/or calculated:
- O1 is the location of the source for the left-hand image.
- H’ is the location of the center of the beam from the source (the center of the
hot spot seen on the radiograph), called the Principle Point.
- c is the distance between O1 and H’, referred to as the principle distance.
- b is the distance that the source is moved parallel to the plane of the object, referred to as the base.
- P’ and P” are the locations of the true point P on the two radiographs, left and right respectively.
- The x- and y-coordinates for P’ and P” (x’, y’ and x”, y”) are measured using the left and right radiographs, with the location of H’ as the origin.
After the radiographs have been taken, the measurement of x- and y-coordinates for each point of interest can be facilitated (when using digital radiographs) by using the grid system and ruler found on most image-enhancement software. It is important to note that H’ has the same absolute location on both radiographs, even thought the location of the hot spot has shifted on the right-hand radiograph. It is also important to note, when measuring the x- and y- coordinates on a computer, that the correct image size is used. After measurements are completed, the following formulas are used to determine the true x, y, and z coordinates of each point:
x = ( b / (b + x’ – x”) x’
To assist in using this technique, a set of sample calculations have been made using a setup with three nails. The radiographs of the three nails are shown in Figure 85. The images have a true size of 10.5 inches long by 8 inches high, and the nails are 3.5 inches long, with their head hidden in the board beneath them. The imager was placed directly behind the three nails and 24 inches from the source. The source was moved horizontally 6 inches for the second radiograph. The locations of the coordinate system for each radiograph are shown in red. All measurements (in inches) and calculations are shown in Table 3.
For each of the three nails the points at the top and base were measured. The head of each nail was in the board underneath, so the full length of the nails could not be measured. It can be seen from the above calculations that the nail on the right is closest to the imager (which would be located at a z-coordinate of 24). The nails are spaced roughly two inches apart. The length of the nails (calculated using the true end points in three dimensions), is close to the expected exposed length of about 3.25 inches. The y-parallax is very small (the maximum value was 0.03 inch), indicating that these measurements are reliable for identifying the location and size of the nails.
A second test of the equations was done using the three angled screws shown in Figure 81. These screws had a slightly more complex placement than the three nails in Figure 84. The calculations for these are shown in Table 4. They show the three screws are placed close to the imager (z values between 21 and 24); the left screw has the bottom point back toward the imager; the middle screw is nearly vertical and the right screw has the bottom pointing toward the source. The lengths of all three screws are estimated at 3 inches.
Research into the integration of radiography and photogrammetry reached its apogee by 1980. Before the maturation of current medical imaging technologies (e.g., computed tomography scanning, magnetic resonance imaging) x-ray photogrammetry was a viable area of medical research. By 1970, Hallert published X-Ray Photogrammetry: Basic Geometry and Quality in what became the standard for understanding the transference and transformation of “optical” principles to the projective geometry of radiographs. A medical text, it focused on the geometrical relations of bone replacements and reconstructions (e.g., knees, hips, jaws) and attempted to develop and document techniques for mapping locations and alignment. The 1989 publication by the American Society for Photogrammetry and Remote Sensing’s “X-Ray Photogrammetry, Systems and Applications” by S.A. Veress represents the highpoint of literature in the field.
However, despite the eclipse of the x-ray photogrammetry research in the medical field, optical photogrammetry has more recently advanced in fields ranging from accident reconstruction to archaeology. Advances in digital photography and personal computing hardware and software have made photogrammetry more physically portable, cost accessible and simpler to use. The combination of these advantages has led to combined software and hardware bundles costing less than $1500. Responding to this opportunity, the researchers explored the possibility of coupling the rapidly evolving off-the-shelf photographic software with radiographic image analysis.
Multiple perspectives and calculated dimensions
While photogrammetry is similar to stereo-photography, the possibility of accurately inferring position and measurement of the actual objects from their images distinguishes this technique. Mathematical calculations based on the geometric laws are well established as they relate to optical lenses. This is the basis of modern “optical” photogrammetry (Karara, 1989). Theoretically, x-ray projection is subject to the same laws, albeit several practical problems have impeded the use of x-ray photogrammetry.
A review of the sources of error in x-ray photogrammetry and techniques for correcting or overcoming these demonstrated that controlling the imaging environment, and improving the imaging technology might adequately offset inherent difficulties in post-processing (Veress, 1989). However, at least two key sources of error remain problematic for those attempting to apply photogrammetric principles to x-rays: the penumbra effect, and the ambiguity of the focal point. “The penumbra effect tends to spread out the edge gradients and results in a soft edge on the film” (Veress, 1989). Even more significant is the problem of calibration. In the absence of the focal point and focal length properties of a lens the principal point and principal distance provide the necessary metric inputs for post-processing. The principal point is that point on the imaging plate perpendicular to the focal spot, with the principal distance being that from the focal spot to the principal point (Figure 85). Veress’s review of the literature covers apparatus and techniques for overcoming the calibration problem. He distinguished between physical approaches and semi-analytical approaches.
Veress’s analysis is relevant to the present discussion because it highlights both the nature of the problem and the potential direction for mitigating these problems. Both physical techniques and semi-analytical techniques were tried in our empirical investigations.
Software advances: digital imaging and computational algorithms
Karara (1989) divides the technological development of “non-topographic” (i.e., close range) photogrammetry into eras: 1900-1960 as ‘Analog’; 1960 onward as ‘Analytical’; and 1980 onward as ‘Digital’. The implication of this typology has been a rapid expansion in both recording instrumentation (i.e., ‘cameras’) and in post-processing software. Within the field of architectural photogrammetry this has resulted in several research initiatives, and a smaller number of commercialized software products. Two commercial products marketed for the photogrammetric post-processing of close range photographs are Vexcel Corporation’s FotoG software (http://www.vexcel.com) and EOS Systems’s Photomodeler (http://www.photomodeler.com).
As Photomodeler has developed a more explicit marketing approach to the architectural market and it was known to one of the author’s of this report (Koziol) it was chosen as the product for preliminary assessment in our review. This choice is not meant as an endorsement of this product, and as is concluded below, other products may prove to be more adaptable to the tasks outlined here.
Photomodeler: An off the shelf software possibility
EOS Systems’s Photomodeler is an evolving (currently in Release 5) set of tools for processing inputted geometry into spatial coordinates. Some combination of ‘camera’ properties and graphically picked common ‘points’ on overlapping images, and/or additional geometric constraints (i.e., known points or relationships) are required for computation to proceed and be successful. In theory, the principal distance could be measured between the imaging plate and the x-ray source, and entered as one parameter, and through calibration (e.g., using marker(s) with a known location between the source and imaging plate) several images from differing camera stations should yield a vector model in coordinate space (i.e., x, y, z coordinates).
These several assumptions were used in designing and running a series of analyses. In general, the ‘gross’ spatial relationships of denser objects within an assembly were discernible, but the amount of effort to attain these results seem inordinate to the information gained. As this preliminary conclusion may be the result of inexperience and inefficiency, we document the procedure here as the basis for possible improvements on the process or logic employed.
In each of the experiments, multiple images were produced by varying the spatial relationships of the imaging plate, object, and source. A variety of assemblies, angles, and distances were tried. For example, Figure 86 shows the assembly with three screws in the 4-inch by 4-inch wood block which were x- rayed with the source shifted three inches each direction from center. Of particular interest is the spatial relationship between the small “marker” on the imaging plate as related to the corner of the wood block (circled). The positioning of the elements is shown in Figure 87.
Although it would seem that a more extreme angling of the x-ray source would have yielded a greater difference in perspective, this option, often used by “optical” photographers, was practically limited by the need to minimize both the penumbra effect and the dissipation of x-ray strength at the edges of the imaging plate.
The identification of common points between multiple images is a key aspect of inputting adequate data for the Photomodeler computational algorithms to run with minimal error. This software requires the operator to visually identify and mark these points. The screen layout allows this to be done with side-by-side images (Figure 88).
This process is facilitated by several tools provided in the user interface, but it remains a task that is both time-consuming and subject to operator-caused inaccuracies. After a sufficient number of points are identified, it is possible to process the images into a coordinate model. Figure 89 shows marked points and lines, and Figure 90 shows the processed “model”.
Each point and line in this processed model can be located in coordinate space. The built-in viewer in Photomodeler allows the operator to rotate and query the model.
Real-time rotation of the model reveals that the piece of rebar (large triangle in model) sits “forward” of the other inclusions. (The second image of Figure 91 shows this.) While this model provides photogrammetrically derived “information,” the process to attain it was tedious and pending future findings to the contrary, the result seemed to offer few analytical benefits.
Photomodeler creates its files in a proprietary format, but allows for easy export as a ‘dxf’ [Drawing eXport Format] file. Hence, the model can be incorporated into Autodesk AutoCad, or most other three-dimensional software packages.
Although Photomodeler did not prove to be immediately useful for this analysis, experimenting with the software helped identify several of the operational constraints in setting up future studies. The present experiments with Photomodeler would have benefited from better “front end” documentation of physical measurements in a format that could be used in Photomodeler. Additionally, a better means of identifying and recording known spatial positions (e.g., on the imaging plate) may have allowed for more extensive input of point data, and hence, improved the accuracy of the output model. The small pin on the imaging plate was useful, we believe that an even more precise multi- point reference system would have benefited the experiment.
This system might be a modification of some of the calibration systems reviewed by Veress (1989). One possibility would be a grid of known dimension (previous researchers have used one made of lead wire set into an incised grid of Plexiglas) set directly on the imaging plate. This could then be calibrated with known (i.e., marked) locations on the ‘surface plane’ of the object assembly. However, while such a calibration system could in-principle improve the accuracy of the output, it would also require greater attention to setup and data entry.
Recommendations for future research and applications in 3-D analysis
Despite not identifying a “breakthrough application” for the photogrammetric analysis of radiographs of building assemblies, the present study is far from conclusive. Communications between one of the researchers (Koziol) and several industry and academic experts suggests that “the mathematics are known” for developing a set of computer algorithms capable of efficiently converting radiographic images into coordinate models. However, it would take additional effort to (1) “tweak” Photomodeler, (2) develop a sufficiently sophisticated physical calibration apparatus, or (3) identify a different software package without the same limitations (for this application) as Photomodeler. In determining whether to undertake further investigation, it would also be prudent to conduct a potential benefits analysis of such a project before committing additional resources.