Objective #2: Develop a Spatial Database Prototype in New Mexico based on the Logical Data Model
The NMCRIS spatial database prototype consists of a database server component, serving as the main data repository, which is connected over local and internet computer networks to multiple client components (applications) used for data capture and query (Figure 3). Our implementation strategy divided the effort into three overlapping sets of tasks focusing on the spatial database server and two application development tasks:
- Creating the spatial data server
- Developing the data capute application
- Developing the spatial query application
Creating the Spatial Data Server
During the early project phases, emphasis was placed on training staff and creating a stable test environment for the prototype separate from the production NMCRIS. Installation of SDE was trouble-free. Following the translation of our logical data model to a physical model in Oracle and SDE, existing spatial data was then converted from Arclnfo to SDE. As applications were developed for data capture, the ARMS staff was able to immediately use SDE for query and analysis tasks using ArcView as the interface to SDE.
Many decisions had to be made in creating the physical design and integrating SDE with NMCRIS. Some of the more important ones are listed below and are documented in Appendix 3:
- What SDE layers are required to represent cultural resources?
- How should SDE layers be related to existing database tables?
- What metadata, if any, should be stored along with each feature?
- What coordinate system and projection will be used?
- How will features be represented? What “shape masks” will be used in each layer?
- How should the SDE tables be populated?
- What is the appropriate spatial index grid size for each layer?
- How will average feature size (no. of vertices) affect database space allocations?
- Who should have access to SDE-related database tables and for what functions?
These decisions and many others were postponed until ARMS technical staff had completed the first round of training in Redlands CA (“SDE Administration”) and the logical data model was developed. Training was absolutely essential. Most decisions also benefit from a comprehensive understanding of the underlying database structure and the Oracle RDBMS NMCRIS has been operational for over five years so we were not faced with major database design issues we just had to create new SDE layers and relate them to our existing Oracle tables. Having a well thought out logical data model from the first FGDC workshop was also essential to a smooth implementation.
SDE modifications to the NMCRIS physical database design are illustrated in Figure 4. Table 3 lists the five active layers created to represent Resources (archeological sites) and Investigations (archeological surveys) A layer was also created to represent USGS Quadrangle boundaries used mainly for query purposes. In the future ARMS will provide additional layers for non-archeological Resources, Resource Aggregations and other entities. Appendix 3 provides details of the content and structure of the five main SDE layers.
Application development efforts first focused on the ArcView data capture tool. This application, allows the ARMS staff to perform on-screen (“heads-up”) digitizing on a report-by-report basis. The primary design goal for this application was to provide a means for non-technical staff to quickly capture geospatial data with minimal training and disruption of work flow. The application thus closely follows current processing procedures where reports are processing in serial fashion and spatial data are transferred from the source graphics in each report to a master Set of USGS 7.5’ topographic maps. Appendix 4 presents a description and illustration of this tool.
During testing of this application, ARMS staff realized significant gains in productivity. We estimate this gain to be 2 -10 times greater productivity over our established digitizing methods, depending on the complexity and density of features involved. These gains are so significant that the test system has since become our production system for survey area data collection.
After basic data capture functionality was established, ARMS technical staff prepared system requirements and a scope of work for ESRI–Boulder to develop a text-based query application using SDE. This application uses the SDE “C Application Programming Interface” to spatially enable a simple text-based query tool that has been in use for the past 8 years by ARMS staff and outside users. The basic query program runs on one of the ARMS UNIX servers and is accessed via Telnet and dial-up connections. This extension greatly expands the utility of this simple text- based tool by using SDE technology to resolve fairly complex spatial queries and generate reports. Previously, users were able to enter a single rectangular query area defined by four UTM coordinates to query archeological site locations and could only retrieve information on previous surveys through USGS 7.5’ quadrangle units. Using SDE, users will now be able to specify any number of query areas in any configuration, define a buffer zone, and automatically receive information about archeological sites, surveys, and other investigations taking place within their query area. This insures that comprehensive pre-field records checks are conducted. In addition, SDE automatically reprojects UTM coordinates between New Mexico’s two Zones an operation that also previously required two separate queries. Appendix 5 provides some insight into the functionality of this component. The query logic built into this application will be an essential part of many other business processes at ARMS and we expect it will be integrated into many future applications.
Choices, Choices, Choices
SDE provides a robust environment in which to capture manage and distribute complex spatial data It does this mainly through the underlying RDBMS: spatial data, indexes, and other related data are stored and maintained directly in RDBMS tables. But SDE is of little utility without client applications to enter, edit, and distribute information.
There are many technical directions and these are changing almost on a daily basis. Table 4 lists some of these choices along with some pros and cons of each. We have made our decisions in this area based on our development schedule, our available technical skills, and our existing information system resources. We picked ArcView/Avenue for data capture mainly because it can access SDE right out of the box and the digitizing functionality is built in. We were thus able to concentrate on the actual SDE transactions rather than worrying with user interface issues ArcView requires a significant amount of desktop computing power and some training, but for us these costs do not outweigh the time required to develop a new user interface. We would still need ArcView for query and analysis purposes, so ArcView was the logical choice.
In the long run, we will probably use several different tools for data distribution. We are relying on the “C” API for our basic query tool in order to leverage SDE’ s query capabilities as widely and quickly as possible and with minimal end-user training costs. We were able to use consultants to develop the spatial query program, thus allowing us to concentrate on the data capture applications in-house. In the future, we will be developing graphic query tools so that we can archive our fragile paper maps and make spatial data more accessible to cultural resource managers over the internet.
In our proposal, we stated that “GIS has the potential to fundamentally change many SHPO operations” but that to be successful, the technology “must be fully integrated into the daily work routine of SHPOs… before these kinds of fundamental changes can be realized.” GIS must “help solve basic information systems functions relating to data collection, quality assurance, and data delivery, or the added expense of GIS may be difficult to justify.”
It is expensive. Historically, it has been hard to justify. We believe SDE justifies the expense. Our experience gained in the course of this project confirms our original suspicions: SDE solves problems in the capture, management, and distribution of cultural resource information. Spatial data collection has been one of the biggest obstacles for NMCRIS. – The process is now greatly simplified over our previous approach. SDE allows an efficient transactional approach to data capture, uncomplicated by feature topology or the need to subdivide space into manageable units. With SDE, sites, districts, and buildings — rather than space — are the central organizing principle. This allows us to maintain a more logical and efficient work flow, and requires far less training, than our previous approach to spatial data capture. Put simply, we can be more productive with SDE.
Spatial data management functions are handled by the underlying RDBMS — a very mature and robust technology. All records — spatial and non-spatial — are inserted, modified, indexed, and deleted in the same database environment allowing, for example, automatic recovery of digitized site boundaries following a system crash. Administrative costs for spatial data management are thus rolled into our overall RDBMS administration, resulting in significant savings. Moreover, the integrity and security of spatial data are greatly increased in the RDBMS environment.
SDE-based data distribution options are many and diverse. Powerful client applications are available out-of-the-box with ArcView or may be developed in a wide variety of environments. Effective SDE- based solutions can range from simple text reports, as in the NMCRIS SDE enhanced query program, to custom map server applications requiring that the user have only a Web Browser and internet connection. Combined with SDE, these tools are the key to integrating spatial data into the daily work routine of cultural resource managers and finally realizing true value from GIS technology.
We have yet to see any serious technical obstacles to using this technology. Could it be improved? You bet! There are several areas in need of improvement. First of all, it is too expensive. Competing products and increasing economies of scale are needed to bring this technology down to an affordable level. This is already happening. SDE’ s reprojection and administrative tools could also use some enhancements and we’d like to be able to use SDE to handle raster data such as USGS Digital Raster Graphics, as well as vector data. The good news is that these, and many other improvements, will probably be implemented in the next version of SDE and possibly in competing products.
Cost may be the most significant liability of the technology. SDE and RDBMS still require a fairly substantial investment. Initial costs for hardware and software are much lower than when ARMS invested in RDBMS almost a decade ago, but the figures are still substantial for a small state agency. Annual maintenance costs are substantial, but recruiting and retaining skilled technical staff is an even bigger expense. Adult supervision is required SDE and RDBMS technology should not be implemented without skilled technical staff or consultants. Training is essential to retain staff and keep up with technology nobody or at least nobody you can afford has all the required technical skills. Consultants and partnerships with other better technically endowed agencies should also be considered.
There are also some indirect benefits to consider. Because SDE allows location to be fully integrated into NMCRIS, we have begun to see more errors. Many of these are simple locational errors (“these coordinates are in the wrong UTM zone!”) and are easy to fix once isolated, but having site and survey locations at ones fingertips has revealed more logical errors (e.g., “what is this Anasazi site doing in Catron County?”) The long-term result: better error trapping and more reliable data.
Many other phenomena have spatial dimensions besides sites and surveys. Ecological zones, soil types, and land ownership, for example, are often cited as critical in an archeological CRIS SDE provides a relatively straightforward method for utilizing these data for query and analysis without requiring observations in the field and another data item on the site form. Accurate, consistent spatially referenced data are widely available and are getting better every day. The result better analyses and better management decisions.
In summary, we have found through direct experience that SDE solves most difficult problems surrounding spatial data in a CRIS environment. The investment is substantial. For New Mexico and other “high-volume” states the expense of implmenting GIS using SDE is proportional to its benefits.