OpenTopography

OpenTopography Facility Cyberinfrastructure

OpenTopography System Architecture

The architecture of the OpenTopography System is show below. A brief summary of the various components are shown below the architecture diagram. You can click on the various components in the architecture diagram to jump to the description of the component.

Gridsphere Portal Framework

Gridsphere portal framework provides an advanced open-source portlet based Web portal. The portlet model provides users a customized and flexible user interface and allows for each portlet to be deployed on its own if necessary.  In OpenTopography, Gridsphrere is used to manage all data access and processing aspects of the system.

More information: http://www.gridsphere.org

Expression Engine

Expression Engine is a content management system written in PHP.  It is highly flexible and supports plugins, modules and extensions for added customization and functionality.  In OpenTopography, Expression Engine manages all of the standard website content such as the news items, blog, and other pages.

More Information: http://www.expressionengine.com/

Multi-Node Partitioned IBM DB2 Database

Currently, all LIDAR point cloud data is hosted in a multi-partitioned IBM DB2 database. In the multiple partition configuration the LIDAR database tables are partitioned across multiple machines or “nodes”. Each node is a dual processor machine with 8GB of Memory and attached to 2 TB (approximately - varies by node) of local disk. Each partition is managed by an independent database manager, each with its own data, configuration files, indexes, and transaction logs.

This architecture provides better scalability and performance. New machines can be added to the cluster and the database can be expanded across them.

View Real Time System Load

Point Cloud Processing (Compute) Cluster

The point cloud extracted from the database is processed on this cluster using Local Bining Algorithms and/or Spline Interpolation Algorithms.

The local binning algorithm utilizes the elevation information from LiDAR returns contained within a circular search area (radius defined by the user) to calculate the DEM elevation at each grid cell (resolution specified by the user). Five values are computed for each node in a grid: 1) the minimum, 2) maximum, 3) mean, and 4) inverse distance weighted mean of the local points, and 5) the number of points in the search area. If the number of points in the search radius is 0, the grid node is assigned a null value. The local binning algorithm was written by HanSuk Kim (UCSD).For more information on this algorithm please refer to the documents section under the resources tab. Note that the defaults are that the search radius is grid resolution * sqrt(2)/2 which gives optimal coverage for each grid cell (but the area of the search radius is 1.57x larger than the grid cell area).

Interpolation of the LiDAR data and calculation of derived products is performed via GRASS Open Source GIS. GRASS is running on the SDSC LiDAR Compute Cluster and is accessed in this workflow via web service architecture. Currently, spline interpolation jobs are limited to 1.6 million points.

Current System Status: http://www.opentopography.org/index.php/about/systemmetrics

Kepler Scientific Workflow System

Kepler is a cross-project, multi-disciplinary collaboration to build open source tools for scientific workflows that provide domain scientists with an easy-to-use, yet powerful system for capturing and automating their ad-hoc process. Kepler is built on top of the PtolemyII system developed at UC Berkeley, which provides a set of java APIs for modeling heterogeneous, concurrent and hierarchical components by means of various models of computations.

More Information: https://kepler-project.org/

Backend Databases

In addition to the main LIDAR DB2 partitioned database, there are a number of backend databases that include PostgreSQL databases that support Gridsphere, jforums and integrated user authentication database, MySQL database for supporting Expression Engine and an IBM DB2 database for supporting LIDAR Point cloud monitoring and user job statistics and personal workspace.

File Servers

Standard DEM Tile Servers: These servers are hosted at ASU and deliver the tiled DEM data data are available via OpenTopography.  The DEMs are typically delivered as bare earth (ground) and full feature (all returns) surfaces organized into tiles (e.g. 1 km2). The DEMs are in a GIS compatible format and are compressed (zipped) to reduce their size.  Also running on these machines is custom logging software that allows OpenTopography to track data access.

Streaming Google Earth HillShades Image File Server

LiDAR Point Cloud Bulk Download File Server: The bulk download feature is designed for advanced users seeking to access large amounts of data quickly and who have the bandwidth, expertise, and software necessary to manage the gigabytes of data this method delivers.

Visualization Web Services

INCA

The Inca system provides user-level Grid monitoring with periodic, automated user-level testing of the software and services required to support Grid operations.

More Information: OpenTopography INCA Monitoring Service

INCA Project: http://inca.sdsc.edu/

jForums

JForum is a powerful and robust discussion board system implemented in Java available freely under the BSD Open Source license. Built around a MVC framework, it can be deployed on any servlet container or application server.

More Information: http://www.jforum.net/

Research Areas

Performance Evaluation of On-Demand Provisioning of Data Intensive Applications (e.g. LiDAR Topography)