This report explores the use of range sensing technologies in reconstructing three dimensional surface profiles through the implementation of remote data access and control protocols. Software components that handle data publishing, storage, and analysis are developed using C# on the .NET platform. The project combines skills from various engineering disciplines, namely mechatronics, computer vision and sensor technology. The final product encorporates several hardware and software components that enable registered clients to remotely access the range sensor and aquire total control over the sensor prototype.

[Top]

Introduction

Several methods to digitize and reconstruct three dimensional models of objects have evolved rapidly in recent years. The high speed of digitizing technologies is mainly attributed to advances in the fields of physics, electrical engineering, and information technology. Moreover, the digitization and reconstruction of three dimensional shapes have numerous applications in areas of manufacturing, virtual simulation, science, and medicine (Cureless). Range sensing is a well established technique for detecting object depths. Through this method, structured light striking an object is observed by a camera to reveal surface profiles that are converted to depth data (DePiero & Trivedi). Applying similar techniques for practical purposes requires a well-established system designed to enable remote control and configuration of the sensing devices.

This paper presents ProRAST.NET (Remote Client and Web Service), a range sensing system for the reconstruction of 3D objects. ProRAST is based on a fully automated process whereby an object scan can be conducted over the internet while having total control over the lab setup and retrieved results. The novelty of ProRAST stems from applying the latest Web Service technology, provided in Microsoft’s .NET Framework, in order to provide real-time access and control over the range sensing apparatus. All software components were developed in Visual Studio .NET using C# as a programming language.

First, we provide a brief introduction to the various 3D computer vision techniques. An in-depth description of the different hardware and software components is then presented. Finally, the significance of the system is described in view of possible future applications.

[Top]

Calibration

The point of calibration is to extract a camera system’s intrinsic and extrinsic parameters. With these parameters at hand, one may compute the world coordinates of a particular point in space (the upper case ‘P’ in Figure 1), given the corresponding image coordinates of the same point on the image plane (the lower case ‘p’ in Figure 1). World coordinates are often represented as X, Y and Z; and image coordinates by x_img and y_img or simply x and y.

Figure 1. Relationship between image and world coordinates

Without going into the details of derivation, calibration seeks to solve the M matrix in the following equation:

Where and . The M matrix is called the projection matrix. Its coefficients are:

To determine the coefficients of the projection matrix, one must resolve to the method of eigenvalue decomposition. There are seven unknown camera parameters, and a system of seven equations may be formed using the above matrix. The coefficients formed by these seven equations require seven sets of world coordinates and their corresponding image coordinates. The seven points are often selected from a simple calibration pattern such as the one shown in Figure 2 below. Usually the world coordinates of these seven points are predetermined and the corresponding image points may either be manually extracted from the image or extracted using an edge or corner detection algorithm.

Figure 2. A photo showing a point's image coordinates on a calibration pattern and its corresponding world coordinates

Therefore in developing a program to solve the calibration problem, one would first determine the M matrix using eigenvalue decomposition and seven sets of image and world coordinate; and then use this matrix in a function that takes image coordinates and returns world coordinates.

[Top]

Computer Vision Techniques

Methods for reconstructing three dimensional models of objects are classified as either passive or active. The major difference that distinguishes the two techniques is related to the object-system interaction that occurs through the data acquisition process.

Passive methods

Passive methods for reconstructing objects don’t incorporate any means of direct intervention with the studied object. Conventional cameras capture two dimensional images of the object and computational approaches infer the 3D representation from one or more images. Therefore, capturing a number of images at different angles with respect to an object enables the development of a fully representative model. Advantages of passive imaging techniques include low cost associated with the simplicity of the imaging hardware (DePiero & Trivedi). Moreover, no special energy source is required to illuminate the scene. However, several challenges arise due to loss of information needed to map a 3D scene onto a 2D image. The benefit of minimum hardware complication is compensated through the implementation of intricate computation approaches. For successful passive modeling, objects should be sufficiently textured to allow image feature detection (Bouguet).

Active Methods

Processing difficulties encountered in passive methods can be overcome by using specialized illumination sources and detectors that are implemented in active imaging techniques. Active systems use laser radar or one of the various forms of structured lighting. Structured light methods model an object by scanning the surface through a continuous emission and detection process. Triangulation is then applied to correlate the captured image coordinates with those of the real object. Unlike passive methods, active imaging requires an energy source to interact with the scanned object in order to highlight object features that can be detected by analyzing pixels of captured images (Trucco & Verri).

Hardware Components

The prototype built for ProRAST (see Figure 1) consists of the following devices:

· CEMAR Pico Laser Line Projector mounted on a clamp stand, and its laser line impinging on the test object

· Canon S30 Digital Camera fixed atop a tripod in a position and angle to capture shots of the test-object

· BiStep 2A Stepper Motor Controller

· A linear stage with a platform to displace the test-object in a single direction

· Stepper motor coupled to the linear stage via a spring-like joint

· DC Power Supply

· Logitech Web-Cam used to capture streaming images of the experiment

· USB Hub

· Server Machine with Internet Connection

Refer to Figure 2 for the setup model generated in Pro/Engineer 2001.

In order to control the experiment from a computer, it is essential to use devices that can be linked to a serial or USB port. Moreover, since the experiment is to be conducted remotely, we had to install devices that are controllable through code as the whole system is managed from a single application. Accordingly, control over the hardware (namely the stepper motor, digital camera and web cam) was achieved by invisibly launching the Command Line Prompt and sending it device control commands.

Figure 3. ProRAST Components

Figure 4. Isometric view of the hardware setup

[Top]

Linear stage

The linear stage used is a retired one obtained from the Precision Engineering Department at MIT. This particular stage was ideal for the range sensing application for it displayed a sufficient level of precision. It can be seen from the above figure that the stepper motor is coupled to the threaded shaft by a special joint. The rotation of the motor is translated to displacement of the platform along the stage. The platform is coupled to the threaded shaft via a screw bearing which makes this motion possible. In addition to the protective covering which prevents foreign particles from causing abrasion, lubrication of the threaded shaft with grease was necessary to ensure smooth operation. Another reason the linear stage is attractive for this application is the fact that displacement is restricted to only one direction and shaking in the other two directions is negligible.

Figure 5. Linear Stage

[Top]

Stepper Motor

Stepper motors are a special type of motor. Instead of rotating continuously, like DC motors, they move incrementally in steps. This type of motor was used because the application necessitates the capturing of images of test objects at fixed intervals. The stepper motor used here is a unipolar NEMA 23KSM motor rated at 7.8 volts and 0.9 amperes. This motor rotates in increments of 1.8^o and has a holding torque of 82 oz-in. Stepper motors may be interfaced with a controller allowing for accurate positioning and manipulation.

Figure 6. NEMA 23KSM Stepper Motor

Cemar Pico Laser

The CEMAR pico laser emitter projects a 5mWatt bright red laser line across sections of the test object. The laser is emitted from a single point. At its center, the line is crisp and uniform in intensity; however it does experience some fringing at the edges. In some cases this presented some inaccuracy. This may be remedied by using more than one of these emitters mounted side by side, to form a long continuous line across the object’s surface.

Figure 7. Laser components and laser scanning test object

[Top]

Digital camera

The digital camera used here is the Canon S30 PowerShot. The digital camera is also interfaced to the server. This permits control of the frequency of pictures taken and synchronization with the stepper motor motion. Images from the camera are directly streamed to the server and formatted accordingly.

Figure 8. Digital camera fixed on tripod

Stepper Motor Controller

The BiStep2A04 stepper controller is used to control the action of the stepper motor. The controller is interfaced to the server via the serial port allowing for complete control of the motors, including total monitoring of current conditions. The controller draws from the same 7.8 volt power supply to the stepper motors. The logic units on the controller tend to generate significant heat and are cooled with a heat sink and fan mounted on top of them.

Figure 9. BiStep2A04 Stepper Motor Controller

[Top]

Software Components

Several software components were developed to serve three main purposes: image processing and client-side interface, data storage, and remote lab access.

1- Image Processing and Display

A fundamental part of any range sensing setup requires efficient and accurate processing tools that handle image analysis, data storage and graphical representation. ProRAST .NET is the client application that we developed in order to handle all programming needs required to analyze images and provide a rich interface that enhances the user’s conception of the range sensing process. This section will first present the flowchart of the software architecture and then elaborate on the various interface controls that are embedded in ProRAST.

ProRAST .NET is designed to provide users total access to manipulate form controls as the scanning process progresses. This asynchronous behavior is represented by parallel processes initiated after ‘Image Loading’ in the processing flowchart in Figure 10. Image processing occurs instantaneously as images are captured and delivered to the client application (ProRAST .NET). To illustrate, images received by the client machine are saved in a temporary folder that will be deleted when the application is closed. A separate thread is launched to watch for new files, which are automatically delivered to the temporary folder. Once a new image is detected, a pixel by pixel analysis of the image is conducted, coordinate data corresponding to relevant pixels are stored and the new results are reflected in the project form display.

Figure 10. Analysis Flowchart

[Top]

Image Processing

Extensive implementation of the ‘GDI+’ class and its corresponding members in C# enables processing digital images. A thorough breakdown of a bitmap object provides information on the RGB (Red, Green and Blue) combination of each pixel. This approach is crucial for detecting the laser line projecting in each image. Moreover, several trials were carried out in order to determine the best performance of the system under varying conditions. Figure 11 and Figure 12 show sample camera images taken in darkness and with a camera flash respectively. A similar result is achieved for the two extremes of lighting conditions. The result is shown in the detected line plot in Figure 13. Detecting a laser line in an image is affected by the following criteria:

· Lighting conditions

· Object color and light reflection

· Image background colors

· Detection sensitivity to redness in pixel colors

As the first three factors listed above deal with real world conditions that relate to the scanned object, the fourth factor is solely dependant on algorithms that determine whether a pixel point belongs to a laser line or not.

Figure 11. Image captured in darkness

Figure 12. Image captured with flash mode on

Figure 13. Cross Section Plot

Each bitmap pixel is characterized by R (Red), G (Green) and B (Blue) parameters that describe the color of the pixel. The value of each of these parameters ranges between 0 and 128 based on the intensity of each of the three parameters. Moreover, pixels are conceived as belonging to the laser line only if their redness component exceeds a previously set standard. It is not enough to measure the red component only, as the red component alone is not fully representative of redness in a pixel color. Instead, all red green and blue components must be considered. The redness of a pixel is dependant on the gap between the R component and the sum of the values of the G and B components. The greater the gap, the closer the pixel color is to red.

ProRAST is built to process images of object surfaces that don’t include deep intrusions or cavities. This limitation is basically attributed to the fact that the laser line detection process assumes that no two pixels, located on the same vertical pixel column, can both belong to the laser line. Accordingly, the program detects the pixel with highest redness in each column of pixels. The x and y pixel coordinates of each accepted point are stored in an array for later conversion into real world coordinates.

[Top]

Control Rich Interface

As mentioned previously, ProRAST handles the processing of images in a separate thread which allows a user to interact with the form controls that reveal various aspects of the scanned object. A snapshot of the interface is provided in Appendix A. Following are the main features of the interface:

· Cross Section View

· 3D Model View

· Longitudinal Profile

· Profile Track Bar

· Image Track Bar

· Image List View

Captured images are loaded into the temporary folder with a sequential naming system. When a new image is spotted by the file listener, it is processed and its pixel data is added to two dimensional arrays that store the coordinates of all points that constitute successive profiles. At the end of the process, the ‘Captured Image’ window and the ‘Cross Section View’ window are updated with the last processed image and its detected laser line respectively. The functionalities of the other application controls are described below:

3D Model View: A three dimensional model of the scanned object is reconstructed by the 3D Graph control provided in the National Instruments Component Works Suite. 3D plotting is possible by providing the control with three matrixes that contain the X, Y and Z coordinates of every pixel point lying on a cross section of the object. For more information on data handling at run time and dimensioning of matrixes, refer to the ‘Data Structure’ section. The generated model is updated to reflect new sections as the scanning process progresses. Since image processing proceeds in a separate thread, users can zoom in and rotate the model, thus having a better conception of the modeled object.

Longitudinal Profile: The longitudinal profile represents the surface alignment of the scanned object in the direction parallel to the translational motion of the moving stage. Unlike the ‘Cross Section View’ which displays a plot of the object cross section at the laser plane, the longitudinal profile is generated through data manipulation rather than image processing. To illustrate, in order to determine the elevation (Z coordinate) of all points on a particular alignment specified by the ‘Profile Track Bar’, a search for point coordinates is performed on the data stored in the application memory.

Profile Track Bar: Embedded on the ‘Cross Section Plot’ is the ‘Profile Track Bar’ which denotes the alignment of a profile to be displayed in the ‘Longitudinal Profile Window’. The slider position on the bar corresponds to an alignment taken through the cross section point at that location.

Image Track Bar: The ‘Image Track Bar’ is displayed vertically on the right side of the main analysis window (refer to Appendix A). The track bar allows easy navigation through the various sections of the scanned object. All form contents are reloaded to match with the selected cross section.

Image List View: A list of the captured images is continuously updated with new images as they are loaded. For each image, the profile characteristics (average depth, minimum depth and maximum depth) are summarized. A checkbox, beside each cross section item in the list, enables selecting multiple items and then saving these images in a user specified directory.

[Top]

2- Data Structure

Project data constitutes of tri-axial coordinate values of all pixel points belonging to the detected profile lines. Data is stored in the real world coordinate form after transforming the image coordinate values into real world values. Data is filled into three matrixes where each one serves as a container for one of the X, Y or Z coordinate. Refer to Figure 14 for a graphical representation of a typical matrix used to store point coordinates.

Figure 14. Matrix Representation

ProRAST .NET provides the option of saving a project for later reference. Accordingly, the ‘Save Project’ command stores coordinate data (stored in the three coordinate matrixes) in one XML file. Moreover, digital images captured for profile detection are also transformed into a text format and stored in the same XML file. The schema of the file is based on four tables: one for each of the X, Y and Z coordinates, and one for storing the images in a string format. This utility provides the user with the convenience of having all project related information within one file instead of dealing with multiple image and text files.

If further analysis need be conducted on the data presented in the list view, a user can easily export the list view content to MS Excel by clicking the ‘Export to Excel’ command. This can be done when the scanning process is over, in case of an online experiment, or after an xml file has been loaded, in case of an offline loaded project.

[Top]

3- Remote Access and Control

Remote login is provided to allow users access to the experiment through the internet and therefore control the hardware, reset the system and perform a complete analysis without going to the lab.

The flowchart for ProRAST .NET is shown in Figure 15. Users can either work offline by loading a previously saved project (in an xml format), or connect to the lab server and conduct a real-time experiment. In either case, the same rich interface supports graphical representation and profile plotting.

All ProRAST .NET client applications connect to the lab server hosting one Web Service, which includes several WebMethods that handle all client–server communications. WebMethods can be categorized into three groups that perform the following functions:

Check the registration of new users:

The lab server keeps track of registered users that are capable of accessing the experiment by referring to a User Registration XML file. This file is used to validate a user’s name and password and store other user relevant information. The schema for the User Registration XML file is provided in Appendix B.

Check the lab status and hardware status:

In order to avoid multiple client access to the lab server and therefore interrupt a scanning process that has been already started, it is necessary to maintain the status of the login sessions in a separate XML file stored on the server. Refer to Appendix C for the schema of the Session XML file.

Before granting clients access to the lab experiment, the server runs two checks. First, the lab experiment must be available for a user; no one should be logged in to the lab server. This test is performed by checking the ‘InSession’ value stored in the Sessions XML file. ‘InSession’ is set to ‘true’ as long as a user is connected to the lab server. Another user cannot disrupt the connected user until the ‘InSession’ is ‘false’. Second, the experiment hardware should be turned on and ready to operate. This test is conducted through a WebMethod that detects any failure to communicate with the stepper motor controller, digital camera, and the Web Cam.

Control the hardware:

All hardware control functions are accessible through the Remote Control Form shown in figure 16. The Reset Motor command allows a user to adjust the platform location in case a connection failure occurred during the last session.

Figure 15. ProRAST .NET Flowchart

[Top]

The ‘Step & Shoot’ frame allows the user to capture images at specified locations by incrementally moving the object. The platform location with respect to the boundaries of the positioning stage is represented on a slider bar in the same frame. In addition, by turning on the Live Lab window, a user can instantaneously see the real time motion of the object and its location with respect to the mechanism boundaries and the laser line plane.

The ‘Batch Mode’ frame provides controls that would start a scanning process and therefore lunch the analysis environment, which is shown in Appendix A. The user is required to specify the number of cross sections and the spacing interval between two consecutive sections.

Figure 16. Remote Control Form

Control of the stepper motor is achieved using a console based application, called SerTest, developed for the BiStep2A04 controller. The application also allows a batch-based script to control the stepper motor. All data received by SerTest is echoed back to the command prompt, thus allowing the operator to see the response to any command (or set of commands). Command strings on SerTest are issued as follows:

SerTest text1 text2 text3 …

Where “text1”, “text2”, and “text3” are the actual strings sent to the controller. Each of these strings is a separate command. They are sent one at a time to the stepper motor on port COM1 and the completion of these commands is indicated with the return of a “*” character on the command line. For a complete listing of the commands, refer to Appendix D.

[Top]

Applications and Future Considerations

The lab prototype built for ProRAST is a static apparatus that may not be operated outside the scope of the lab; however, the developed system has laid the foundation for more practical sensors that can rely on ProRAST’s software architecture to handle data transfer and storage. The ProRAST project may be deployed in a number of civil engineering applications that require the detection of certain features of interest. One important feature of the ProRAST device is its non-invasive nature. This is attractive for applications that require detection of features without any physical contact.

Depending on the accuracy, efficiency, and reduction in size this range sensor can undergo, it can perhaps be mounted on a robot and used for clogging or scaling detection in sewer pipes. In this case, the projected light would be radial and would fall in a similar stripe-like manner on the pipe interior. The whole system, however, would be displacing forward through the pipe in this case, while the object remains in place. Figure 17 illustrates this concept for a sewer pipe being scanned for scaling along it walls. Here, the laser is mounted on a “black box” and is moved along the axis of the pipe using a threaded shaft mechanism, similar to the one used in the linear stage. There are certain issues involved in developing such a device, namely the power supply required on board and the remote communication in case streaming data is required. Additionally, a mobile version may be deployed in hazardous environments which are too dangerous for humans. A similar setup can also be used for crack detection in critical structures or surfaces.

Conclusion

The overall performance of the system is measured by its processing capability as well as its remote operation. ProRAST has been efficient in maintaining the range sensor experiment in operation mode without interruptions that would otherwise result from multiple web clients accessing the lab server at the same time. In addition, the implementation of a Web Service for trasferring client requests to the server and sending back text-formatted image data has been successfully established as no connection delays are encountered.

Despite its remoting effieciency, ProRAST still lacks a high level image processing algorithm that accounts for variations in intrinsic and extrinsic camera parameters. Moreover, a noise filter for laser line detection will provide more accurate depth values compared to the ones currently obtained using linear calibration. In addition, a newer version of ProRAST .NET would encorporate a calibration method similar to the one presented in the begining of the report. The point of calibration is to extract camera parameters that are dependent on the system setup. With these parameters, one may setup a transfer function to translate image coordinates to world coordinates. With the consideration of future updates, we designed ProRAST based on an easily upgradable architecture whereby revamped modules can be implemented in the client application to serve the needs of individual clients.

[Top]

Appendix

· Appendix A – ProRAST .NET Processing Interface

· Appendix B – User Registration Schema

· Appendix C – Session Schema

· Appendix D – BiStep2A04 Instruction Manual

Appendix A – ProRAST .NET Processing Interface

[Top]

Appendix B – User Registration Schema

Appendix C – Session Schema

Appendix D – BiStep2A04 Instruction Manual

(available at BiStep2A vendor’s website)

References

Bouguet, J. (1999). PhD thesis: Visual Methods for Three – Dimensional Modeling, California Institute of Technology, Pasadena.

Cureless, B.L. (1997). PhD thesis: New Methods for Surface Reconstruction from Range Imaging, Department of Electrical Engineering, Stanford University, California.

DePiero, F.W. and Trivedi, M.M. (1997). PhD thesis: 3-D Computer Vision Using Structured Light: Design, Calibration and Implementation Issues, Electrical and Computer Engineering Department, The University of Tennessee, Knoxville.

Trucco, E. and Verri, A. (2001). Introductory Techniques for 3D Computer Vision, Prentice-Hall, New York.

[Top]