Implementing Vision Using a GAME BOY^® Camera on the MRM

1. Abstract

One of the factors that has limited the prevalence of vision systems in amateur robotics has been the lack of affordable processing power in a small package. Another has been the difficulty of obtaining a low-cost vision sensor that is easy to interface with a controller board.

This article describes how to implement a simple monochrome vision system on a robot powered by the Mini RoboMind (MRM). The MRM is a small, powerful, inexpensive 68332-based robot controller designed by Mark Castelluccio. The vision sensor is a CMOS image sensor taken from the GAME BOY^® Camera, an accessory for the GAME BOY^® handheld gaming device.

This article covers the basics of interfacing the image sensor with the controller board, and includes code samples for both the MRM and a host PC (connected via serial port) to allow captured images to be viewed on the host PC. Image processing techniques are beyond the scope of this article, but I've included a few pointers.

The host PC code for this article is written in Java^TM, and has been tested on Windows^® and Linux^® platforms. In theory, it should also run on any other operating system that supports Java^TM 2, as long as there is also a working implementation of the Java^TM COMM API available (the Java^TM COMM API is a platform-independent programming interface that Java^TM applications can use to access communication devices such as serial ports and parallel ports).

Figure 1. Lord Nelson - Mini Sumo Robot.

Figure 2. GAME BOY^® Camera.

Although it seems increasingly difficult to obtain the now obsolete GAME BOY^® Cameras, it is still possible to mail-order them from some retailers at the time of writing. Try navigating to MySimon.com, and enter GAME BOY Camera in the search field. There are also a few people selling them on the eBay auction site too.

// camera.c
// gbcam library usage example
//
// Copyright (C) 2001 Dafydd Walters
// dwalters@users.sourceforge.net
//
// Permission is hereby granted, free of charge, to use,
// modify and distribute this example source code without
// restriction.  Please note that such unrestricted
// permission does NOT apply to the MotoRobots libraries
// themselves.  The MotoRobots libraries are distributed
// under the terms of the GNU Lesser General Public License.
//
// THE AUTHORS AND THE CONTRIBUTORS TO THIS WORK
// DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE
//

#include <68332/qsm.h>
#include <68332/sim.h>
#include <68332/mrm/gbcam.h>

#define BAUD_RATE 19200

int main(void)
{
    char c = 0;
    int i;

    set_processor_speed(CLOCK16MHZ);
    sciinit(BAUD_RATE);
    gbcam_init();

    while (c != 'x')
    {
        if (havebyte())
        {
            // Receive byte from serial port
            c = inbyte();

            switch (c)
            {
                case 's' :  // Set exposure time

                    gbcam_exposure();
                    outbyte('s');
                    break;

                case 'p' :  // Take a picture

                    gbcam_shoot();
                    for (i = 0; i < gbcam_IMAGE_SIZE; i++)
                        outbyte(gbcam_imageBuffer[i]);
                    break;
            }
        }
    }

    // return control to CPU32BUG monitor
    return 0;
}

/*
 * GBCamView.java
 * gbcam library image viewer example for host PC
 *
 * Copyright (C) 2001 Dafydd Walters
 * dwalters@users.sourceforge.net
 *
 * Permission is hereby granted, free of charge, to use,
 * modify and distribute this example source code without
 * restriction.  Please note that such unrestricted
 * permission does NOT apply to the MotoRobots libraries
 * themselves.  The MotoRobots libraries are distributed
 * under the terms of the GNU Lesser General Public License.
 *
 * THE AUTHORS AND THE CONTRIBUTORS TO THIS WORK
 * DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE
 */

import javax.swing.*;
import java.awt.*;
import java.io.*;
import java.awt.event.*;
import javax.comm.*;
import java.util.*;

public class GBCamView extends JFrame {

  JPanel buttonPanel = new JPanel();
  JButton exposureButton = new JButton();
  JButton captureButton = new JButton();
  ImagePanel imagePanel = new ImagePanel();
  String serialPortName = null;
  SerialPort port = null;
  InputStream in = null;
  OutputStream out = null;


  public static void main(String[] args) {
    GBCamView viewer = new GBCamView();
    viewer.pack();
    viewer.show();
  }


  public GBCamView() {
    super("Camera Image Viewer");
    try {
      initGUI();
    }
    catch(Exception e) {
      e.printStackTrace();
      System.exit(1);
    }
    loadConfig();
    promptSerialPort();
    setupPort();
  }


  private void promptSerialPort() {

    Vector v = new Vector();

    // Obtain a list of serial ports
    Enumeration en = CommPortIdentifier.getPortIdentifiers();
    while(en.hasMoreElements()) {
      CommPortIdentifier cpi = (CommPortIdentifier) en.nextElement();
      if (cpi.getPortType() == CommPortIdentifier.PORT_SERIAL)
        v.addElement(cpi.getName());
    }

    if (v.size() == 0) {
      // No serial ports found
      JOptionPane.showMessageDialog(this,
        "No serial ports found", "Error", JOptionPane.ERROR_MESSAGE);
      System.exit(1);
    }

    serialPortName = (String) JOptionPane.showInputDialog(this,
      "Please confirm serial port selection", "Camera Image Viewer",
      JOptionPane.QUESTION_MESSAGE, null, v.toArray(), serialPortName);

    if (serialPortName == null)
      System.exit(1);
    else
      saveConfig();
  }


  private void setupPort() {

    try {

      CommPortIdentifier cpi = CommPortIdentifier.getPortIdentifier(
        serialPortName);

      port = (SerialPort) cpi.open("Camera Image Viewer", 3000);
      port.setSerialPortParams(19200,
        SerialPort.DATABITS_8,
        SerialPort.STOPBITS_1,
        SerialPort.PARITY_NONE);
      port.setFlowControlMode(SerialPort.FLOWCONTROL_NONE);
      port.disableReceiveThreshold();
      port.disableReceiveTimeout();
      in = port.getInputStream();
      out = port.getOutputStream();
    } catch (Exception e) {
      JOptionPane.showMessageDialog(this,
        "Unable to setup serial port: " + e, "Error",
        JOptionPane.ERROR_MESSAGE);
      System.exit(1);
    }
  }


  private void loadConfig() {
    Properties p = new Properties();
    try {
      p.load(new FileInputStream(System.getProperty("user.home") +
              System.getProperty("file.separator") + ".gbcamview.properties"));
      serialPortName =  p.getProperty("gbcamview.port");
    } catch (Exception ex) {
      serialPortName = null;
    }
  }


  private void saveConfig() {
    Properties p = new Properties();
    p.setProperty("gbcamview.port", serialPortName);
    try {
      p.store(new FileOutputStream(System.getProperty("user.home") +
              System.getProperty("file.separator") +
              ".gbcamview.properties"), "Camera Viewer Settings");
    } catch (Exception ex) {
      JOptionPane.showMessageDialog(this,
        "Unable to save settings", "Error", JOptionPane.ERROR_MESSAGE);
    }
  }


  private void initGUI() throws Exception {
    exposureButton.setText("Set Exposure");
    exposureButton.addActionListener(new java.awt.event.ActionListener() {
      public void actionPerformed(ActionEvent e) {
        exposureButton_actionPerformed(e);
      }
    });
    captureButton.setText("Capture");
    captureButton.addActionListener(new java.awt.event.ActionListener() {
      public void actionPerformed(ActionEvent e) {
        captureButton_actionPerformed(e);
      }
    });
    imagePanel.setBorder(BorderFactory.createEtchedBorder());
    imagePanel.setMinimumSize(new Dimension(266, 256));
    imagePanel.setPreferredSize(new Dimension(266, 256));
    this.addWindowListener(new java.awt.event.WindowAdapter() {
      public void windowClosing(WindowEvent e) {
        this_windowClosing(e);
      }
    });
    buttonPanel.add(captureButton, null);
    this.getContentPane().add(buttonPanel, BorderLayout.SOUTH);
    buttonPanel.add(exposureButton, null);
    this.getContentPane().add(imagePanel, BorderLayout.CENTER);
  }


  void exposureButton_actionPerformed(ActionEvent e) {
    try {
      // make sure input buffer is empty
      while (in.available() > 0)
        in.read();
      // transmit 's' command character to set exposure
      out.write('s');
      // wait for acknowledgement character
      in.read();
    } catch (IOException ex) {
      System.out.println("Exception occurred: " + ex);
    }
  }


  void captureButton_actionPerformed(ActionEvent e) {

    try {
      // make sure input buffer is empty
      while (in.available() > 0)
        in.read();
      // transmit 'p' command character to take a picture
      out.write('p');
      for (int y = 0; y < 123; y++) {
        for (int x = 0; x < 128; x++) {
          int pixel = in.read();
          if (pixel < 0 || pixel > 255)
            throw new Exception("Illegal pixel value");
          imagePanel.pixels[x][y] = pixel;
        }
      }
    } catch (Exception ex) {
      System.out.println("Exception occurred: " + ex);
    }
    // paint the image
    imagePanel.repaint();
  }


  void this_windowClosing(WindowEvent e) {
    // Window is closing, so end the application
    System.exit(0);
  }
}


// ImagePanel is a class that encapsulates the display of the image
// using an array of pixel intensities.
class ImagePanel extends JPanel {

  public int[][] pixels = new int[128][123];

  protected void paintComponent(Graphics g) {
    g.setColor(getBackground());
    g.fillRect(0, 0, getWidth(), getHeight());
    int ox = (getWidth() - 256) / 2;
    int oy = (getHeight() - 246) / 2;
    for (int y = 0; y < 123; y++) {
      for (int x = 0; x < 128; x++) {
        int pixel = pixels[x][y];
        g.setColor(new Color(pixel,pixel,pixel));
        g.fillRect(x*2 + ox, y*2 + oy, 2, 2);
      }
    }
  }
}

• Download the Java^TM Communications zip file, and extract to an empty directory
• Place the file win32com.dll in the <jdk>\jre\bin directory
• Place the file comm.jar in the <jdk>\jre\lib\ext directory
• Place the file javax.comm.properties in the <jdk>\jre\lib directory
• Do not alter the CLASSPATH environment variable
• To test the installation, open a command prompt window, change to the commapi\samples\BlackBox directory and run

  java -cp BlackBox.jar BlackBox

This assumes you are using Java^TM 2 SDK 1.2 or later. In the steps listed below, <jdk> refers to the root directory of your Java^TM SDK installation, so if you installed Java^TM SDK in /usr/java/jdk1.3.1_02 replace all reference to <jdk> with /usr/java/jdk1.3.1_02

• Download this i386 binary package: rxtx-bin-1.4.13.i386.tar.gz
• Extract the files from the archive with

  tar -zxf rxtx-bin-1.4.13.i386.tar.gz

  This will create a subdirectory rxtx-bin-1.4.13.i386 containing the extracted files.
• Download Sun's Java^TM Communications for Solaris
• Extract the files from the archive with

  tar -zxf <archive-name>.tar.Z

  This will create a subdirectory commapi containing the extracted files.

Log in as root to perform the remaining steps, and thereafter to run any COMM API programs (including GBCamView and BlackBox).

• Place all of the libSerial* and libParallel* files from RXTX in the <jdk>/jre/lib/i386 directory
• Place the file jcl.jar from RXTX in the <jdk>/jre/lib/ext directory
• Place the file javax.comm.properties from RXTX in the <jdk>/jre/lib directory
• Place the file comm.jar from commapi in the <jdk>/jre/lib/ext directory
• Do not alter the CLASSPATH environment variable
• To test the installation, open a command prompt window, change to the commapi/samples/BlackBox directory and run (as root):

  java -cp BlackBox.jar BlackBox

If you are feeling adventurous, you can build RXTX version 1.4.13 from the source code. Here is the full package: rxtx-1.4-13.tar.gz.

If you're feeling really adventurous and have plenty of time to waste, you could try to get one of the later versions of RXTX working. At the time of this writing, 1.5 was still experimental.

One final point. The serial port devices on some Linux distros (e.g. Red Hat 7.0 - 7.2) are owned by root and have permissions set so that ordinary users cannot open them. This is easy enough to change. To allow all users to access COM1 for example, enter the following (as root):

chmod 666 /dev/ttyS0

and for COM2:

chmod 666 /dev/ttyS1

Since you will be running the COMM API code as root anyway, this is probably moot, however.

• If you see an error such as this: "Exception in thread 'main' java.lang.NoClassDefFoundError: javax/comm/CommPort", this probably means that the COMM API files have not been installed in the correct directories. Carefully double-check the file locations described above. Beware that at the time of this writing, the documentation that comes with Java^TM Communications is a little out of date, and describes the installation steps and file locations for java 1.1 (which are different to Java^TM 2 SDK 1.2 and above, which I have described above).

• From Explorer, or My Computer, double-click on the icon for the host program file (GBCamView.jar) you downloaded

• Open a command prompt window
• Change to the directory where you downloaded the host program (GBCamView.jar)
• Run the host program as an executable jar file:

  java -jar GBCamView.jar

With the MRM hooked up to the host PC via the serial port, point the camera at a subject, then click the Set Exposure button on the GUI. By metering the brightness and contrast of the scene, this will automatically set the exposure time. This takes around 15 seconds. Try to keep the camera and subject fairly stationary during this exposure-setting time, as the algorithm takes multiple exposures. After setting the exposure time, clicking the Capture button will then take a picture, download it to the PC, and display it. The download takes about 10 seconds. It is not necessary to reset the exposure time between successive image captures, provided the lighting conditions remain similar.

Troubleshooting

• You may have problems running the host program if there is more than one version of Java^TM installed on your machine, and the version for which you installed COMM API is not the version you are using to run the host program. A symptom of incorrectly configured COMM API is an error issued by the host program as soon as it is launched. You may be surprised to find multiple versions of the Java^TM runtime installed on your system (e.g. IE or Netscape may have a Java^TM plug-in). If in doubt, search for the java executable on your hard drive (java.exe on Windows^® systems) - there are typically one or two instances of java.exe for each version of the Java^TM SDK or JRE. If you find multiple versions, you can follow the instructions above to install COMM API on the others as required.
• To make sure that the MRM is responding to commands from the host PC via the serial port, and that the correct serial port is selected, try opening a terminal session at 19200 BAUD (e.g. using HyperTerminal or Kermit), and issue the 's' and 'p' commands manually. You should see a single 's' echoed in response to the 's' command after a pause of about 15 seconds, and a large stream of seemingly random characters in response to the 'p' command.