Learning Physically-Instantiated Game Play Through Visual Observation

Home Research ICRA 2010 Scheme→C Hurd N900 Code Reading CV

Andrei Barbu, Siddharth Narayanaswamy, and Jeffrey Mark Siskind

Proceedings of 2010 IEEE International Conference on Robotics and Automation

The paper is available here

Learned rules

Tic-Tac-Toe (text)
Hexapawn (text)
Variant A (text)
Variant B (text)
Variant C (text)
Variant D (text)


These are unedited screendumps of the application while playing a training set, learning the rules for each of the six games, and then playing one valid game with the generated rules. Note the binocular camera output, the automatic detection of the board, updates of the board state and the printout of the rules.

Variant A
Variant B
Variant C
Variant D Training
Variant D Play

Background knowledge

Common (text)
Initial board (text)
Legal move generator (text)
Outcome predicate (text)


Running this software requires Scheme->C and associated infrastructure, QobiScheme, SWI Prolog, and CProgol 4.4, and Matlab R2008a. A 32-bit binary version of OpenCV is included in this package, as due to an OpenCV bug a more recent version than generally available in most distributions is required. 32-bit binaries are provided for SWI Prolog and CProgol as well. This software does run correctly on 64-bit machines as long as 64-bit libraries are provided.

The software can be found here.

Technical drawings/descriptions

The robot is composed of Lynxmotion, Hitec and several custom parts.

The arm base construction process is documented here. Note that we use only one servo, a Hitec HS-5995TG, hence we replace the ASB-13 bracket with an ASB-04B; we weld the L bracket to the base base instead of screwing it in for a stronger joint that vibrates less. The lower (shoulder), middle (elbow) and front (hand), servos are connected similarly to the Lynxmotion SES arm. We use different servos than the SES arm; the shoulder is a Hitec HS-985MG, the elbow is a Hitec HS-645MG, the hand and fingers are Hitec HS-5475. These servos have been carefully chosen to minimize the amount of vibration, and to allow the two most taxed ones to relax by having analog servos installed in those positions. We use different link lengths (AT-05 for both), and use a different spring configuration (we add one spring attached to the lower back servo mount screw and the hub screw at the top of the shoulder C bracket). The gripper is a custom design. To the end of the wrist L bracket (part of the standard joint configuration), we connect a C bracket, ASB-09B, on to which we mount two multi-purpose brackets, ASB-04B. On each of these we mount one servo, Hitec HS-5475, one offset bracket, ASB-11B, one hub, HUB-08, one tube, AT-01. We cover the grippers using a rubber end cap, REC-06. The servos are powered by an adjustable Mean Well SP-320-5 at 7 volts.

The camera mount is built on top of a MAB-01, with an aluminum replacement milled multipurpose bracket, and a welded instead of screwed L bracket joint. Two ASB-516B are joined and connected to the base, at the 4th hole. At the other end, leaving two holes empty, one hub, HUB-08, is attached, along with 3 tubes, AT-06s, connected by end-to-end hubs. The tubes can, optionally, be machine pressed to fit more tightly onto the hubs so as to reduce vibration. Similarly one more tube, AT-01 is mounted on top. One channel, ASB-507B, is mounted on top of this assembly with one hub, HUB-08, perpendicular to the camera mount. At each end of the top of this channel one C bracket, ASB-09B, is mounted, bent to roughly 15-20 degrees so that the cameras will face the playing surface. Two Logitech QuickCam Orbits, with their base disassembled, are mounted to the top of these brackets. The mount has one custom plastic connector, to which the USB connector from the Logitech base is glued; the connector then is screwed into the side of ASB-09B bracket, held in place by one screw which goes through both the custom connector and the USB connector. The previous operation requires the removal of several small pieces of plastic from the sides of the Logitech USB connector. The camera then slides into the standard Logitech connector, now attached to the mount, requiring no modifications to the camera eye itself and allowing the cameras to be swapped. This setup also provides for optional stereo sound recording via a microphone built-in to each Logitech camera base.

The robot is controlled via a Lynxmotion SSC-32 controller, which comes with both ample documentation and source code.

A list of required Lynxmotion parts is available here

The staging areas are built out of several pieces of glued plywood, to these specifications.

The housing is built of out two identical surfaces, connected and build to these specifications, the bases are screwed in to these specifications.

The game hardware is available from amazon.