In a hospital in Houston, two surgeons appear to be performing a difficult procedure on a cardiac patient. In fact, only one of the doctors in the room is real. The other is a replica-a lifelike physical model whose shape, appearance and movements precisely mimic those of a specialist in Tokyo who is performing the actual work.

This scenario may seem like science fiction, but research required to realize it has already begun, in a collaborative research project between Carnegie Mellon University and Intel. The goal of the project, which Intel has labeled Dynamic Physical Rendering (DPR), is to create a new form of media the researchers call pario-Greek for "to bring forth" or "to make."

What the researchers propose to make are moving, physical, three-dimensional replicas of people or objects, so lifelike that human senses would accept them as real. This would eliminate the need for cumbersome virtual reality gear and overcome the viewing angle limitations of modern 3D approaches. The replicas would mimic the shape and appearance of a person or object being imaged in real time, and as the originals moved, so would their replicas. These 3D models would be physical entities, not holograms. You could touch them and interact with them, just as if the originals were in the room with you.

When you finished using a replica for one purpose, you could transform it into another useful shape. A human replica could morph into a desk, a chair could become a keyboard, a lamp could be transformed into a ladder.

Replicas could be scaled up or down in size, to suit the needs of a particular application. A damaged nerve ending could be rendered at ten times its actual size, making it easier to repair. The Super Bowl could be replicated in miniature, with players two inches tall scrambling across a tabletop "field".

The DPR project was begun at Carnegie Mellon University, spearheaded by Seth Goldstein, an Associate Professor in the Computer Science Department. The project is the brainchild of Goldstein and Todd Mowry, Director of Intel Research Pittsburgh, who first discussed the idea at a conference in 2002. Mowry wanted to improve on two-dimensional videoconferencing, and Goldstein was interested in nanotechnology. They decided to merge their interests. They determined that, by taking advantage of advances in nanoscale assembly, they might create human replicas from ensembles of tiny computing devices that could sense, move, and change color and shape, enabling more realistic videoconferencing. The same meeting environment, with people and objects, could appear at each location, in real form or as replicas. A movement or interaction at any location would be reproduced at all of them. Every meeting could be face-to-face.

What began as a novel idea has evolved into an ambitious collaboration involving almost 30 researchers. Jason Campbell, a senior researcher at Intel Research Pittsburgh, is the Principal Investigator for the DPR project. Goldstein is leading the project for Carnegie Mellon, and Mowry provides additional leadership. The project is being funded by Intel, Carnegie Mellon University, the National Science Foundation, and the Defense Advanced Research Projects Agency (DARPA).

 
At a high level, there are two steps in dynamic physical rendering: capturing a moving, three-dimensional image and rendering it as a physical object. Much research has already been done in 3D motion capture, as illustrated in blockbuster movies like "Polar Express" and "Lord of the Rings." Researchers at Carnegie Mellon University also are exploring 3D image capture, in the Virtualized Reality project. They have developed technology that points a set of cameras at an event and enables the viewer to virtually fly around and watch the event from a variety of positions. The DPR researchers believe a similar approach could be used to capture 3D scenes for use in creating physical, moving 3D replicas.

The major challenge for the DPR researchers is how to reproduce images as physical, moving replicas. Replicas will be created from a form of programmable matter-a kind of high-tech modeling clay which the Carnegie Mellon collaborators have dubbed claytronics. Claytronics can be formed into different shapes, and it can change color, through light-emitting diodes on its surface. Embedded photo cells will enable it to sense light, so that a human replica can "see."



Figure 1. Creating a claytronics replica from a 3D image


Claytronics might even simulate the texture of the person or object being replicated. A replica will have computing capabilities, but these will be accessed through touch, voice, or another natural interface rather than a keyboard or mouse.

The basic unit of claytronics is what the Carnegie Mellon researchers refer to as a catom (claytronics atom). Replicas will be formed from ensembles of tiny catoms. Catoms will be as close to spherical as possible to support multiple packing densities.

An ensemble might contain millions or billions of catoms, which must coordinate and cooperate in order for the ensemble to function. So researchers must consider both the function of individual catoms and their behavior as part of an ensemble.

Each catom will have the minimum combination of computation and actuation needed to contribute to the ensemble. To support scaling, the researchers are looking for algorithms whose running times are proportional to the longest diagonal of the ensemble. If the algorithms require more running time than this, they will quickly become intractable. In addition to programming individual catoms, researchers are focusing on developing programming models which can facilitate the task of coordinating massive numbers of catoms, all of which are simultaneously executing within an ensemble.

From a hardware standpoint, researchers are striving to minimize the complexity of each catom. This will reduce its cost and improve manufacturability. And with fewer parts that could break, simple catoms will be more robust.

 
While catoms will be simple in design, each will have four capabilities:

Computation: Researchers believe that catoms could take advantage of existing microprocessor technology. Given that some modern microprocessor cores are now under a square millimeter, they believe that a reasonable amount of computational capacity should fit on the several square millimeters of surface area potentially available in a 2mm-diameter catom.

Motion: Although they will move, catoms will have no moving parts. This will enable them to form connections much more rapidly than traditional microrobots, and it will make them easier to manufacture in high volume. Catoms will bind to one another and move via electromagnetic or electrostatic forces, depending on the catom size.

Imagine a catom that is close to spherical in shape, and whose perimeter is covered by small electromagnets. A catom will move itself around by energizing a particular magnet and cooperating with a neighboring catom to do the same, drawing the pair together. If both catoms are free, they will spin equally about their axes, but if one catom is held rigid by links to its neighbors, the other will swing around the first, rolling across the fixed catom's surface and into a new position.

Electrostatic actuation will be required once catom sizes shrink to less than a millimeter or two. The process will be essentially the same, but rather than electromagnets, the perimeter of the catom will be covered with conductive plates. By selectively applying electric charges to the plates, each catom will be able to move relative to its neighbors.

Power: Catoms must be able to draw power without having to rely on a bulky battery or a wired connection. Under a novel resistor-network design the researchers have developed, only a few catoms must be connected in order for the entire ensemble to draw power. When connected catoms are energized, this triggers active routing algorithms which distribute power throughout the ensemble.

Communications: Communications is perhaps the biggest challenge that researchers face in designing catoms. An ensemble could contain millions or billions of catoms, and because of the way in which they pack, there could be as many as six axes of interconnection.

Another unique feature of catom networks is that catoms are homogeneous. Thus, unlike cell phones or other communications devices, the identity of an individual catom is sometimes (but not always) unimportant. An application is more likely to care about routing a message to the catoms comprising a specific physical part of an ensemble (for instance, the catoms comprising a "hand") rather than sending the same message to specific catoms based on their serial numbers. Furthermore, catoms may be in motion periodically, as the shape of the ensemble changes.

To address these challenges, researchers are investigating new routing techniques that focus on the location and function of the catoms at a given point in time. They are also focusing on building communications highways within ensembles, to limit the complexity of the routing problem.

 
The DPR researchers have made good progress in tackling the challenges of motion, power, and communications. And they have achieved an important milestone: the development of a working, two-dimensional catom prototype that is 44 millimeters (roughly two inches) in diameter (Figure 2). Now they are striving to make the hardware more reliable and easier to manufacture. Once they have refined the prototype, they will begin to experiment with shape control, communication and routing algorithms.

Figure 2. Prototype of a catom of programmable matter, 44 mm (roughly 2") in diameter. The perimeter is covered with 24 electromagnets which enable catoms to attract one another and move throughout an ensemble.

The hardware goal for the coming year is to develop catoms that are reasonably robust, and inexpensive enough that kits containing 100 or so catoms could be sold to other researchers who could perform their own experiments and explore new programming models and applications. The software goal is to develop infrastructure for creating simple but interesting software programs to control catom ensembles.

 
The potential applications of dynamic physical rendering are limited only by the imagination. Following are a few of the possibilities:

Medicine: A replica of your physician could appear in your living room and perform an exam. The virtual doctor would precisely mimic the shape, appearance and movements of your "real" doctor, who is performing the actual work from a remote office.

Disaster relief: Human replicas could serve as stand-ins for medical personnel, firefighters, or disaster relief workers. Objects made of programmable matter could be used to perform hazardous work and could morph into different shapes to serve multiple purposes. A fire hose could become a shovel, a ladder could be transformed into a stretcher.

Sports instruction: A renowned tennis teacher, golf instructor, or soccer coach could "appear" at clinics in multiple locations.

Entertainment: A football game, ice skating competition or other sporting event could be replicated in miniature on your coffee table. A movie could be recreated in your living room, and you could insert yourself into the role of one of the actors.

3D physical modeling: Physical replicas could replace 3D computer models, which can only be viewed in two dimensions and must be accessed through a keyboard and mouse. Using claytronics, you could reshape or resize a model car or home with your hands, as if you were working with modeling clay. As you manipulated the model directly, aided by embedded software that's similar to the drawing tools found in office software programs, the appropriate computations would be carried out automatically. You would not have to work at a computer at all; you would simply work with the model.

Using claytronics, multiple people at different locations could work on the same model. As a person at one location manipulated the model, it would be modified at every location.

Five years from now, the DPR researchers expect to have working ensembles of catoms that are close to spherical in shape. These catoms still will be large enough that no one will confuse a replica with the real thing (for that, catoms will probably have to shrink to less than a millimeter in diameter). But the catoms will be sufficiently robust that researchers can experiment with a variety of shapes, test hypotheses about ensemble behavior, and begin to envision where the technology might lead within a decade or two.

While the potential applications of dynamic physical rendering are exciting (see box), the work being done at Intel Research Pittsburgh and Carnegie Mellon University has broader implications. At its core, the research involves learning to design, power, program and control a densely packed set of microprocessors. These are similar to the key challenges facing the computer industry today. As a result, the DPR research is likely to produce new insights and technologies that could influence the future of computing and communications.

Once researchers have demonstrated the feasibility of the pario concept, this could inspire the same level of engineering effort that led to the development of tiny, inexpensive transistors. Researchers believe such an effort could produce equivalent results, one day making mass production of tiny, inexpensive catoms a reality.

If, in 1960, someone had suggested that one day you could buy a million transistors for a penny, the prediction would have seemed outlandish. But today Intel sells transistors for less than a micro cent, thanks to the continuing technology advances predicted by Moore's Law. It's not unreasonable to predict that one day far in the future, it may be possible to buy a million catoms for a penny.

But dynamic physical rendering could become viable long before Moore's Law drives down the cost of a catom to a micro cent. Even if catoms could be produced for a dollar each, some visualization applications might be economically viable. Certain other applications, such as programmable antennas, could be attractive even if a catom sold for tens or hundreds of dollars.

Whatever the cost, building catoms that are one millimeter in diameter-small enough to create convincing replicas-will be a difficult engineering challenge. But given current industry knowledge and the state of the art of silicon technology, it is not outside the realm of possibility. The challenge lies less in developing new technology than in bringing together a number of research areas in which the industry has made tremendous technical progress in the last decade. The Dynamic Physical Rendering project represents an important step in meeting that challenge.

 
The Challenge: Create physical, three-dimensional replicas of people or objects, so lifelike that human senses would accept them as real. This would eliminate the need for cumbersome virtual reality gear and overcome the viewing angle limitations of most existing 3D applications.

The Solution: In the Dynamic Physical Rendering (DPR) project, researchers at Intel Research Pittsburgh and Carnegie Mellon University are making good progress toward achieving this vision. There are two steps involved: capturing a 3D image and rendering it as a physical object. Much work has been done in 3D image capture. The major challenge is to develop realistic 3D replicas.

A form of programmable matter that Carnegie Mellon researchers call claytronics will be used to create replicas from 3D images. The basic unit of claytronics is what the Carnegie Mellon researchers refer to as a catom (claytronics atom). Replicas will be formed from ensembles of catoms. Catoms will have four capabilities: computation, motion, power, and communications.

Researchers have already created a prototype catom that is 44 millimeters in diameter. The goal is to eventually produce catoms that are one or two millimeters in diameter-small enough to produce convincing replicas.

Potential Impact: Claytronics could enable a wide range of novel applications. Human replicas could serve as stand-ins for medical personnel, firefighters, or disaster relief workers. A damaged nerve ending could be rendered at ten times its actual size, making it easier to repair. The Super Bowl could be replicated in miniature on a tabletop "field". Grandparents who live far away could "appear" in the home of their grandchildren and interact as though they were physically present. Replicas could supplement or replace 3D computer modeling. The list of potential applications is limited only by the imagination.

The DPR research has broader implications as well. At its core, the research involves learning to design, power, program and control a densely packed set of microprocessors. These are similar to the key challenges facing the computer industry today. As a result, the research is likely to produce new insights and technologies that could help to shape the future of computing and communications.