Raw material selection for object construction

Abstract

Agents, such as assembly robots, are typically incapable of building objects without a predefined goal or predefined set of materials. Extending the construction capabilities of agents to objects and materials that an agent has not seen before would therefore greatly improve the scope of objects that agents can construct. An important step in building novel objects is the ability to recognise combinations of raw materials which are likely to be useful. As a step toward automating this step, we aim to exploit the intuition that the visual characteristics of candidate raw materials provide useful cues to their potential combinations. Toward this end, we present a Siamese neural network based model that is able to recognise unseen raw materials present in objects given a list of candidate material images. We demonstrate the utility and efficacy of our model within two domains. The first is a single material selection domain that uses the ShapeNet 3D model dataset where we attempt to recover the materials present in a model. The second is a multiple material domain using the adventure game Minecraftwherewepredictthecombinationsofmaterialsthatwillresultinatargettool. Weempirically demonstrate that by recognising the visual similarities between objects and materials our model is able to learn from a subset of object material pairs and generalise to unseen objects, materials and texture packs. We perform such tests by showing that our model outperforms chance and baseline methods.