In recent studies, authors have revealed that robots can form concepts and understand the meaning of words through inference. The key idea behind these is “multimodal categorization” of robot's experience. However, previous studies considered only nonhierarchical categorization methods, which lead to nonhierarchical concept structure. Obviously, our concepts have hierarchical structure that makes our inference more efficient and accurate. In this paper, we propose a novel hierarchical categorization method. The method is an extension of multimodal latent Dirichlet allocation (MLDA) to hierarchical MLDA using nested Chinese restaurant process, which makes it possible for robots to acquire concepts in hierarchical structure. We show that a robot can from the hierarchical concept structure based on the multimodal information, which is captured by the robot itself. Moreover, by focusing on the common features of each category in the hierarchy, the robot is able to infer unobserved information including the meaning of words.