OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

           1,2Lixin Yang*            1Kailin Li*            1Xinyu Zhan*            1Fei Wu            1Anran Xu            1Liu Liu            1,2Cewu Lu
1Shanghai Jiao Tong University, 2Shanghai Qi Zhi Institute
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

Abstract

Learning how humans manipulate objects requires machines to acquire knowledge from two perspectives: one for understanding object affordances and the other for learning human's interactions based on the affordances. Even though these two knowledge bases are crucial, we find that current databases lack a comprehensive awareness of them.

In this work, we propose a multi-modal and rich-annotated knowledge repository, oakink, for visual and cognitive understanding of hand-object interactions. We start to collect 1,800 common household objects and annotate their affordances to construct the first knowledge base: oak. Given the affordance, we record rich human interactions with 100 selected objects in oak. Finally, we transfer the interactions on the 100 recorded objects to their virtual counterparts through a novel method: tink. The recorded and transferred hand-object interactions constitute the second knowledge base: ink.

As a result, oakink contains 50,000 distinct affordance-aware and intent-oriented hand-object interactions. We benchmark oakink on pose estimation and grasp generation tasks. Moreover, we propose two practical applications of oakink: intent-based interaction generation and handover generation.

Data

To download our dataset:

  1. Download the object models used in OakInk: obj_all.zip (108M). Download this file to a new folder and extract it with:

    unzip obj_all.zip
  2. Download the annotated files for OakInk-Image: anno.zip (33M). Download this file to a new folder and extract it with:

    unzip anno.zip
  3. Download the annotated files for OakInk-Shape: oakink_shape.zip (31M). Download this file to a new folder and extract it with:

    unzip oakink_shape.zip
  4. For easier downloading, we divided the image data into 20 parts. Please download the files to the same folder and unzip them individually:

We will release a well designed toolkit later.

Contact

Send any comments or questions to Lixin Yang: email.


Last updated on 2022/03/28