Knowledge about the factors that govern object representations in the brain is central to a systematic understanding of object recognition. To date, these factors and their interaction remain debated. Among many, three candidate factors have been prposed: 1) location invariance, 2) category preference and 3) location information. Location invariance designates the robustness of object representation to changes in the location in space. Category preference designates the preferential activation of cortical patches by a specific category of objects. Location information refers to information about where in space a particular object is located. To investigate the extent to which each of these factors governs object representation we used perceptual and imagery paradigms in combination with high-resolution fMRI and multivoxel pattern classification. In conjunction our results provide a systematic account of the principles related to object recognition in the ventral stream.