Neuroimaging studies suggest that understanding emotions in others engages brain regions partially common to those associated with more general cognitive Theory-of-Mind (ToM) functions allowing us to infer people’s beliefs or intentions. However, neuropsychological studies on brain-damaged patients reveal dissociations between the ability to understand others’ emotions and ToM. This discrepancy might underlie the fact that neuropsychological investigations often correlate behavioural impairments only to the lesion site, without considering the impact that the insult might have on other interconnected brain structures. Here we took a network-based approach, and investigated whether deficits in understanding people’s emotional and cognitive states relate to damage to similar or differential structures. By combining information from 40 unilateral stroke damaged patients, with normative connectome data from 92 neurotypical individuals, we estimated lesion-induced dysfunctions across the whole brain, and modeled them in relation to patients’ behavior. We found a striking dissociation between networks centered in the insular and prefrontal cortex, whose dysfunctions led to selective impairments in understanding emotions and beliefs respectively. Instead, no evidence was observed for neural structures shared between the two conditions. Overall, our data provide novel evidence of segregation between brain networks subserving social inferential abilities.