The ability to select, within the complexity of sensory input, the information most relevant for our purposes is influenced by both internal settings (i.e., top-down control) and salient features of external stimuli (i.e., bottom-up control). We here investigated using fMRI the neural underpinning of the interaction of top-down and bottom-up processes, as well as their effects on extrastriate areas processing visual stimuli in a category-selective fashion. We presented photos of bodies or buildings embedded into frequency-matched visual noise to the subjects. Stimulus saliency changed gradually due to an altered degree to which photos stood-out in relation to the surrounding noise (hence generating stronger bottom-up control signals). Top-down settings were manipulated via instruction. Participants were asked to attend one stimulus category (i.e., “is there a body?” or “is there a building?”). Highly salient stimuli that were inconsistent with participants’ attentional top-down template activated the inferior frontal junction and dorsal parietal regions bilaterally. Stimuli consistent with participants’ current attentional set additionally activated insular cortex and the parietal operculum. Furthermore, the extrastriate body area (EBA) exhibited increased neural activity when attention was directed to bodies. However, the latter effect was found only when stimuli were presented at intermediate saliency levels, thus suggesting a top-down modulation of this region only in the presence of weak bottom-up signals. Taken together, our results highlight the role of the inferior frontal junction and posterior parietal regions in integrating bottom-up and top-down attentional control signals.