The objective of the study was to examine how social withdrawal in infants aged 12 months predicted emotional and behavioural problems at ages 3 and 5 years. The sample included 1,586 infants from the French Eden Mother-Child Cohort Study who had a measure of social withdrawal with the Alarm Distress BaBy scale at age 1 year; among these children, emotional and behavioural difficulties were rated by mothers using the Strength and Difficulty Questionnaire (SDQ) at 3 years for 1,257 (79 %) children and at 5 years for 1,123 (72 %) children. Social withdrawal behaviour at age 1 year was significantly associated with the SDQ behavioural disorder scale at 3 years, independently of a host of familial and child temperament confounders. The association with the relational disorder, prosocial and total difficulty scales was close to significance at 3 years after taking into account familial and temperament confounders. Social withdrawal significantly predicted the three aforementioned scales when measured at 5 years. No significant predictivity of the emotional scale and hyperactivity scale was detected at any age. This study made with a large longitudinal sample confirms the negative effects on development of social withdrawal behaviour, shedding light on the unfolding of behavioural disorders and relational difficulties in children; this calls for early detection of sustained social withdrawal behaviour, as it seems to hamper emotional development.