Introduction: Autism spectrum disorder (ASD) and Attention-Deficit Hyperactivity Disorder (ADHD) have been associated with facial affect recognition (FAR) alterations. Methods: This study examined accuracy and response times for general and specific FAR in whole face and eye-region stimuli. FAR was assessed in matched samples of children and adolescents with ASD (n = 35), ADHD (n = 32), and typical development (TD) (n = 32) aged 8.6-15.9 years (M = 11.6; SD = 2.0). Results: Compared to TD, the ASD group performed less accurate and showed longer response times for general and specific FAR, mostly driven by problems in neutral and happy face identification. The ADHD group responded faster than the ASD group for global FAR. No differences between ADHD and TD were found. Attentional distractibility had a significant effect on FAR performance in ASD and ADHD. Conclusions: Findings confirm FAR alterations in ASD, but not ADHD, and endorse effects of attentional distractibility on FAR in ASD and ADHD. FAR and attention function training is clinically meaningful in ASD. Future studies should include control for visual attention and facial configuration skills, use naturalistic FAR material and also investigate implicit FAR.