Infants with typical hearing show consistent patterns while scanning a speaker’s face, focusing on the eyes at 4 months and transitioning to equal looking time to the eyes and mouth by 12 months. This scanning of a speaker’s face in infancy assists in understanding of the speaker’s facial expressions and message. However, it is unknown how infant age or auditory status affects scanning of a speaker’s face. This study explored face scanning patterns in 6- and 18-month-old infants with typical hearing and infants and toddlers with hearing loss to determine the effect of chronologic age, sensory modality, and auditory status on face scanning patterns. Participants include two groups of infants with typical hearing (6 and 18 months of age) to compare to infants and toddlers with hearing loss. Participants with hearing loss ranged from 11 to 35 months and varied in device configuration and degree of hearing loss. Eye tracking technology was used to examine face scanning patterns to audiovisual and visual only stimuli. This study shows both 6- and 18-month-old infants with typical hearing spent more time viewing the speaker’s mouth while observing both audiovisual and visual only stimuli. Infants with hearing loss view a speaker’s mouth more than the eyes during audiovisual stimuli regardless of chronologic age, duration of experience with auditory technology, or device configuration. However, in the visual only condition, older infants with hearing loss who use cochlear implants spent a greater proportion of time viewing a speaker’s eyes, while all younger participants attended longer to a speaker’s mouth. In conclusion, similar looking patterns between 6- and 18-month old infants provide novel information on face scanning patterns and suggest a possible link among face scanning patterns, task complexity, and processing ability. Findings from the infants and toddlers with hearing loss help provide preliminary data to better understand early face scanning patterns and thus determine risk and protective factors associated with positive language development in this population.