Phonemes and prosodic contours are fundamental elements of speech used to convey complementary meanings. Perceiving these elements requires mapping variable acoustic cues onto discrete categories along ventral and dorsal speech streams. While traditional models make clear predictions, exactly where and when this acoustic-to-categorical mapping occurs remains unclear. Using magnetoencephalography and behavioural psychophysics, combined with time-resolved representational similarity and multivariate transfer entropy analyses, we show how phonemes and prosody propagate along the dual streams and how their categorical representations are gradually formed. Contrary to theoretical predictions, acoustic and categorical representations occur in parallel, rather than serially, across time and space for both elements. Moreover, prosody categories extend further along both streams than phoneme categories, with differently weighted contributions of posterior temporal areas. These results highlight a shared principle of parallel acoustic and categorical processing, yet partially distinct abstraction mechanisms for phonemes and prosody, key to access the multilayered meaning of speech.