Oh, by 'associated with semitic speakers' I'd assume a straightforward statistical correlation in living people, without necessarily inferring cause. I actually have no idea what sorts of languages were spoken in North Africa prior to the 7th century (other than guessing that Phoenician and Latin were probably popular amongst the elite at various points, for reasons quite unconnected to the longer-term origins of the bulk of the population).