Engagement matters a great deal, I find. Medium is as much a social platform as it is a writing platform and nobody wants to feel taken advantage of or that their interaction is one-sided.
I don't have a ton of followers (like 2.4k) but they're quality followers. And I don't follow many people, only those whose writing truly interests me.
It really is a kind of you get what you put in kind of place.