Because the standard way of speaking English happens to be the way many white, educated, powerful people speak English, speakers who don’t lay claim to those identities or deviate from that way of talking are conceived of as having an accent.