Our country was founded and built on the back of slavery. Our founders were colonizers, assuming and asserting that the white men should naturally be in charge. Only white men could vote, hold government office, own property, and all non-white men (and all women) were assumed to be subservient - in the minds of the white men as well as those being repressed.
The 3/5 compromise and the electoral college was enshrined in our country's Constitution with that in mind - the white men would represent the "others" in their states and districts, and enslaved blacks would count as 3/5 of a person when it came to allocating the number of representatives the states would send to the House (and thus the number of electors). This is because in some areas the slaves outnumbered the white owners.