Graveyard reads, in part, “Whenever your non-token units die, bury them here.” The key word being “your.”
In this case, only units your opponent controls will be buried in the Graveyard when they die. Similarly, if you have a Graveyard and kidnap something that dies, it will be buried in your Graveyard.