This still leaves some wiggle room. Note sharpobject’s summarizing of the ability as “when the last one is removed” which is agnostic about who does the removing. It’s possible that the bolded you in the fading help text is a use of the universal “you” to mean “anyone” like in a declarative “when you’re hungry, you eat.” …or it’s possible it really does mean “you” as in “you, the controller of the thing with fading.”
Another possibility is that it’s similar to the “Sac the Weak” ruling in free for all: it doesn’t give you any gold because it isn’t you who is destroying the units, but rather Sac the Weak forces your opponents to sacrifice the units themselves. In a similar way, fading could be a thing where you (the controller of the thing with fading) are always the person who is removing or adding time runes to things, even if you are being forced to by an effect controlled by your opponent.
I think it has to be clarified from on high either way. Personally, I really hope you can kill your opponent’s fading stuff by removing the last time rune from it.