Oh wait, I think I figured it out: you just have to reply to something other than the latest post
EDIT: yup, confirmed