just a wild guess here, but,
when you did this, was the copter in the same place as you initialized it, when you activated the FS/ret home function? as in, it has to be away from home to know it has to return home. if it was still sitting where you set as home, then just shutting the motors...