You did not mention if you did a compass calibration. as for... "flyaways" that apparently Phantoms are plagued with".
Most of the time it's due to lack of compass calibration. And from what you described is just that. Also that plug you found loose is for the GPS module which is in the top...