Well, I disassembled everything again, and while inspecting the motors, I noticed that one of them was missing a C-clip.
Put it back, tested it with the APM and I was able to hover without modifying the PID's.
But I wanted to confirm if the problem was the HF Sport, so, put it back to the quad...