I think your position is perfectly reasonable. I also think a system where someone’s hiring chances are effected by what former colleagues who didn’t even work closely with the person think of them is messed up.
You did the right thing, stay with your principles.
I had a situation where an employee and myself evaluated each other. I remained professional and neutral or positive. She, on the other hand, was quite negative. I have suffered because of this. I have tried several times to bad mouth her but just can’t bring myself to do so. It is a difficult situation.
Did you give just your professional opinion about this coworker? Did you keep the personal opinions out? That is the best way to do it.