I think you have to differentiate both the scenario based on additional property and thereby you can use that additional property in the WHEN rule. I have not tried this personally. But give me some more details if this is not getting solved.
I think it would be good to by-pass the party addition step for Unfollow operation. Is it possible to find out if there is any specific parameter being passed for Unfollow operation? You can use that value to bypass or restrict the creation of work party.