"Yes, &" doesn't mean "Shut up and do it my way". It is not a weapon to be wielded by the first person who speaks.
"Yes, &" is about collaboration. That means that I need to give and take. For those of you not familiar with "give and take" it means you and I both have to "shut up" at some point during the scene and let our partner contribute too. It also means we must speak up and build on what has come before at some point too.
If my partner does an "&" that I did not expect, it is my job to go with it and "Yes, &" their idea. I shouldn't get mad, correct, or try to "fix" the scene. I NEED TO GO WITH IT. "Yes, &" is not a billy club, it is not first come first serve, it is not "Yes, & we will all do as you say." It's isn't even "Yes, I totally agree." "Yes, &" is magic. It is our way to discover the scene, environment and relationship TOGETHER! It is about us building and discovering together. Bit by bit. Giving and taking. Listening and accepting. Both embracing the unknown.