User list - Show from group

We have several private/dormant groups we don't want to show up in the drop-down menu. How to exclude them or possibly removing the "Show from group" downmenu completely defaulting to All users without changing the php code?

It is ok to show these groups on the Administration side for user searches, but not on the public side.

Thanks.