the idea that restrooms, locker rooms, etc need to be single-sex spaces in order for women to be safe is patriarchy's way of signalling to men & boys that society doesn't expect them to behave themselves around women. it is directly antifeminist. it would be antifeminist even if trans people did not exist. a feminist society would demand that women should be safe in all spaces even when there are men there.
btw this is maybe the single most key distinguishing feature of the terfy strains of radical feminism, the seed all the rest of it springs out of: they have absolutely no faith in the ability of feminism to actually destroy patriarchy. they do not think feminism can truly build a better world. they cannot really even imagine that possibility. they think patriarchy is an inevitable natural consequence of unchangeable biological facts, and therefore the goal of feminism can only be to mitigate the worst effects of patriarchy, not to get rid of it.
they can imagine a society where women get some designated safe spaces without men around. they cannot imagine a society where the presence of men is not inherently a danger to women.