some people only approach and understand polyamory as a weird kink and not like. regular ass family planning. financial interdependence, fair allocation of resources, sharing household labor, healthcare, child-rearing if that's in the cards for you.
there's nothing wrong with casual sex, i love casual sex. people get really stuck on the casual sex though. sometimes it's "sex is evil for (x) reason" that creates the stigma around polyamory. sometimes people jump in without know what they're doing and have no hard relationship skills, or notice the community here has the same problems the larger culture has.
it's not really a sex thing, dating more than one person. you don't even need to have sex with your partners to be polyam. some people just have different family structures. a system strictly structured around american individualism and the nuclear family enables abusers and alienation.