Gojo’s worrying comes from a place of possession, as much as it shames him to admit (it doesn’t shame him at all, but he knows it should, and he’s a decent enough actor to play the part). You’re his student, one that he’s put a lot of stock into, even more so than Okkotsu or Itadori.
He had found you scared and panicking when your abilities had first manifested and suddenly the monsters lurking in the dark were thrown in front of your eyes, giving you no chance to process anything. You were well past the years when sorcerers first had this awakening - a semi-functioning adult with a stuffy office job and an even stuffier apartment - and it was always a bit harder for adults to accept that monsters were, in fact, very much real and walking amongst humans.
You weren’t a particularly strong sorcerer, barely on par with another grade 4, but Gojo didn’t think many people were particularly strong anyways. There was a few in very, very rare cases, but for the most part, Gojo saw them all as potential liabilities when it came down to a tough fight - he’d rather have them out of his way so he could go all out.
He had gotten you assigned under his care, showing you the ropes of this new world that terrified you so, and you had learned quite a bit so far. Not enough that would grant you the confidence to venture out on your own and away from him (because why would he willingly let a scared, clingy thing like you go that easily?), but enough so he wouldn’t have to worry about someone having to scrape your insides off the ground with a shovel.
You were well under his thumb when Itadori came into the picture, snug and comfortable no matter how much he mushed and mushed. When he told you to do something, you did, no matter if it was a request to fetch him a drink, or something more…intimate.
Itadori was no threat, of course, a teen who was too dumb and pure for the position he had out himself in, but Sukuna, that damned curse, was something close to it.
He liked you, craved you, wanted for you in the same way that Gojo had the moment he saw you sniveling and hiccuping in that alley. It was an all consuming want—the kind that would gnaw at you from the inside bit by bit, intestine by intestine, organ by organ, until there was nothing left but that damned craving. Gojo would have died if he hadn’t been able to have you. He would have combust and taken the world with him, he’s certain of it.
And now that he’s had you -in every way and position imaginable- he’ll be damned if he lets some curse who only gets 5 minutes of playtime take you away from him.