Drew Starkey Responds to Fans Shipping His Outer Banks Character Rafe with Madison Bailey’s Kiara: ‘Wait, What?’ (Exclusive).Linh

It’s the era of enemies-to-lovers, and some Outer Banks fans have been pining after a rather surprising couple whose storyline on the show would be exactly that trope — but the actors aren’t too sure about the idea.

In season 3 of the Netflix series, the interminably angry Rafe Cameron (Drew Starkey) and Kiara Carrera (Madison Bailey) became unlikely allies as they were both kidnapped and held hostage together by a fellow gold hunter. It was the first time the characters had spent any real time together onscreen, and some fans saw their tension as something that could have a romantic tinge to it — but Starkey and Bailey were not on the same wavelength.

“Madison and I were like, ‘Wait, what? No, that doesn’t make much sense,'” Starkey, 30, tells PEOPLE of his and his costar’s response to fans rooting for a romance between Rafe and Kiara.

“Yeah, I mean, we talked about this before. I think it’d be especially unfair to Kiara, you know?” he continues. “I think she deserves a little better than that.”

Chase Stokes, who plays Kiara’s lifelong friend John B. and the love interest to Rafe’s sister Sarah Cameron (Madelyn Cline), also weighed in, telling PEOPLE of the pairing, “It actually makes no sense in my brain.”

In season 4, Starkey says that his character has “found someone of his own, someone who makes him feel comfortable this season and who loves him for where he’s at, and I think that’s huge,” referring to Sofia (Fiona Palomo), who fans first met in season 3.

“I mean, I think he was in a place of drowning in pessimism about the world. He’s lost the only person that he’s really, really cared about and looked up to,” Starkey continues, referring to the death of Rafe and Sarah’s father Ward Cameron (Charles Esten) in season 3.

“And now, all of a sudden this person comes out and he can find comfort in another human being and be himself. So yeah, I’m happy for him.”

Aside from finally having somewhat of a stable partner for the first time in the upcoming season, Rafe remains plagued by his grief over his father’s death. Starkey says of playing into those emotions, “I think there’s something internal that’s always underneath the surface. It’s like a part of you is also lost when you lose someone that close to you.”

“I know in Rafe’s experience at least, I just think he’s kind of turned more towards apathy about the world,” he continues.

“I mean, he has a future in mind with reconnecting with his family and building his family, continuing his legacy, his family’s legacy, but that’s about it. But yeah, I mean, I think we all lose a part of ourselves in a grieving process.”