Kody Brown Claims Estranged Daughter Maddie ‘Never’ Told Him She Was Pregnant

Kody Brown is opening up about his estrangement from some of his kids.

On a new episode of Sister Wives, Kody, 55, gets into a fight with his wife Robyn Brown, 45, as the two discuss their family’s estrangement. As Robyn tells her husband that his estrangement from his other kids is making her “lose respect” for him, Kody pushes back and claims that the blame is on his kids.

“You saw Madison take her kids and scuttle them away from me,” Kody says as the two stand outside in the snow. “She never told me she was pregnant. She never told me she was going to have a baby.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

The scene then shifts to a confessional with Janelle Brown, 55, one of Kody’s ex-wives, who shares daughter Madison with Kody. 

“The only thing the kids are upset with him [about] is the way he has treated his family,” Janelle explains. “The way he has completely like ditched out.” 

“And really, Maddie doesn’t have any contact with him,” she continues. “She is very protective of her children. Kody has not been there since Evie was born, and Evie is 3½.” 

“And she didn’t want him to just drop in and be like, ‘Oh, I’m your grandpa.’ And them be like, ‘What? Who is this guy?’ You know what I mean?”

In his confessional, Kody claims that it’s not a fair expectation to assume that grandparents can be in their family’s lives all the time.

“It’s [an] unrealistic expectation for grandparents to be in their grandchildren’s lives all the time, especially if you move your children to an entirely different coast. I have work and a life in Flagstaff,” Kody asserts.

The scene then switches back to Kody’s fight with Robyn. 

“They’re purposefully leaving me out of their lives to punish me for a crime I didn’t commit,” he laments. “I am only guilty of not falling madly in love with their mothers. Well, Madison actually called Janelle, saying ‘Dad said he didn’t love you.’” 

“I know, but couldn’t you like reach out to her?” Robyn asks. 

“Every time I talk to her, she spreads gossip about me to the rest of the family,” Kody alleges. 

Sister Wives airs Sundays at 10 p.m. ET on TLC.