B. Smith’s husband has come under fire for openly dating a white woman as his wife, a famed model and restaurateur, battles Alzheimer’s disease and has faced blistering backlash from fans.

And now, Dan Gasby tells PEOPLE that his wife told him to move on.

Surely she doesn’t remember saying that, right?

Blackface pictures from N.C. Gov. Roy Cooper’s college yearbook at UNC Chapel Hill

“B. told me to live, to go on, as I’ve told her,” explained Dan Gasby, who opened up about his relationship with his live-in girlfriend Alex Lerner in PEOPLE.

The three of them live together in the couple’s home in the Hamptons home when Lerner is visiting from Manhattan. But many have criticized Gasby and demanded he be arrested and claiming that he is abusing his wife.

But he counters: “B. asked me to go on with my life,” he says. “Anyone who says B. didn’t do that doesn’t know B. We wanted each other to be happy.”

The social media backlash has been harsh and swift from those condemning him for bringing another woman into the home while B. Smith suffers from the disease.

Prep school students clap back at NYC Mayor over blackface tweet

A number of people have come to Gasby’s defense and noted that he shouldn’t be judged because despite the outside relationship, he has still served as his wife’s caretaker.

“What we would like people to know is that when somebody has Alzheimer’s, how devastating it is, how 24/7 it is, how draining it is,” he says.

“You can be a caregiver and be loving to the patient and your loved one but you can still find some joy. It doesn’t need to be one or the other.”