Robert has been married two times over the course of his life. His first marriage was in 1976, to actress and singer Diahnne Abbott. Throughout the '70s and '80s, Diahnne had a number of supporting parts in movies, including Robert vehicles like Taxi Driver and The King of Comedy. During their marriage, they had a son together named Raphael, who used to be an actor and now works in real estate. Robert also adopted Diahnne's daughter from a previous marriage. 

Robert's marriage with Diahnne ended in 1988, and his next came almost a decade later when he married actress Grace Hightower in 1997. They had a son together in 1998 and split up just a year later. Their divorce was never finalized, though, and they renewed their vows in 2004. After more than 20 years, though, it was reported in 2018 that they had split up again. Since then, Robert has been attending events solo.