In the lines before it is implicating that his family doesn't necessarily want the two together. Musically at this point in the song, all instrumentals drop except for the bass and a simple beat. This implies that not only would he give up all these possessions, he would give up his family as well.