You're right purplepeaceday about this song being about his marriage gone wrong! At the end, he realizes he still misses her and wishes he could get back together and work things out somehow.
I like most of the lyrics in this song except the part about living with someone who was dealing with slaves but I am hoping Bob was talking about this guy who treated people he lived with like slaves and that led to his spirit-dying and his lady had to sell everything she owned just to pay the rent and so she became cold-hearted.
"When the bottom fell out, I became withdrawn" has to do with being at your wits end and just becoming more introverted due to feeling so down. That led him to missing his ex-wife.