Herschel and his wife of 40 years, Pam, are proud parents of two sons, Herschel IV (Kip) and Cole, and one daughter, Charlotte, who is an elementary school teacher in Alpharetta. Kip is an Air Force Captain and his wife, Missy, is a high school mathematics teacher. Cole is a former Navy Lieutenant who now works as an urban planner in Atlanta. His wife, Johanna, is an environmental engineer for the State of Georgia. 

Herschel and Pam have 3 grandchildren.