Mitchell was twice married. His first marriage was to Mary Middleton Elwyn (1838–1862), a daughter of Dr. Alfred L. Elwyn of Philadelphia. Before her death, they were the parents of two children:

• John Kearsley Mitchell (1859–1917), a neurologist who married Anne Keppele Williams in 1890.
• Langdon Elwyn Mitchell (1862–1935), a playwright who married actress Marion Lea in 1891.