Gilder was born on February 8, 1844 at Bordentown, New Jersey. He was the son of Jane (Nutt) Gilder and the Rev. William Henry Gilder, and educated at his father's seminary in Flushing, Queens. There he learned to set type and published the St. Thomas Register. Gilder later studied law at Philadelphia.