He used that money later to buy Manhattan real estate, the basis of the family’s later fortune. In 1832 J. Joseph Henry built the Federal-style townhouse later known as the J.J. Henry House.