Initially, the AMA named their new school for Black Charlestonians in honor of abolitionist Lewis Tappan. Thomas Cardozo briefly served as the first principal, before his brother Francis replaced him. The Cardozo brothers were the sons of Lydia Williams, a free woman of color, and Isaac N. Cardozo, a Jewish businessman and employee of the United States Custom House in Charleston. Sent abroad to Great Britain for theological studies, Francis L. Cardozo returned to the United States in 1864 and became pastor of the Temple Street Congregational Church in New Haven, Connecticut. In August 1865, he replaced his brother as principal of Charleston’s new AMA school.
One of the most pressing problems Francis Cardozo faced in his new position was finding a permanent home for the school. In late 1865, General Rufus Saxton of the Freedmen’s Bureau offered the use of a school building on St. Philip Street. The AMA agreed and promptly renamed the school after Saxton. Soon after, however, local city leaders demanded the return of the building to its pre-Civil War owners. Fortunately, Cardozo learned that the AMA was expecting a pledge of $10,000 from the estate of Northern philanthropist Reverend Charles Avery to establish a college for Black students in Atlanta, Georgia. At Cardozo’s request, the AMA persuaded the Avery trustees to make a similar contribution to purchase a residence for teachers at 121 Bull Street in downtown Charleston, as well as an adjacent lot for the erection of a permanent school building. Built by the Freedmen’s Bureau for $17,000, the new school building opened on May 7, 1868 as the Avery Normal Institute. Later that same year, Cardozo left Avery to fill the office of South Carolina’s Secretary of State. Throughout the Reconstruction era, he served as an outspoken advocate for education.