By the turn of the century, Charleston’s cotton factory building became a dreary symbol of industrial failure, an embarrassing episode in the city’s history that most white people preferred to forget. Affluent men who were asked to invest in civic projects often hesitated to commit funds to enterprises touted as fail-safe, preferring instead to put their money in projects, including textile mills, located elsewhere. This feeling intensified when Charleston's former cotton factory president J.H. Montgomery fell from a scaffold at the new mill near Gainesville, Georgia, and died in November 1902. Less than a year later, at the end of May 1903, a tornado hit the same factory, killing eighty workers and injuring many more. The public concluded that the whole enterprise had been somehow cursed. As the Evening Star in Washington D.C. put it, the “hoodoo” that plagued the cotton factory in Charleston followed the machinery to Gainesville. Despite these claims, this new plant, which included a model mill village, prospered and continued to produce cloth into the 1970s.

Elsewhere, the textile industry flourished, and more and more mill towns sprang up to house—and regulate—those who tended the increasingly automated rows of spindles and looms. Child labor laws eventually would mandate that the workers be adults: in 1938 Congress passed the Fair Labor Standards Act, which set minimum ages for employment. But until the 1960s, exclusionary racial protocols at southern mills held firm. Textile mills hired black workers only in the capacity of custodians or cotton cleaners. Other industries, including tobacco manufacturers, accepted black labor as long as the workers were segregated and blacks were relegated to the most onerous and menial jobs. In the textile factories, however, white resistance to black labor was fierce and enduring.           

The very week J.H. Montgomery died, the American Cigar Company proposed to take over the Charleston factory building rent-free for five years. It leased the building until 1912, when it completed the purchase. For the next seven decades the building would be used to manufacture tobacco products in Charleston, employing both black and white labor.