Three million dollars belonging to 61,000 African Americans. That's how much accumulated wealth vanished when the Freedman's Savings and Trust Company failed in June 1874. Earlier that year, Frederick Douglass had become the bank's President just after it moved its headquarters to a prominent location on the southeast corner of Lafayette Square, where the Treasury Annex now stands. The bank was failing due to disastrous investments into various industries and corporations and the subsequent cover-up of those faulty ventures. Douglass was unaware of the rampant corruption within the bank and, once he discovered how weak the institution was, he attempted to stabilize it by investing thousands of dollars of his own funds.

The Freedman's Bank building was an imposing structure that cost $260,000 to build. Describing his reaction to seeing the building for the first time, Frederick Douglass wrote, "The whole thing was beautiful. I had read of this bank when I lived in Rochester, and had indeed been solicited to become one of its trustees, and had reluctantly consented to do so: but when I came to Washington and saw its magnificent brown stone front, its towering height, its perfect appointments, and the fine display it made in the transaction of its business, I felt like the Queen of Sheba when she saw the riches of Solomon, that 'half had not been told me'." Many African Americans believed, like Douglass, that the bank was a foundation for new beginnings and opportunities to thrive economically, as well as socially and politically.