In 1821 Greene County, Illinois, was organized and the newly platted town of Carrollton became its seat of government. That same year, J.W. Skidmore, merchant, erected a tow-room building on the northwest corner of the public square. In December, 1821, the county commissioners allowed that he be paid $4.00 for the use of his room for holding court. Samuel Lee purchased the building and the two lots with the intention of adding on it to the south. One of its most esteemed men, Lee served the county in its early years as county clerk and recorder, circuit clerk and justice of the peace. He married 16-year-old Mary Ann Faust, sister-in-law of Mr. Skidmore, in 1824. Before his grand house could be completed, Samuel Lee died in September, 1829.

Lee's will directed that "the brick mansion house I have recently commenced in Carrollton shall be finished to the use of my wife Mary Ann Lee." His plans were carried out to his specifications, possibly by Moses Stevens, builder of the second county courthouse across the street in 1832. The one-and-a-half story house in the Federal style which resulted became the home of Lee's widow and their two small children. On April 27, 1831, Mrs. Lee married Edward Baker, a young lawyer, who later became a close friend of Abraham Lincoln in Springfield, where the Bakers moved in August, 1835. The next year they sold their "brick mansion house", as described in the deed, to Orange Heaton. Moving to California in 1852 and then to the new state of Oregon in 1860, Edward Baker was elected U.S. Senator.

At Lincoln's inauguration in March, 1861, Baker, known for his oratory skills, made the shortest public speech of his career with these words: "Fellow citizens, I introduce to you Abraham Lincoln, the President-elect of the United States." With the outbreak of the Civil War, he organized the California Regiment and was commissioned its Colonel. He was killed October 21, 1861 during the battle of Ball's Run, Virginia.

Charles Drury Hodges bought the Lee-Baker mansion in March, 1850 from Orange Heaton. A lawyer and merchant, Hodges served as county judge, circuit judge, representative in the U.S. Congress and Illinois state senator during his long public service career. About 1854 Judge Hodges added a second story to the east part of the original Lee house, incorporating it into a two-story Italianate style wing. With its wide front veranda and its yard enclosed with a wrought iron fence (both now gone), the imposing Victorian structure was a Carrollton showpiece.

After Judge Hodges' death in 1884, his family continued to make it their home until Mrs. Hodges died in 1899. The following year their son Beverly C. Hodges converted the family home into the Hodges Office Building. He ended the 70-year ownership of the house by the family when he deeded the property to Dr. N.D. Vedder in 1921. For most of the 20th century this 19th century house has been host to numerous businesses and professional services, including lawyers, dentists, doctors, and insurance agents.

A gradual deterioration began to plague the old brick edifice until 1970 its very existence was threatened. In 1975 Attorney William Vogt and then the Greene County National Bank bought and transferred the historic house to the Greene County Historical and Genealogical Society in 1982. Efforts begun in 1978 to place the house on the National Register of Historic Places were renewed in on November 3, 1980 when such designation was bestowed. The society is a volunteer, no-profit organization dedicated to preserving some of the past of Greene County in its museum, its family files, and its Brick Mansion House.

(Based on research materials and writings of Fern Bruner and Judge Jack A. Alfeld.)