Those Who Did Not Fall in Battle: The Lee Family and Wartime Loss

William Henry Fitzhugh Lee, aka Fitzhugh, aka Rooney

By Colin Woodward

Wartime Loss

As the cliché goes, the Civil War was a conflict that pitted “brother against brother.” But that was not true for most American families, and it certainly wasn’t true for the Lee family. All of Lee’s brothers supported the Confederacy. His brother Smith served as a Confederate naval officer. All of Lee’s sons, furthermore, served in the Confederate military. The war, nevertheless, took an enormous toll–even on those who survived it with their lives and limbs intact.

One did not have to lose a son, brother, or husband in battle to experience tragedy during the Civil War. All three of Robert E. Lee’s sons survived the war. But the Lee family suffered tragedy all the same. The Lees lost their home Arlington early in the war to Federal troops. After four years of horrendous fighting, Robert E. Lee lost the war and his citizenship.

Long before the guns at Appomattox fell silent, General Lee lost a sister and daughter to illness as well as several grandchildren. Lee also nearly lost his son, William Henry Fitzhugh, in 1863. Fitzhugh languished for months in a Union prisoner of war camp before being released.


Called Fitzhugh for short and often referred to as simply “F” in his father’s letters, Fitzhugh also went by the nickname “Rooney.” He was Robert E. Lee’s second son (Custis was the oldest) and he served as a cavalryman during the war. Rooney rose to the rank of major general. Custis was a brigadier general, though most of his time in the army was not spent in field command. Robert, Jr., began the war as a private and ended the war as a major in charge of troops defending Richmond.

custis lee and taylor 1865
Custis Lee with his father and Walter Taylor in 1865, shortly after the war. Source: Library of Congress

Against the odds, all of Lee’s sons survived the war. Generals and other officers had a better chance of getting killed than privates did, but none of Lee’s sons suffered such a fate. Nevertheless, it was the rare officer who did not get killed, wounded, or captured, or suffer the loss of a loved one.

Fitzhugh was wounded in the fighting of June 9 at Brandy Station, the largest cavalry battle of the war. He was sent to Ashland, Virginia, to heal, but this did not stop Yankee troops from seizing him. Union officials eventually sent him to Fort Lafayette prison camp in New York, where he stayed until 1864.

This letter of June 1863 describes the circumstances surrounding Fitzhugh’s capture:

By then, the 26 year old Fitzhugh had already lost two young children to disease. While in captivity, he suffered another loss: his wife, Charlotte Georgiana Carter Wickham Lee, who was only 22. She died the day after Christmas in 1863. She is buried in Shockoe Cemetery in Richmond, where her grave has the epitaph, “How Sweet Her Memory.”

Interior of Arlington. From the Library of Congress online collections.

Ann Kinloch Lee Marshall, Unionist

Robert E. Lee survived the war. But as head of the South’s most aggressive army, he saw death all around him. Soldiers died by the thousands from battle and disease. But those behind the lines also suffered.

Lee’s older sister Ann died in 1864. Unlike Robert, she was supported the Union–the only one of Robert’s siblings to do so. She married Louis William Marshall, a judge and native of Maryland. When he resigned from the United States army, Robert E. Lee wrote to his sister Anne about his reasons for siding with Virginia in the secession crisis.

With all my devotion to the Union and the feeling of loyalty and duty of an American citizen, I have not been able to make up my mind to raise my hand against my relatives, my children, my home…I know you will blame me; but you must think as kindly of me as you can, and believe that I have endeavored to do what I thought right.

The war effectively ended their relationship. Anne died on 1864 February 20. Few of her letters survive and no photographs. Historians are not even sure where she died, though it most likely seems to have been Baltimore.

Annie Lee, the General’s Daughter

Probably the deepest loss General Lee suffered during the war was his daughter, Anne. She died of an illness in October of 1862 in North Carolina at the age of 23. She had moved to North Carolina to escape the ravages of the war in Virginia. Her death was a great blow to the family, but it happened at a time when Robert E. Lee had little time for mourning.

This may be the only photograph of Annie Lee (on right), pictured her with girls who are likely cousins from the Stuart family. If it is her, she is turned from the camera to hide the injury she sustained to her right eye as a little girl. Source: The Robert E. Lee Family Cooking and Housekeeping Book by Anne Carter Zimmer.

Over the course of the war, Robert E. Lee’s letters become increasingly fatalistic. The general never lost faith in the healing power of Christianity. In fact, the war strengthened his views that the dead were enjoying a better life in the hereafter.

annie lee
Annie Lee? There is reason to doubt that this photo, which has made the rounds on the internet, is her. As a girl, Annie lost one of her eyes in an accident with scissors. This photo bears no such disability.

The Lees also lost people close to them, such as William Orton Williams, who befriended Lee’s daughter Agnes. The controversial and perhaps mentally unstable Williams was not killed in battle but hanged in July 1863 for being a traitor. The incident, which happened a few days after Lee’s defeat at Gettysburg, further embittered General Lee against the Yankees.

By the end of the war, General Lee was a man without a home or a country. When he had heard of the death of A. P. Hill at Petersburg, he said, “He is now at rest, and we who are left are the ones to suffer.” That fairly well summed up the general’s view as well as that of many near and dear to him.



Author: leefamilydigitalarchive

The Lee Family Digital Archive is an online repository documenting the Lee family of Virginia. The editor of the LFDA is Dr. Colin Woodward, a historian of the South and published scholar. The site, which is free and open to the public, is located at The LFDA contains 4,000 letters, documents, books, legal papers, and references sources, covering more than 300 hundred years of Virginia and American history. The site is updated Monday-Friday and contains many items never before published. Please check us out on the web. Comments and suggestions are appreciated, and reference questions are promptly answered.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s