Benjamin Butler

Benjamin Franklin Butler (November 5, 1818 – January 11, 1893) was an American major general of the Union Army, politician, lawyer, and businessman from Massachusetts. Born in New Hampshire and raised in Lowell, Massachusetts, Butler was a political major general of the Union Army during the American Civil War and had a leadership role in the impeachment of U.S. president Andrew Johnson. He was a colorful and often controversial figure on the national stage and on the Massachusetts political scene, serving five terms in the U.S. House of Representatives and running several campaigns for governor before his election to that office in 1882.

Butler, a successful trial lawyer, served in the Massachusetts legislature as an antiwar Democrat and as an officer in the state militia. Early in the Civil War he joined the Union Army, where he first gained renown when he refused to return escaped slaves, designating them as contraband of war, an idea that the Lincoln administration endorsed and that played a role in making emancipation an official war goal. Later in the war, he was noted for his questionable military skills and his controversial command of New Orleans, which made him widely disliked in the South and earned him the "Beast" epithet. His commands were marred by financial and logistical dealings across enemy lines, some of which may have taken place with his knowledge and to his financial benefit.

At the request of General-in-Chief Ulysses S. Grant, President Abraham Lincoln relieved Butler from the posts he held in the Union Army after his failure in the First Battle of Fort Fisher, but he soon won election to the United States House of Representatives from Massachusetts. As a Radical Republican he considered President Johnson's Reconstruction agenda to be too weak, and he advocated harsher punishments of former Confederate leadership and stronger stances on civil rights reform. He was also an early proponent of impeaching Johnson. After Johnson was impeached in early 1868, Butler served as the lead prosecutor among the House-appointed impeachment managers in the Senate impeachment trial proceedings. Additionally, as Chairman of the House Committee on Reconstruction, Butler authored the Ku Klux Klan Act of 1871 and coauthored the landmark Civil Rights Act of 1875.

In Massachusetts, Butler was often at odds with more conservative members of the political establishment over matters of both style and substance. Feuds with Republican politicians led to his being denied several nominations for the governorship between 1858 and 1880. Returning to the Democratic fold, he won the governorship in the 1882 election with Democratic and Greenback Party support. He ran for president on the Greenback Party and the Anti-Monopoly Party tickets in 1884, having unsuccessfully sought the Democratic nomination as well.

Early years

Benjamin Franklin Butler was born in Deerfield, New Hampshire, the sixth and youngest child of John Butler and Charlotte Ellison Butler. His father served under General Andrew Jackson at the Battle of New Orleans during the War of 1812 and later became a privateer, dying of yellow fever in the West Indies not long after Benjamin was born.West (1965), pp. 8–9 He was named after Founding Father Benjamin Franklin. His elder brother, Andrew Jackson Butler (1815–1864), served as a colonel in the Union Army during the Civil War and joined him in New Orleans. Butler's mother was a devout Baptist who encouraged him to read the Bible and prepare for the ministry. In 1827, at the age of nine, Butler was awarded a scholarship to Phillips Exeter Academy, where he spent one term. He was described by a schoolmate as "a reckless, impetuous, headstrong boy", and regularly got into fights.

Butler's mother moved the family in 1828 to Lowell, Massachusetts, where she operated a boarding house for workers at the textile mills. He attended the public schools there, from which he was almost expelled for fighting, the principal describing him as a boy who "might be led, but could not be driven." He attended Waterville (now Colby) College in pursuit of his mother's wish that he prepare for the ministry, but eventually rebelled against the idea. In 1836, Butler sought permission to go instead to West Point for a military education, but he did not receive one of the few places available. He continued his studies at Waterville, where he sharpened his rhetorical skills in theological discussions and began to adopt Democratic Party political views. He graduated in August 1838. Butler returned to Lowell, where he clerked and read law as an apprentice with a local lawyer. He was admitted to the Massachusetts bar in 1840 and opened a practice in Lowell.West (1965), pp. 17–23

After an extended courtship, Butler married Sarah Hildreth, a stage actress and daughter of Dr. Israel Hildreth of Lowell, on May 16, 1844. They had four children: Paul (1845–1850), Blanche (1847–1939), Paul (1852–1918) and Ben-Israel (1855–1881). Butler's business partners included Sarah's brother Fisher and her brother-in-law, W. P. Webster.

In 1844, Butler was elected a member of the American Philosophical Society.

Law and early business dealings

Butler quickly gained a reputation as a dogged criminal defense lawyer who seized on every misstep of his opposition to gain victories for his clients, and also became a specialist in bankruptcy law. His trial work was so successful that it received regular press coverage, and he was able to expand his practice into Boston. George Riley worked at his Boston law office.

Butler's success as a lawyer enabled him to purchase shares in Lowell's Middlesex Mill Company when they were cheap.Hearn (2000), p. 19 Although he generally represented workers in legal actions, he also sometimes represented mill owners. When he became more politically active, he advocated the passage of a law establishing a ten-hour day for laborers, but he also opposed labor strikes over the matter. He instituted a ten-hour work day at the Middlesex Mills.Quarstein (2011), p. 29

Pre-Civil War political career

During the debates over the ten-hour day a Whig-supporting Lowell newspaper published a verse suggesting that Butler's father had been hanged for piracy. Butler sued the paper's editor and publisher for that and other allegations that had been printed about himself. The editor was convicted and fined $50, but the publisher was acquitted on a technicality. Butler blamed the Whig judge, Ebenezer Rockwood Hoar, for the acquittal, inaugurating a feud between the two that would last for decades and significantly color Butler's reputation in the state.

Butler, as a Democrat, supported the Compromise of 1850 and regularly spoke out against the abolition of slavery. At the state level, he supported the coalition of Democrats and Free Soilers that elected George S. Boutwell governor in 1851. This garnered him enough support to win election to the state legislature in 1852. His support for Franklin Pierce as president, however, cost him the seat the next year. He was elected a delegate to the 1853 state constitutional convention with strong Catholic support, and was elected to the state senate in 1858, a year dominated by Republican victories in the state. Butler was nominated for governor in 1859 and ran on a pro-slavery, pro-tariff platform. He lost to incumbent Republican Nathaniel Prentice Banks.

In the 1860 Democratic National Convention at Charleston, South Carolina, Butler initially supported John C. Breckinridge for president but then shifted his support to Jefferson Davis, believing that only a moderate Southerner could keep the Democratic party from dividing. A conversation he had with Davis prior to the convention convinced him that Davis might be such a man, and he gave him his support before the convention split over slavery. Butler ended up supporting Breckinridge over Douglas against state party instructions, ruining his standing with the state party apparatus. He was nominated for governor in the 1860 election by a Breckinridge splinter of the state party, but trailed far behind other candidates.

Civil War

Although he sympathized with the South, Butler stated, "I was always a friend of southern rights but an enemy of southern wrongs" and sought to serve in the Union Army. His military career before the Civil War began as a private in the Lowell militia in 1840. Butler eventually rose to become colonel of a regiment of primarily Irish American men. In 1855, the nativist Know Nothing governor Henry J. Gardner disbanded Butler's militia, but Butler was elected brigadier general after the militia was reorganized. In 1857 Secretary of War Jefferson Davis appointed him to the Board of Visitors of West Point. These positions did not give him any significant military experience.

1860

After Abraham Lincoln was elected president in November 1860, Butler traveled to Washington, D.C. When a secessionist South Carolina delegation arrived there he recommended to lameduck President James Buchanan that they be arrested and charged with treason. Buchanan rejected the idea. Butler also met with Jefferson Davis and learned that he was not the Union man that Butler had thought he was. Butler then returned to Massachusetts, where he warned Governor John A. Andrew that hostilities were likely and that the state militia should be readied. He took advantage of the mobilization to secure a contract with the state for his mill to supply heavy cloth to the militia. Military contracts would constitute a significant source of profits for Butler's mill throughout the war.Hearn (2000), p. 24

Petitioning for military leadership appointment

Butler also worked to secure a leadership position should the militia be deployed. He first offered his services to Governor Andrew in March 1861. When the call for militia finally arrived in April, Massachusetts was asked for only three regiments, but Butler managed to have the request expanded to include a brigadier general. He telegraphed Secretary of War Simon Cameron, with whom he was acquainted, suggesting that Cameron issue a request for a brigadier and general staff from Massachusetts, which soon afterward appeared on Governor Andrew's desk. He then used banking contacts to ensure that loans that would be needed to fund the militia operations would be conditioned on his appointment. Despite Andrew's desire to assign the brigadier position to Ebenezer Peirce, the bank insisted on Butler, and he was sent south to ensure the security of transportation routes to Washington.Hearn (2000), p. 25 The nation's capital was threatened with isolation from free states because it was unclear whether Maryland, a slave state, would also secede.Wells (2011), p. 34

1861: Baltimore and Virginia operations

The two regiments Massachusetts sent to Maryland were the 6th and 8th Volunteer Militia. The 6th departed first and was caught up in a secessionist riot in Baltimore, Maryland on April 19. Butler traveled with the 8th, which left Philadelphia the next day amid news that railroad connections around Baltimore were being severed. Butler and the 8th traveled by rail and ferry to Maryland's capital, Annapolis, where Governor Thomas H. Hicks attempted to dissuade them from landing. Butler landed his troops (who needed food and water), occupying the Naval Academy. When Hicks informed Butler that no one would sell provisions to his force, Butler pointed out that armed men did not necessarily have to pay for needed provisions, and he would use all measures necessary to ensure order.

After being joined by the 7th New York Militia, Butler directed his men to restore rail service between Annapolis and Washington via Annapolis Junction, which was accomplished by April 27. He also threatened Maryland legislators with arrest if they voted in favor of secession, and he seized the Great Seal of Maryland, "without which no legislation could become law." Butler's prompt actions in securing Annapolis were received with approval by the US Army's top general, Winfield Scott, and he was given formal orders to maintain the security of the transit links in Maryland. In early May, Scott ordered Butler to lead the operations that occupied Baltimore. On May 13 he entered Baltimore on a train with 1000 men and artillery, with no opposition. That was done in contravention of Butler's orders from Scott, which had been to organize four columns to approach the city by land and sea. General Scott criticized Butler for his strategy (despite its success) as well as his heavy-handed assumption of control of much of the civil government, and he recalled him to Washington. Butler shortly after received one of the early appointments as major general of the volunteer forces. His exploits in Maryland also brought nationwide press attention, including significant negative press in the South, which concocted stories about him that were conflations of biographical details involving not just Butler but also a namesake from New York and others.

Fort Monroe, Virginia

When two Massachusetts regiments had been sent overland to Maryland, two more were dispatched by sea under Butler's command to secure Fort Monroe at the mouth of the James River. After being dressed down by Scott for overstepping his authority, Butler was next assigned command of Fort Monroe and of the Department of Virginia. On May 27, Butler sent a force north to occupy the lightly defended adjacent town of Newport News, Virginia at Newport News Point, an excellent anchorage for the Union Navy. The force established and significantly fortified Camp Butler and a battery at Newport News Point that could cover the entrance to the James River ship canal and the mouth of the Nansemond River. Butler also expanded Camp Hamilton, established in the adjacent town of Hampton, Virginia, just beyond the confines of the fort and within the range of its guns.

The Union occupation of Fort Monroe was considered a threat to Richmond by Confederate General Robert E. Lee, and he began organizing the defense of the Virginia Peninsula in response. Confederate General John B. Magruder, seeking to buy time while awaiting men and supplies, established well-defended forward outposts near Big and Little Bethel, only from Butler's camp at Newport News as a lure to draw his opponent into a premature action. Butler took the bait, and suffered an embarrassing defeat at the Battle of Big Bethel on June 10. Butler devised a plan for a night march and operation against the positions but chose not to lead the force in person, for which he was criticized.Quarstein and Mroczkowski (2000), p. 48 The plan proved too complex for his inadequately trained subordinates and troops to carry out, especially at night, and was further marred by the failure of staff to communicate passwords and precautions. A friendly fire incident during the night gave away the Union position, further harming the advance, which was attempted without knowledge of the layout or the strength of the Confederate positions. Massachusetts militia general Ebenezer W. Peirce, who commanded in the field, received the most criticism for the failed operation. With the withdrawal of many of his men for use elsewhere, Butler was unable to maintain the camp at Hampton, although his forces retained the camp at Newport News.Quarstein and Mroczkowski (2000), p. 49 Butler's commission, which required approval from Congress, was vigorously debated after Big Bethel, with critical comment raised about his lack of military experience. But his commission was narrowly approved on July 21, the day of the First Battle of Bull Run, the war's first large-scale battle. The battle's poor outcome for the Union was used as cover by General Scott to reduce Butler's force to one incapable of substantive offense, and it was implicit in Scott's orders that the troops were needed nearer to Washington.

In August, Butler commanded an expeditionary force that, in conjunction with the United States Navy, took Forts Hatteras and Clark in North Carolina. That move, the first significant Union victory after First Bull Run, was lauded in Washington and won Butler accolades from President Lincoln. Butler was sent back to Massachusetts to raise new forces. That thrust Butler into a power struggle with Governor Andrew, who insisted on maintaining his authority to appoint regimental officers, refusing to commission (among others) Butler's brother Andrew and several of the general's close associates. The spat instigated a recruiting war between Butler and the state militia organization. The dispute delayed Butler's return to Virginia, and in November he was assigned to command ground troops in Louisiana.

While in command at Fort Monroe, Butler had declined to return to their owners fugitive slaves who had come within his lines. He argued that Virginians considered them to be chattel property, and that they could not appeal to the Fugitive Slave Law of 1850 because of Virginia's secession. "I am under no constitutional obligations to a foreign country," he said, "which Virginia now claims to be." Furthermore, slaves used as laborers for building fortifications and other military activities could be considered contraband of war. "Lincoln and his Cabinet discussed the issue on May 30 and decided to support Butler's stance". It was later made standard Union Army policy to not return fugitive slaves.Finkelman (2006), p. 277 This policy was soon extended to the Union Navy.

New Orleans

Butler directed the first Union expedition to Ship Island, off the Mississippi Gulf Coast, in December 1861, and in May 1862 commanded the force that conducted the capture of New Orleans after its occupation by the Navy following the Battle of Forts Jackson and St. Philip. In the administration of that city he showed great firmness and political subtlety. He devised a plan for relief of the poor, demanded oaths of allegiance from anyone who sought any privilege from government, and confiscated weapons.

However, Butler's subtlety seemed to fail him as the military governor of New Orleans when it came to dealing with its Jewish population, about which the general, referring to local smugglers, infamously wrote, in October 1862: "They are Jews who betrayed their Savior, & also have betrayed us."

Public health management

In an ordinary year, it was not unusual for as much as 10 percent of the city's population to die of yellow fever. In preparation, Butler imposed strict quarantines and introduced a rigid program of garbage disposal. As a result, in 1862, only two cases were reported.

Civil administration difficulties

Some of his acts were highly unpopular. Most notorious was Butler's General Order No. 28 of May 15, 1862, that if any woman should insult or show contempt for any officer or soldier of the United States, she may be treated similarly to a "woman of the town plying her avocation," i.e., a prostitute. This was in response to various acts of verbal and physical abuse inappropriate for "respectable" women, including mocking the funeral cortège of a fallen soldier, spitting in the faces of U.S. officers, pouring chamber pots full of human excrement on patrolling U.S. soldiers, and, in one notorious case, pouring urine on Union Navy commander Admiral David Farragut.

"Butler's 'Woman Order' was immediately effective. Insults by word, look or gesture abruptly ceased.... Throughout the South, however, the Woman Order evoked a universal shout of execration". Butler's insistence on prosecuting the woman as any other person "aiding the Confederacy" provoked angry jeers from white residents of New Orleans, who amplified a narrative that he used his power to engage in the petty looting of New Orleanians. " r years after the Civil War steamships plying the lower Mississippi were furnished with chamber pots bearing the likeness of 'Beast Butler.

He was nicknamed "Butler the Beast" by Confederate General P. G. T. Beauregard (despite Beauregard's leaving his wife under Butler's personal care) or alternatively "Spoons Butler", the latter nickname deriving primarily from an incident in which Butler seized a 38-piece set of silverware from a New Orleans woman who attempted to cross Union lines while using a pass that permitted her to carry nothing more than the clothing on her person.

Cotton seizures

Shortly after the Confiscation Act of 1862 became effective in September, Butler increasingly relied upon it as a means of grabbing cotton. Since the Act permitted confiscation of property owned by anyone "aiding the Confederacy," Butler reversed his earlier policy of encouraging trade by refusing to confiscate cotton brought into New Orleans for sale. First, he conducted a census in which 4,000 respondents failing to pledge loyalty to the Union were banished. Their property was seized and sold at low auction prices in which his brother Andrew was often the prime buyer. Next, the general sent expeditions into the countryside with no military purpose other than to confiscate cotton from residents who were assumed to be disloyal. Once brought into New Orleans, the cotton would be similarly sold in rigged auctions. To maintain correct appearances, auction proceeds were dutifully held for the benefit of "just claimants", but the Butler consortium still ended up owning the cotton at bargain prices. Always inventive of new terminology to achieve his ends, Butler sequestered, or made vulnerable to confiscation, such "properties" in all of Louisiana beyond parishes surrounding New Orleans.

Censorship of newspapers

Butler censored New Orleans newspapers. When William Seymour, the editor of the '' New-Orleans Commercial Bulletin'', asked Butler what would happen if the newspaper ignored his censorship, an angry Butler reportedly stated, "I am the military governor of this state — the supreme power — you cannot disregard my order, Sir. By God, he that sins against me, sins against the Holy Ghost." When Seymour published a favorable obituary of his father, who had been killed serving in the Confederate army in Virginia, Butler confiscated the newspaper and imprisoned Seymour for three months.

Execution of William Mumford

On June 7, 1862, Butler ordered the execution of William B. Mumford for tearing down a United States flag placed by Admiral Farragut on the United States Mint in New Orleans. In his memoirs, Butler maintained that " party headed by Mumford had torn down the flag, dragged it through the streets and spit on it, and trampled on it until it was torn to pieces. It was then distributed among the rabble, and each one thought it a high honor to get a piece of it and wear it." Butler added that these actions were "against the laws of war and his country."

Before Mumford was executed, Butler permitted him to make a speech for as long as he wished, and Mumford defended his actions by claiming that he was acting out of a high sense of patriotism. Most, including Mumford and his family, expected Butler to pardon him. The general refused to do so, but promised to care for his family if necessary. (After the war, Butler fulfilled his promise by paying off a mortgage on Mumford's widow's house and helping her find government employment.) For the execution and General Order No. 28, he was denounced (December 1862) by Confederate President Jefferson Davis in General Order 111 as a felon deserving capital punishment, who, if captured, should be "reserved for execution".

Recall

Although Butler's governance of New Orleans was popular in the North, where it was seen as a successful stand against recalcitrant secessionists, some of his actions, notably those against the foreign consuls, concerned Lincoln, who authorized his recall in December 1862. Butler was replaced by Nathaniel P. Banks. The necessity of taking sometimes radical actions and the support he received in Radical Republican circles drove Butler to change political allegiance, and he joined the Republican Party. He also sought revenge against the more moderate Secretary of State Seward, whom he believed to be responsible for his eventual recall.

Butler continues to be a disliked and controversial figure in New Orleans and the rest of the South.

Louisiana Native Guard

On September 27, 1862, Butler formed the first African-American regiment in the US Army, the 1st Louisiana Native Guard, and commissioned 30 officers to command it at the company level. This was highly unusual, as most USCT regiments were commanded by white officers only. "Better soldiers never shouldered a musket," Butler wrote, "I observed a very remarkable trait about them. They learned to handle arms and to march more easily than intelligent white men. My drillmaster could teach a regiment of Negroes that much of the art of war sooner than he could have taught the same number of students from Harvard or Yale." The regiment would serve Butler effectively during the Siege of Port Hudson. Butler organized three regiments totaling 3,122 soldiers and officers.

Army of the James

Butler's popularity with the Radicals meant that Lincoln could not readily deny him a new posting. Lincoln considered sending him to a position in the Mississippi River area in early 1863, and categorically refused to send him back to New Orleans. In November 1863, he finally gave Butler command of the Department of Virginia and North Carolina based in Norfolk, Virginia. In January 1864, Butler played a pivotal role in the creation of six regiments of U.S. Volunteers recruited from among Confederate prisoners of war (" Galvanized Yankees") for duty on the western frontier. In May, the forces under his command were designated the Army of the James. On November 4, 1864, Butler arrived in New York City with 3,500 troops of the Army of the James. Secretary of War Edwin Stanton had "requested that Grant send troops to New York City to help oversee the election there. Stanton's concern arose from the city's perennial political and racial divisions, which had erupted during the 1863 draft riots," and because of fear of Confederates coming from Canada to burn the city on Election Day. Grant selected Butler for the assignment. "Even though he knew nothing about the plot o burn the cityand did nothing to prevent it, Butler's mere presence with his 3,500 troops" demoralized the leaders of the conspiracy, who postponed it until November 25, when it failed.

The Army of the James also included several regiments of United States Colored Troops. These troops saw combat in the Bermuda Hundred campaign (see below). At the Battle of Chaffin's Farm (sometimes also called the Battle of New Market Heights), the USCT troops performed extremely well. The 38th USCT defeated a more powerful force despite intense fire, heavy casualties, and terrain obstacles. Butler awarded the Medal of Honor to several men of the 38th USCT. He also ordered a special medal designed and struck, which was awarded to 200 African-American soldiers who had served with distinction in the engagement. This was later called the Butler Medal.

Bermuda Hundred campaign

In the spring of 1864, the Army of the James was directed to land at Bermuda Hundred on the James River, south of Richmond, and from there attack Petersburg. This would sever the rail links supplying Richmond, and force the Confederates to abandon the city. In spite of Grant's low opinion of Butler's military skills, he was given command of the operation.

Butler's force landed on May 5, when Petersburg was almost undefended, but Butler became unnerved by the presence of a handful of Confederate militia and home guards. While he dithered, the Confederates assembled a substantial force under General P. G. T. Beauregard. On 13 May, Butler's advance toward Richmond was repulsed. On May 16, the Confederates drove Butler's force back to Bermuda Hundred, bottling up the Union troops in a loop of the James River. Both sides entrenched; the Union troops were safe but impotent, and Beauregard sent most of his troops as reinforcements to Lee's Army of Northern Virginia. Had Butler been more aggressive in early May, he might have taken Petersburg or even Richmond itself and ended the war a year early, although his two West Pointer corps commanders Maj. Gen "Baldly" Smith and Quincy Gilmore also did not perform well or make up for Butler's limitations as a general.

Despite this fiasco, Butler remained in command of the Army of the James.

Fort Fisher and final recall

Although Grant had largely been successful in removing incompetent political generals from service, Butler could not be easily gotten rid of. As a prominent Radical Republican, Butler was a potential replacement for Lincoln as the party's presidential nominee. Lincoln had even asked Butler to be the 1864 nominee for vice president, as did Treasury Secretary Salmon P. Chase, who sought to replace Lincoln as president. In reply to Lincoln's offer, Butler said, "Tell him ... I would not quit the field esign as major generalto be Vice-President, even with himself as President, unless he will give me bond with sureties ... that he will die or resign within three months after his inauguration. Ask him what he thinks I have done to deserve the punishment ... of being made to sit as presiding officer over the Senate, to listen for four years to debates more or less stupid, in which I can take no part or say a word...."

There was no good place to put Butler; sending him to Missouri or Kentucky would likely end in disaster, so it was considered safer to leave him where he was in Virginia. More worrying was the fact that Butler was one of the highest ranking volunteer major generals in the Union army; next to Grant himself, he was the ranking field officer in the Eastern theater, and command of the Army of the Potomac would default to him in Grant's absence. For that reason, Grant remained with the army as much as possible and only made trips away from the front when it was absolutely necessary.

In December, troops from the Army of the James were sent to attack Fort Fisher in North Carolina with Butler in command. Butler devised a scheme to breach the defenses with a boat loaded with gunpowder, which failed completely. He then declared that Fort Fisher was impregnable and withdrew his troops without authorization. However, Admiral David Dixon Porter (commander of the naval element of the expedition) informed Grant that it could be taken easily if anyone competent were put in charge.

This mismanagement finally led to his recall by Grant in early 1865. As Secretary of War Edwin M. Stanton was not in Washington at the time, Grant appealed directly to Lincoln for permission to terminate Butler, noting "there is a lack of confidence felt in utler'smilitary ability". Grant also voiced his suspicions about corruption going on in Butler's department, including smuggling of supplies to Lee's army, and that Butler arbitrarily arrested anyone who noticed what was going on, although, due to Butler's formidable political connections, nothing came of Grant's complaints. By this point, the presidential election was over, so the administration no longer had to be concerned about Butler's running for president, and, in General Order Number 1, Lincoln relieved him from command of the Department of North Carolina and Virginia and ordered him to report to Lowell, Massachusetts.Foote, pp. 739–740 Grant informed Butler of his recall on January 8, 1865, and named Major General Edward O. C. Ord to replace him as commander of the Army of the James. "Embarrassed and outraged, Butler broke off all relations with Grant and set out to destroy him." Simpson, Brooks D., ''Let Us Have Peace: Ulysses S. Grant and the Politics of War and Reconstruction, 1861-1868'', Chapel Hill and London: The University of North Carolina Press, 1991, p. 210. In 1867, when it seemed that Grant might run for president, Butler "employed detectives in an effort to prove that Grant was "a drunkard, after fast horses, women and whores." Grant, he announced, was "a man without a head or a heart, indifferent to human suffering and impotent to govern."

Rather than report to Lowell, Butler went to Washington, where he used his considerable political connections to get a hearing before the Joint Congressional Committee on the Conduct of the War in mid-January. At his hearing Butler focused his defense on his actions at Fort Fisher. He produced charts and duplicates of reports by subordinates to prove he had been right to call off his attack of Fort Fisher, despite orders from General Grant to the contrary. Butler claimed the fort was impregnable. To his embarrassment, a follow-up expedition led by Maj. Gen. Alfred H. Terry and Brig. Gen. Adelbert Ames (Butler's future son-in-law) captured the fort on January 15, and news of this victory arrived during the committee hearing; Butler's military career was over. He was formally retained until November 1865 with the idea that he might act as military prosecutor of Confederate President Jefferson Davis.

Colonization

General Butler claimed that Lincoln approached him in 1865, a few days before his assassination, to talk about reviving colonization in Panama. Since the mid-twentieth century, historians have debated the validity of Butler's account, as Butler wrote it years after the fact and was prone to exaggerating his prowess as a general. Recently discovered documents prove that Butler and Lincoln did indeed meet on April 11, 1865, though whether and to what extent they talked about colonization is not recorded except in Butler's account.

Financial dealings

Negative perceptions of Butler were compounded by his questionable financial dealings in several of his commands, as well as the activities of his brother Andrew, who acted as Butler's financial proxy and was given "almost free rein" to engage in exploitative business deals and other "questionable activities" in New Orleans. Upon arriving in the city, Butler immediately began attempts to participate in the lucrative inter-belligerent trade. He used a Federal warship to send $60,000 in sugar to Boston where he expected to sell it for $160,000. However, his use of the government ship was reported to the military authorities, and Butler was chastised. Instead of earning a profit, military authorities permitted him to recover only his $60,000 plus expenses. Thereafter, his brother Andrew officially represented the family in such activities. Everyone in New Orleans believed that Andrew accumulated a profit of $1–$2 million while in Louisiana. Upon inquiry from Treasury Secretary Chase in October 1862, the general responded that his brother actually cleared less than $200,000 (~$ in ). When Butler was replaced in New Orleans by Major General Nathaniel Banks, Andrew Butler unsuccessfully tried to bribe Banks with $100,000 if Banks would permit Andrew's "commercial program" to be carried out "as previous to anks'sarrival."

Butler's administration of the Norfolk district was also tainted by financial scandal and cross-lines business dealings. Historian Ludwell Johnson concluded that during that period: "... there can be no doubt that a very extensive trade with the Confederacy was carried on in utler's NorfolkDepartment.... This trade was extremely profitable for Northern merchants ... and was a significant help to the Confederacy.... It was conducted with Butler's help and a considerable part of it was in the hands of his relatives and supporters."

Shortly after arriving in Norfolk, Butler became surrounded by such men. Foremost among them was Brigadier General George Shepley, who had been military governor of Louisiana. Butler invited Shepley to join him and "take care of Norfolk." After his arrival, Shepley was empowered to issue military permits allowing goods to be transported through the lines. He designated subordinate George Johnston to manage the task. In fall 1864, Johnston was charged with corruption. However, instead of being prosecuted, he was allowed to resign after saying he could show "that General Butler was a partner in all he controversialtransactions," along with the general's brother-in-law Fisher Hildreth. Shortly thereafter, Johnston managed a thriving between-the-lines trade depot in eastern North Carolina. There is no doubt that Butler was aware of Shepley's trading activities. His own chief of staff complained about them and spoke of businessmen who "owned" Shepley. Butler took no action.

Much of the Butler-managed Norfolk trade was via the Dismal Swamp Canal to six northeastern counties in North Carolina separated from the rest of the state by Albemarle Sound and the Chowan River