For much of mid-1776, a key question for George Washington centered on what should be done about New-York, and whether it should be defended or abandoned. York Island, upon the tip of which stood the city, had some distinct advantages, but was also highly vulnerable to attack from the water. New Yorkers knew that they could easily be surrounded by the British ships, who only needed a favorable wind to sail up the Hudson or East River and unleash their fire. The British had made a show of this might earlier that July, when the HMS Phoenix and Rose sailed up the Hudson River, examining the woefully inept American defenses on their way up to Haverstraw Bay.

The HMS Phoenix and Rose on the Hudson River. They were attacked by American fire ships and galleys, but this only caused minor damage, and the ships were able to sail back down the Hudson and participate in further engagements. From D.T. Valentine’s Manual of the Corporation of the City of New York, 1864.
In early September, with the British in control of Long Island, and their fleet still anchored off of Staten Island, the concern of holding New York was weighing heavily on Washington and his generals. This was compounded not only by the loss of those who were killed, wounded, captured, or missing from the Battle of Brooklyn, but from entire groups of men who decided to leave and go home after the battle. At least a quarter of the soldiers in the army were still sick, and as the army atrophied, their presence on Manhattan became increasingly perilous.
New Yorkers were expecting another imminent British attack. The army was re-arranged in the early days of September, stretching along the east side of Manhattan from the Battery up to Kingsbridge in the Bronx. Some officers argued that New York should be abandoned, including General Nathanael Greene, who had been sick during the Battle of Brooklyn but, upon recovery, noted that since the troops were scattered along such a long line, over 14 miles in length, that “one part may be cut off before the others can come to their support.” Others debated burning the city rather than leave it to the enemy, although Congress declared that New-York “should in no event be damaged.”

Spoiler alert: New-York was in fact damaged by a terrible fire that broke out on September 21st, 1776. Did an angry American set the fire? Or a frustrated British soldier? No one knows. The ruins of Trinity Church and many other buildings were left that way for years before being rebuilt. From A Brief History of the City of New York by Charles Burr Todd.
At a Council of War held on September 7th, the decision was made, as Washington wrote to John Adams, “to Arrange the Army under Three Divisions, 5000 to remain for the defence of the City, 9000 to Kingsbridge & Its dependancies… The remainder to occupy the intermediate space & support either.” However, by September 12th, the generals voted to reconsider the abandonment of New York, with the majority agreeing that this was the better course. As part of this, some 8,000 troops would be left to defend Fort Washington.
The sick were evacuated from the island first, as this was the most time-consuming activity. Washington moved his headquarters to the Morris-Jumel Mansion in upper Manhattan, and issued orders urging the soldiers to be particularly vigilant, as “we have once found the bad consequences of a surprize [sic]; let the utmost Care be used to prevent another.” By September 14th, many of the soldiers had moved up to Harlem Heights and Kingsbridge, leaving men under General Israel Putnam to guard the points below.

The British and Hessian longboats landing at Kip’s Bay, from a sketch made in 1778 by Archibald Robertson. An officer in the Royal Engineers, Robertson participated in the invasion, and may have based this on an earlier sketch he drew closer to the events. From Iconography of the City of New York, Volume V by I.N. Phelps Stokes.
In the days before this, five British frigates, including the Phoenix, Rose, Roebuck, Orpheus, and Carysfort, had sailed past Governors Island and up the East River, anchoring off of Bushwick Creek. Two brigades of light infantry had landed on Montresor’s Island (now Randall’s Island), which General Henry Clinton favored for a spot to coordinate, as he believed that control of the East and Harlem Rivers would be key to taking control of Manhattan. However, given that the island was in the middle of the treacherous Hell Gate, the Howes declined to base a significant force there, electing instead to invade Manhattan at Kip’s Bay.

The Kip house, shown before it was demolished. It stood around 35th Street and 2nd Avenue. From Old New York, From its Earliest History to About the Year 1868 by L. Bayard Smith.
Kip’s Bay was named after the Kip family, who owned a large farm in the area, called the Kip’s Bay Farm. Jacob Kip had purchased the tract around 1654, and the next year built a large stone house overlooking the bay. The house was made out of yellow bricks from Holland, with a main house and a wing, with the family’s crest sculpted in a prominent spot over the front door. Outside, a farm lane led to the expansive bay. By 1776, Jacob’s elderly grandson and his family were living in the house, and they were present for the invasion.

A soldier’s perspective aboard a British warship as the longboats, laden with men, carry out the invasion. From History of the Kip Family in America by Frederic Ellsworth Kip.
Early in the morning of September 15th, the British frigates sailed close to Manhattan to provide cover for 84 longboats that were preparing to proceed out of the Newtown Creek, carrying 4,000 British and Hessian soldiers. The Americans posted in the lines at Kip’s Bay heard cannons ringing out from the west, as four warships had sailed up the Hudson to draw attention away from the invasionary force, a feint that convinced Washington that the British were targeting Harlem. However, by around 11 o’clock, when the guns opened from the ships on the East River, it was clear where the main attack would be launched.
Withering under the brutal fire from the ships, which lasted an hour, many of the American lines broke, and soldiers flew about in an attempt to save themselves. Private Joseph Plumb Martin wrote that the fire was like a “peal of thunder,” and he dove into a ditch to save himself. According to Major Nicholas Fish, the bombardment seemed to “infuse a Panic thro’ the whole of our Troops.” Washington, corrected in his assumption that the attack would be made farther up the island, rushed towards Kip’s Bay.
Reaching Inclenberg, the Murray estate (near 42nd Street and the New York Public Library), Washington found that his men were streaming up the Post Road, having abandoned their positions. Generals Samuel Parson and John Fellows arrived with reinforcements, who Washington ordered to take the nearby cornfield and fire from behind the walls of the estate. However, the sight of the warships and the longboats that were fast discharging the enemy on the shore, compounded by the panic of their fellow soldiers, caused great disarray, and most of them fled as well.

A depiction of Washington moments away from being killed or captured by the enemy. From The Life of Israel Putnam, Major-General in the Army of the American Revolution by William Cutter.
Washington was incandescent at this, and supposedly struck several officers who were retreating with his riding crop, threw his hat on the ground in a rage, and, according to General William Heath bellowed “Are these the men with which I am to defend America?” George Weedon recorded that he uttered “Good God have I got such Troops as Those?” His anger momentarily getting the best of him, Washington is said to have ridden within a hundred yards of the enemy, and was only saved by one of his aides pulling his horse away. While these tales may be embellished, he did later write to John Hancock of the “disgraceful and dastardly conduct” that he witnessed that day.

The Inclenberg Mansion and other stately homes along the Boston Post Road, depicted around the turn of the 19th century. Inclenberg burned down in 1835. D.T. Valentine’s Manual of the Corporation of the City of New York, 1859.
While the British and Hessians were streaming onto the island, experiencing periodic resistance from American soldiers who remained in the area, they proceeded up Inclenberg and turned to the north to harass those who were retreating. The Hessian jagers once again proved to be particularly frightening, and put surrendering troops to their bayonets. However, at some point in the afternoon, the forward movement of the invading force slowed. This was likely due to the need to await fresh reinforcements, although General Henry Clinton urged for the troops to continue on and decimate the Americans. An apocryphal story that has long been passed down holds that General Howe and several others were at the Inclenberg Mansion having cake and wine with Mary Lindley Murray (the family name was later appended to the hill, becoming Murray Hill).

Fort George and the Independent Battery atop Bayard’s Mount. From Pictorial Field Book of the Revolution, Vol II by Benson John Lossing.
Thousands of American troops were still at risk of being trapped in lower Manhattan, including Generals Henry Knox and Israel Putnam. Knox had been stationed at the redoubt on Bayard Hill, and when the facts of what was transpiring were evident, he tried to get his men to rally and defend the fort. Eventually it was clear that this was far too dangerous, and so he escaped by commandeering a boat on the Hudson River. Putnam, still working to get the men and camp followers to safety, did not know that the Post Road, the main route out of the city, was now fully occupied by the enemy. On the advice of Major Aaron Burr, Putnam was convinced to take the roads up the rugged west side of the island.

A map of New-York, c. 1778. It shows the city clustered at the lower edge of York island, and the roads that wound up the island. From History of the Kip Family in America by Frederic Ellsworth Kip.
New Yorkers who identified with the Patriot cause also clamored to get out of New-York before the British fully took it, scrambling to get onto boats that would take them to New Jersey or points north. Upon reaching the city, the British forces were greeted by a throng of happy Loyalists who expected that they would soon fully stamp out the rebellion. A woman climbed the flagpole over Fort George and another hoisted up the British flag, making a show of trodding upon the flag that had been brought down. According to a Loyalist clergyman, “Joy and gladness seemed to appear in all countenances.”

Position of the British and American armies throughout the day on September 15th, 1776. By nightfall, the British had effectively cut off all of lower Manhattan from the Americans, and they would hold onto the city for the rest of the war. From The Battle of Harlem Heights, September 16, 1776 by Henry Phelps Johnston.
As this was all happening, the bulk of the British and Hessian troops moved up Manhattan, spreading out in a line across the island and stopping in view of the American lines. On the east they were anchored by Hoorn’s Hook, where they took over a small redoubt built by the Americans that overlooked the East River and Hell Gate. On the west they spread around the country estates, with the men camping in the fields and the officers stationed at the Apthorp Mansion. The men rested that night, secure that they had won the day, generated considerable chaos, and badly frightened the Americans.