It was the second deadly crane accident in two and a half months in New York, which is undergoing a building boom.
Mayor Michael Bloomberg said yesterday's collapse was "unacceptable" and the city would investigate.
Mr Bloomberg said: "What has happened is unacceptable and intolerable. Having said that, we do not know at the moment what happened or why." He added that it appeared the builders followed regulations.
City building department records show that inspectors stopped crane work twice last month, once because they said a crane lacked the proper permit and was being operated in an unsafe manner.
All three of the victims were building workers and one of the dead men had been in the cab of the crane. One pedestrian was also treated for minor injuries.
The mayor said seven buildings have been evacuated as a precaution following the collapse on the city's Upper East Side neighbourhood.
Mr Bloomberg said the top part of the crane snapped off and fell against the building. Video from the scene showed the upper-floor balconies of the apartment building were severely damaged and a hole extended several stories down the side of the building.
Chaos enveloped the largely residential neighbourhood of town houses and apartment high-rises as dozens of emergency vehicles raced to the scene during the morning rush hour.
In the March 15 accident about two miles to the south, contractors building a 46-storey condominium near the United Nations were trying to lengthen the crane when a steel support broke, killing seven people. A four-storey town house was demolished and several other buildings damaged.