Ministers ruled that the proposals for Barvas Moor, Lewis, in the Western Isles were incompatible with European law.
The moor is designated under the EC Bird Directive and EC Habitats Directive.
Developers Lewis Wind Power said the decision was a bitter disappointment.
Energy minister Jim Mather said he had considered the views of the local authority, the 10,924 objections and the 98 letters of support.
He said: "European legislation requires a specific procedure to be followed when proposals which could potentially affect Special Protection Areas come forward.
"I considered all the relevant issues and concluded it would not be possible to approve this application.
"The Lewis Wind Farm would have significant adverse impacts on the Lewis Peatlands Special Protection Area, which is designated due to its high value for rare and endangered birds.
"This decision does not mean that there cannot be onshore wind farms in the Western Isles.
"I strongly believe the vast renewables potential needs to be exploited to ensure that the opportunities and benefits of new development can be shared across the country in an equitable fashion."
Mr Mather said the Scottish Government is committed to developing renewable energy. He said 13 applications had already been approved, including the second and third largest wind farms in the country.
He added: "I am confident we will reach our ambitious renewable energy targets and confident the Western Isles will play a part in helping to achieve that, to the benefit of the community and to the benefit of Scotland."
Lewis Wind Power - a consortium of Amec and British Energy - applied to construct and operate 234 wind turbines on Barvas Moor and other locations in north Lewis in October 2004.
In December 2006, the application was slimmed down to 181 turbines.
In a statement today, the applicant said: "Lewis Wind Power is bitterly disappointed by the Scottish Government's decision to reject our proposal for a wind farm on Lewis.
"Over the six years of this project, we have conducted extensive environmental and economic studies and designed the development around these findings.
"As a result, we believe we had put forward a detailed case showing the benefits of our proposal and the benefits it would bring to Lewis, the Highlands and Islands region and to Scotland.
"We also believe that during our discussions with the government, we demonstrated that this proposal could have been approved without violating European law."
Lewis Wind Power said it had the backing of the Western Isles Council and major businesses.
The consortium claimed the development would have created 400 local jobs during construction and led to investment at the local Arnish Yard.