The National Audit Office (NOA) said unresolved issues over construction deals, how venues will be used post-2012, and security for the Games could all put the £3 billion budget at risk.
In a report, it also urged the Olympic Delivery Authority (ODA) to hold a "firm line" as plans are developed or risk overshooting the budget.
NAO head Tim Burr said: "The preparations for the Games are well under way.
"But important challenges remain which will become more formidable as the spotlight turns to London after the Beijing Games.
"Uncertainties over the deal for the village, legacy requirements and policing and security may add cost, or compromise the preparations for a successful Games.
"The delivery bodies need to maintain a firm stance on cost and keep in sight the intended legacy benefits too."
The NAO highlighted the lack of a deal for construction of the Olympic Village in east London, which is expected to cost more than £1 billion.
The ODA had expected to secure a deal by last December, but there have been difficulties securing private sector finance amid the downturn in the property market.
Also awaiting finalisation are the "legacy" plans for individual venues, including the main stadium, where it has not been decided whether to accommodate football, rugby or both alongside athletics.
The NAO also said there was still no "firm basis" for security arrangements at the Games. Costed plans are not expected from the Home Office until the end of 2008.
The watchdog's concerns echoed those voiced earlier this week by David Ross, the tycoon appointed by new Mayor of London Boris Johnson to review preparations for the Games.
The report added: "With the fixed deadline for the start of the Games, a degree of pragmatism on the part of the Olympic Delivery Authority has been necessary in balancing the need to move forward the construction programme against the possibility of stakeholders' requirements changing.
"There will be a risk of cost overruns and loss of time unless a firm line is adopted on subsequent calls for changes in the designs for the venues and infrastructure."
Commenting on the NAO Report, ODA Chairman John Armitt said: “I welcome the National Audit Office’s recognition that good progress has been made in preparing the Olympic Park site in the last year. We have started construction early and are on track, endorsed recently by a positive report from the International Olympic Committee (IOC).
“As the report highlights, there are cost pressures in some areas. Equally there are potential savings in others. Overall we remain within the budget set out by Government. The project is well managed and I am confident we can deliver as we enter the big build phase.
“Given the challenging economic environment, we are in ongoing discussions about the level of public investment in the Olympic Village. We expect to finalise this later in the year. Work has started on site and this will be a world-class development capable of delivering returns on any investment when the homes are sold after the Games.
“Seventy-five pence of every pound we spend is for long-term regeneration so a great deal of work is already being done to create a lasting legacy for the project.”