The administrative tribunal of Paris condemned the plan, launched in 2011 by the city council, as "illegal", complaining that the plan contained insufficient information and that the financial contribution from the French Tennis Federation (FFT) was too low.
The tribunal, which was hearing an appeal from powerful residents associations, said that the city authorities could appeal the decision within two months.
The FFT, which has backed the extension to the cramped site, had said the project would cost 340 million euros ($444 million), up from the original figure of €273 million.
"The FFT is surprised by the decision of the administrative tribunal relating to the Roland Garros modernisation project," said a statement from the sport's governing body in France.
"Contesting the grounds for cancellation, and convinced that the project is respectful of the site, we will appeal the decision and request a stay of execution."
"The determination of the FFT to carry out the project remains intact -- its realisation is indeed vital for the sustainability of the French Open, a major part of France's sporting heritage that contributes to the international influence of France and French tennis."
The Roland Garros stadium, constructed in 1928 and named after a famous French aviator, is sandwiched between the Bois de Boulogne and residential property on the western outskirts of Paris.
With the growing popularity of the French Open, staged annually in late May to early June, space has become a problem and the controversial plan is to increase the surface area from 8.5 hectares (21 acres) to 12.5 hectares with lights and a roof to be installed over the Philippe Chatrier centre court.