The 2014 report, obtained by the satirical weekly Le Canard Enchaine, found that a total of 50,000 "occasional public service workers" had not been hired on proper terms.
As a result, contributions to France's welfare system or value-added tax (VAT) were not paid.
The justice ministry alone hired 40,500 of them, according to the probe, carried out by inspectors of the finance, judicial and social services departments.
The workers included interpreters, experts and social helpers, who are often employed on a daily basis and paid by the hour.
"The Ministry of Justice applies no requirement for social security contributions" for these casual workers, the report said.
Instead, "remuneration is treated as payment for work by external suppliers, yet no steps are taken to apply VAT," it said.
The document said that the practice was long-standing.
If a class action lawsuit were to be launched, the cost for the public purse would be around half a billion euros ($600 million), it said.
Several suits have already been filed individually, mainly by translator-interpreters, who say the practice has caused them to have gaps in their contributions to France's welfare system.
"These are people who work exclusively at the request of police or judicial authorities. It's 100 percent of their professional activity," lawyer David Dokhan told France Inter radio, adding that his clients had "no pay slip" and "no social protection."
Justice Minister Christiane Taubira "is keen to act on this issue," a ministry spokesman told AFP, adding that an "action plan" had already been drawn up, which includes the rapid recruitment of 45 interpreters.
A ruling is expected early next year to clarify the status of casual hirees and there will be a "gradual" move towards payment of social security contributions, he added.
France is wrestling with the dilemma of how to reduce government spending and curb its budget deficit yet maintain its lavish social safety net.
Cutting down on waste and tackling tax avoidance have become big subjects of debate.
The changes will have a high cost for state coffers, as well as for ministerial workers whose earnings have been 30 percent higher because of the avoided deductions, according to the report.
The increased cost may also discourage recruitment of casual workers, the document said.