Laura Flessel, a popular former Olympic fencing champion, said in a statement she had stepped down for “personal reasons”, without elaborating.
Flessel “decided to leave for reasons linked to her fiscal situation,” the source told AFP.
French investigative website Mediapart reported that tax investigators had recently notified France's tax infraction commission (CIF) due to suspicions she had under-reported her earnings.
Officials suspect that up to tens of thousands of euros of earnings from her media rights business, Flessel & Co., might not have been properly declared over the past three years, the report said.
If confirmed, she could face criminal charges.
“She did not make any mistake, that is not why she's leaving,” a source close to the former minister said, adding that “her husband was the director” and co-owner of the media rights company.
The company, in which Flessel owned 87.5 percent and her husband Denis Colovic 12.5 percent, was wound down in September 2017, according to records seen by AFP.
France's tax authority said it was unable to comment on a confidential fiscal matter.
Despite keeping a low profile, Flessel had some of the highest poll ratings among President Emmanuel Macron's cabinet after taking office last year, four months before Paris won its bid to host the 2024 Olympics.
She had been overseeing preparations for the Games at a time of growing tension between French sports federations and the state over funding, though a source close to Flessel had denied that was her reason for leaving the cabinet.
Macron replaced her with Roxana Maracineanu, a silver medalist in backstroke at the 2000 Summer Olympics in Sydney.
The cabinet change, which included a new environment minister after Nicolas Hulot suddenly quit last week, comes as Macron attempts to rebound from his lowest approval ratings since taking office in May 2017.