"I think of what happened almost four weeks ago now," the visibly emotional artist said. "The heart of Paris and the heart of France beats in the heart of each city."
She continued: "We are one heart and our heart can beat as one... The power of love is greater than the love of power.
"I came here when I was 20 and it was here, in Paris, that I decided to make music. Thank you Paris for planting that seed in my heart!"
Swathed in a French flag, she sang the French national anthem, La Marseillaise, bringing the concert-goers to their feet.
The star then moved to Place de la Republique, which has become an unofficial hub for mourners of the November 13 attacks, and sang a series of songs to a small crowd.
"It was inspiring, it was raw, it was real," said Pete Hall, a 45-year-old tourist from London who had just left Madonna's concert at Bercy when he saw on social media she was at the square in central Paris.
"She was paying her respects to Paris," he said.
Wearing a down jacket with a fur hood over her head and accompanied by her guitarist Monte Pittman, Madonna sang her hits "Like a Prayer" and "Ghosttown", as well as John Lennon's "Imagine", according to those who attended the impromptu concert.
"She wanted to pay tribute, she loves this city," Pittman told AFP. "It is magical. I will never forget this."
The November 13 attacks in Paris, which were carried out by the Islamic State group and were the deadliest ever in France, killed 130 people and left 350 others wounded.
Days after the bloodshed, Madonna paid a tearful tribute to the victims on stage by singing the classic French song "La vie en rose," accompanied only by a guitar, during a concert in Stockholm.