The operatives are installed at the rebel command for the eastern front, at the dysfunctional oil refinery in Zuwaytina, about 150 kilometres southwest of the opposition capital Benghazi.
They are equipped with telecommunications equipment and housed in two shipping containers, within walking distance of the headquarters of Fawzi Bukatif, commander of the eastern front.
He has been working out of a large office with walls covered in maps and satellite photos.
There are at least two Frenchmen, and several Britons in mismatched camouflage outfits.
In late April, Britain, France, Italy, Egypt and the United States announced that they had sent military advisers to the National Transitional Council, the rebels' de facto government.
Britain's Defence Minister Liam Fox said Thursday that NATO is contributing intelligence and reconnaissance equipment to the search for Qaddafi but he refused to confirm reports that Britain's SAS special forces were working with the Libyan rebels to track down Qaddafi.
"I can confirm that NATO is providing intelligence and reconnaissance assets to the NTC (National Transitional Council) to help them track down Colonel Qaddafi and other remnants of the regime," who fled before advancing rebel forces on Tuesday, he told Sky News.
The Ministry of Defence said Fox was referring to "various assets such as military planes."
The Daily Telegraph newspaper, quoting defence sources, said SAS members were sent to Libya several weeks ago and played a key role in coordinating the battle for Tripoli.
With the majority of the capital now in rebel hands, the SAS had been ordered to switch their focus to hunting down Qaddafi, the Telegraph said.
They were wearing civilian clothes and armed with the same type of weapons used by the rebel forces, the paper said.
"We never comment about special forces," Fox said in a separate interview with BBC radio.
Asked what role Britain was playing on the ground in Libya, Fox told the BBC: "We have always had some advisors to the NTC (as) we have made clear from the outset, helping them with communications, helping them with logistics, the chain of command and so on.
"And we would of course want to continue with those relationships."
The rebels who overran the Libyan capital and captured Qaddafi's Bab al-Azizya compound on Tuesday, have offered a $1.7 million reward for the capture of the elusive strongman, dead or alive.
The Zuwaytina refinery served as the rebel headquarters while directing weeks-long battles for the key oil hub of Brega, 90 kilometres to the east, where the insurgents scored a knockout blow against pro-Qaddafi forces in an operation backed by NATO air strikes.
After the rebels stormed Tripoli on Sunday, Qaddafi's forces retreated more than 100 kilometres to the west, stopping at the town of Bin Jawad, 140 kilometers east of the port of Sirte, Qaddafi's hometown and his last remaining bastion.
Bukatif was preparing on Wednesday to move his operational command from Zuwaytina to the town of Ras Lanouf, about 20 kilometres from Bin Jawad.