The US-backed Syrian Democratic Forces (SDF) announced the launch on Sunday of the sweep in Al-Hol camp, which has been rocked by assassinations and breakout attempts.
Kurdish authorities have warned that the settlement, home to almost 62,000 people, is turning into an extremist powder keg because of IS jihadists hiding out among camp residents.
The Kurds' Asayish security forces said they had "detained 53 IS members, including five leaders of IS sleeper cells that carried out violent terrorist attacks in the camp".
They had also "confiscated mobile phones as well as several laptops", the SDF-allied police unit added.
Heavily-armed Kurdish forces stood guard outside the camp as others stormed suspected hideouts inside the vast settlement, an AFP reporter said.
In some sections, residents stood outside their tents watching the anti-terrorist squad scour the area.
Read also: European jihadists' children 'at risk of radicalisation'
Al-Hol is the larger of two Kurdish-run displacement camps for relatives of IS jihadists in Syria's northeast.
It holds mostly Syrians and Iraqis but also thousands from Europe and Asia suspected of family ties with IS fighters.
Many residents see the camp as the last vestige of the IS proto-state that jihadists declared in 2014 across large swathes of both Syria and Iraq.
Kurdish authorities have recorded more than 40 murders in Al-Hol since the start of this year.
They say IS sympathisers are behind most of the murders, while humanitarian aid sources have said tribal disputes could be behind some of the killings.
Simand Ali, a Kurdish official, told AFP jihadists had dug trenches in Al-Hol that they used to hide prohibited electronic devices and other goods.
Those detained so far have mostly been Syrians and Iraqis, he said.
Follow us on Facebook, Twitter and Instagram to stay connected