The protesters included people of Kashmiri origin, as well as Muslims and Sikhs belonging to India, with some carrying banners during the peaceful demonstration.
Modi was elected by a landslide in 2014 but he and his Hindu nationalist Bharatiya Janata Party (BJP) have faced accusations from opponents that they are failing to protect minorities.
He is on a three-day visit to Britain which will include talks with Cameron, lunch with Queen Elizabeth II and a rally with supporters at Wembley Stadium.
Modi was effectively banned from Britain until three years ago over anti-Muslim riots which killed more than 1,000 people in 2002, when he was chief minister of Gujarat state. He has always denied any responsibility.
Nirmala Rajasingam, one of the protest organisers, said: “We have been protesting against the Modi government for a very long time because the matters of the Gujarat atrocities have not been settled yet”.
“Signs of religious antipathy have been there for a long time but since the BJP took power, it’s gone to a new high. There is a very specific project that they are pursuing to make India a country for Hindus only.”
Another protestor, Thaman Singh Sidhu, added: “The Sikh community, the Muslim community, all minorities feel threatened… he has never condemned anything.”
Galloway blasts Modi from outside Downing Street
Former British parliamentarian George Galloway also came up in criticism of Modi, addressing through a video from outside the Downing Street with the people protesting against the red carpet laid out for Modi.
Galloway charged Modi with murdering India of Gandhi, making an interesting comment, “He (Modi) is dripping in blood, that’s why the carpet in Downing Street is red”.
Outside Downing Street today with the people protesting against the red carpet laid out for Modi #India #Gandhi
Posted by George Galloway MP on Thursday, November 12, 2015
“The blood of the people of Gujarat (India) and nowadays the blood of Sikh religious minority, the Tamils and the people of occupied Kashmir”, noted the British politician. “All of them are facing brutal savagery and repression at the hands of the killer Modi.”
“Modi is a prime minister of a great country, but he is no great premier,” he castigated.
Galloway also slammed David Cameron “for honoring a killer Modi,” who he referred as “a man coming from the very organization behind the murder of Gandhi”.