Boris Johnson must "lead or step aside", senior Tory and former minister Tobias Ellwood has told the BBC.
Mr Ellwood, who chairs the defence select committee, said "we need leadership" following reports of parties being held in Downing Street while Covid restrictions were in place.
It comes as hundreds of angry constituents contact their MPs.
The government has urged people to reserve judgement until senior civil servant Sue Gray's inquiry is finished.
But Andrew Bridgen, Conservative MP for North West Leicestershire, said: "I don't need to see what Sue Gray says to know that for me Boris Johnson has lost the moral authority to lead the country.
"If there's another emergency where he has to call on the public to make sacrifices, he doesn't have that authority. That makes his position in my book, as prime minister, completely untenable."
Mr Bridgen is the fifth Conservative MP to publicly declare they have written to the chairman of the 1922 Committee – which organises Tory leadership contests – to say they have no confidence in the prime minister. Fifty-four Conservative MPs have to write a letter to trigger a vote.
Ahead of Boris Johnson's apology on Wednesday, Mr Ellwood, MP for Bournemouth East and one-time defence minister, said Mr Johnson needed to "show some contrition" and get a grip of the situation – or he would be "out of office".
Another former minister told the BBC: "Johnson is toast… if you were the chief whip looking at him you'd say he's not fit to do any other jobs in government, you wouldn't make him a junior minister, he doesn't work hard enough."
And a senior Tory MP said "there is a lot of scepticism around that there is anyone ready to take the reins. That buys Boris time. But he shouldn't confuse that with another chance."
Some Tory MPs said their inboxes have filled up after No 10 apologised to the Queen for two staff parties the night before the Duke of Edinburgh's funeral.
Mr Johnson was not at either of those parties last April – but he faces questions over alleged Covid rule-breaking at No 10.
One senior backbench Conservative MP told the BBC they received more than 200 angry emails about the parties, with only five messages in support of the prime minister.
"Many colleagues now believe Boris won't be leader at next general election… for many of us this feels terminal," the MP said.
A Midlands Tory MP, who won his seat in a former Labour constituency in the 2019 election, said: "The inbox is bad, really bad."
- Kuenssberg: What's a PM to do when even kids joke about his future?
- Will two Tory tribes go to war with the PM?
- No 10 parties: How many wine bottles fit in a suitcase?
The gatherings which Downing Street has apologised to Buckingham Palace for took place on 16 April 2021.
The two parties last April involved about 30 people in total, and are reported to have converged at some point in the Downing Street garden, where they continued past midnight.
Staff were reportedly sent to a nearby shop with a suitcase, which was brought back "filled with bottles of wine".
At the time, England was under "step two" restrictions that stipulated people could not socialise indoors, except with those from their household or support bubble. People could socialise outdoors in groups of up to six people or two households.
The behaviour of No 10 staff has been contrasted with pictures of the Queen sitting alone at the Duke of Edinburgh's funeral because of Covid restrictions.
Boris Johnson's spokesman said it was "deeply regrettable that this took place at a time of national mourning".
Sue Gray is carrying out the investigation into alleged Covid rule breaking in Downing Street and government departments.
Labour leader Sir Keir Starmer said it was now in the "national interest" for the prime minister to resign, saying he had lost his "moral authority".
The Liberal Democrats and the SNP also called for Mr Johnson to resign after he admitted attending a drinks party in the Downing Street garden during lockdown on 20 May 2020. The prime minister said he "believed implicitly that this was a work event".
Speaking to BBC's Politics North, pensions minister Guy Opperman said Mr Johnson's behaviour had been "unacceptable".
The Conservative MP said he felt particularly emotional about the party held in the Downing Street garden on 20 May because around that time Covid restrictions had prevented him from being with his wife and twins when they were ill in hospital. The two boys subsequently died shortly after birth.
Mr Opperman said the prime minister should continue in post while the investigation into government gatherings was completed. But he said Mr Johnson needed to "change his ways" and run Downing Street "in a very, very different way".
It comes as the former head of the government's Covid taskforce has apologised for having leaving drinks in the Cabinet Office when she left the civil service on 17 December 2020, when strict restrictions were in place in England.
Kate Josephs, who is now the chief executive of Sheffield City Council, said she was "truly sorry" for "the anger that people will feel".
More than 40 people were on the guest list, among them senior Downing Street staff and civil servants – and the event appears to have been held in a single room, BBC Newsnight has learned.
No 10 said the inquiry would establish the facts around the nature of the gatherings, including attendance, the setting and purpose – and make them public "in due course".
Timeline: The alleged government gatherings
The government is facing mounting pressure over several events that are alleged to have been held during lockdowns. Here is what we know about them and the restrictions in place at the time:
Boris Johnson announced a plan to take the “first careful steps" out of the lockdown that began in March 2020. But he said people should continue to "obey the rules on social distancing and to enforce those rules we will increase the fines for the small minority who break them”.
Legal restrictions at the time said you could not leave your house without a reasonable excuse and government guidance was that you could meet one person outside of your household in an outdoor setting while exercising.
A photo from May 2020 showed the prime minister and his staff with bottles of wine and a cheeseboard in the Downing Street garden. When asked about it, Boris Johnson said, “those people were at work talking about work”.
About 100 people were invited by email to “socially distanced drinks in the No 10 garden” on behalf of the prime minister’s principal private secretary, Martin Reynolds.
Witnesses told the BBC the PM and his wife were among about 30 people who attended.
Boris Johnson has confirmed he attended the event, saying he was there for 25 minutes and “believed implicitly that this was a work event”.
Boris Johnson announced plans for a “significant return to normality" in England by Christmas "through targeted, local action” instead of national lockdowns.
But he added that the timetable relied on “every one of us staying alert and acting responsibly”.
With cases of coronavirus rising again, the prime minister told people in England that “we are once again asking you to stay at home” as a new national lockdown began.
He said people should only leave their homes “for work if you can’t work from home, for education, and for essential activities and emergencies”. Indoor gatherings with other households were banned, unless they were for work purposes.
Sources told the BBC that Downing Street staff members attended a gathering with Carrie Johnson in the flat where she and the prime minister live. A spokesman for Mrs Johnson denies the party took place.
A leaving event was held for No 10 aide, Cleo Watson, where people were drinking, and Mr Johnson made a speech, according to sources.
The second national lockdown ended after four weeks but Boris Johnson replaced those restrictions with “tough tiers to keep this virus down”.
London was placed in tier two, which banned two or more people from different households from meeting indoors, unless “reasonably necessary” for work purposes.
The Department for Education has confirmed it had an office gathering to thank staff for their work during the pandemic. It says drinks and snacks were brought by those who attended and no outside guests or support staff were invited.
The Conservative Party has admitted that an “unauthorised gathering” took place at its HQ in Westminster. It was held by the team of the party's London-mayoral candidate, Shaun Bailey, who has since stepped down as chair of the London Assembly police and crime committee. The Metropolitan Police is to speak to two people who attended the party.
Multiple sources have told the BBC there was a Christmas quiz for No 10 staff last year. A photo – published by the Sunday Mirror – showed Boris Johnson taking part and sitting between two colleagues in No 10. Mr Johnson has denied any wrongdoing.
London moved into the highest tier of restrictions and Matt Hancock, who was health secretary at the time, said it was important “everyone is cautious” ahead of the festive period.
The Department for Transport apologised after confirming reports of a party in its offices that day, calling it “inappropriate" and an "error of judgment” by staff.
A leaving party was held at the Cabinet Office for the outgoing head of the civil service Covid taskforce – the team responsible for drawing up coronavirus restrictions.
Kate Josephs, now chief executive of Sheffield City Council, apologised for the event, saying she was “truly sorry that I did this and for the anger that people will feel as a result”.
Downing Street originally denied a report by the Daily Mirror that a party took place in Downing Street.
However, a video obtained by ITV News showed the prime minister's then-press secretary Allegra Stratton, joking about reports of an event, saying: “This fictional party was a business meeting and it was not socially distanced.”
Lockdown restrictions were eased in England, with pubs and restaurants allowed to reopen with outdoor service only.
However, working from home continued to be recommended and socialising indoors with people from other households was not allowed. Meeting others outdoors was limited to groups of six people or two households.
Two parties were held by Downing Street staff at No 10, the night before Prince Philip's funeral.
One of the events was a leaving party for the PM's then director of communications James Slack, who has apologised for the event and acknowledged it “should not have happened at the time that it did”.
Boris Johnson was not at either party.