The near-complete official results from Tuesday's election gave ex-military chief Benny Gantz's centrist Blue and White alliance the most seats, with 33 out of parliament's 120.
Netanyahu's right-wing Likud won 31 seats, but neither had an obvious path to a majority coalition.
In a stark admission on Thursday, Netanyahu acknowledged he was unable to form a right-wing coalition as he hoped and called on Gantz to join him in a unity government instead.
Gantz responded by saying he would have to be prime minister in a unity government since Blue and White was the largest party.
The standoff has even raised the possibility of yet another election - which would be the third to be held in a year following April polls that also ended inconclusively.
| Twitter Post |
Israeli President Reuven Rivlin plans to begin consultations with all parties voted into parliament on Sunday to decide who to choose to try to form a government.
Final results will be published on Wednesday and there could be changes before then, Israel's election committee said.
The committee said the results did not include 14 polling stations where verifications were still ongoing.
Israeli media said 99.8 percent of the votes had been counted.
The third-largest total was the mainly Arab Joint List alliance, which won 13 seats.
Its strong showing opened the possibility that its leader Ayman Odeh could become the country's first Arab opposition chief if Likud and Blue and White form a unity government.
A number of analysts attributed the Joint List breakthrough to anger with Netanyahu over actions and rhetoric seen as demonising the country's Arab population helping spur turnout.
'Won the election'
Following the Joint List, the country's two main ultra-Orthodox Jewish parties, Shas and United Torah Judaism, won nine and eight seats respectively.
Ex-defence minister Avigdor Lieberman's nationalist Yisrael Beitenu party also won eight seats, and he could potentially play a kingmaker role.
He has called for a unity government between his party, Likud and Blue and White, excluding the ultra-Orthodox parties, which he accuses of seeking to force religious law onto the country's secular population.
But the main focus in the coming days will be on Netanyahu, Israel's longest-serving prime minister but who is facing potential corruption charges in the weeks ahead.
He was widely seen as hoping to have the new parliament grant him immunity from prosecution, but that may now be impossible even if he does manage to remain premier.
Read more: Trump snubs weakened Netanyahu after election: 'Our relationship is with Israel'
"During the elections, I called for the establishment of a right-wing government," Netanyahu said in a video message on Thursday.
"But unfortunately the election results show that this is not possible."
He went on to call on Gantz to form a "broad unity government today."
The two men shook hands when they crossed paths at a memorial event later in the day, but Gantz made his position clear on Thursday afternoon.
Gantz too called for a unity government, but he insisted that he be prime minister.
"Blue and White, headed by me, has won the election," he said. "Blue and White is the largest party."
The results were the latest sign of Netanyahu's long dominance of Israeli politics weakening.
He suffered one of the biggest defeats of his political career after the April election.
His Likud along with its right-wing and religious allies won a majority, but he failed to form a coalition and opted for a second election rather than risk having Rivlin choose someone else to try.
The 69-year-old's survival as prime minister was far from being ruled out, however. With his sharp political instincts, he has repeatedly bested rivals in the past.