A day after Rep. Paul Ryan (R-WI) said he would run for Speaker if different factions of the House GOP endorsed him, more conservative lawmakers weren't rushing to jump on his bandwagon, worried that Ryan won't be much different than outgoing Speaker John Boehner.

"Many of us really have the sense that Paul Ryan doesn't want the job," said Rep. John Fleming (R-LA), one of many House Freedom Caucus members who raised questions about Ryan.

"If the candidates for Speaker remain as they are, I will be voting for Daniel Webster," said Rep. Mo Brooks (R-AL), another more conservative GOP lawmaker who expressed misgivings about Ryan.

Tea Party lawmakers were especially worried about one of Ryan's conditions to take the job of Speaker, which would limit parliamentary motions on the House floor by conservatives to toss out their leadership.

"I think that would be a mistake," said Rep. Ted Yoho (R-FL), as he was pursued by a pack of reporters following a closed door meeting of Republicans on Wednesday morning.

The reluctance of more conservative members to get on board with Ryan left a number of GOP lawmakers frustrated, as they argued that Ryan is a perfect choice for the party.

"I hope they understand that if we're going to be a team, a team has to practice together," said Rep. Markwayne Mullin (R-OK), who said he had no idea if the House Freedom Caucus members would stick with their endorsement of Webster, and not back Ryan for Speaker.

It's very hard to predict people that are unpredictable," Mullin told me.

But others in the GOP were optimistic.

"I think we will get Paul Ryan over the finish line," said Rep. Darrell Issa (R-CA), who has flirted with his own bid for Speaker.

As for the current occupant of that job, Speaker John Boehner set the internal GOP nomination vote for next Wednesday, October 28, with a vote on the floor for Thursday October 29.

"I think Paul Ryan would make a great Speaker," Boehner told reporters at a news conference, "but this decision is up to the members."

Asked what the backup plan would be if Ryan is rejected, Boehner gave a frank answer.

"Don't know," the Speaker said with a shrug.

And for now, it's not clear whether all wings of the GOP will give Ryan their blessing.