NEW YORK STATE -- The governor acknowledged Tuesday it wouldn't be easy for lawmakers to reach a budget agreement on time and, as always, there are obstacles to doing it, not the least of which is deciding who and what to tax.

Assembly Speaker Sheldon Silver has made it no secret he thinks income taxes of some sort should be on the table during budget negotiations.

"People who earn more money will be called upon to pay a little more," Silver said on Tuesday.

Silver's suggesting a broad based income tax increase that would include more than just the state's very wealthy. Senate democrats haven't ruled the idea out either.

"We've got to do something to raise revenue and we can't cut our way back into prosperity," Silver said on Tuesday.

Senate Republicans, though, say taxing the wealthy won't work.

"I hope the Governor stands firm in opposing any broad based tax, especially increasing the personal income tax which would be devastating in our state," said Senate Majority Leader Dean Skelos on Tuesday.

While Paterson says he opposes an income tax increase, he's proposed eliminating tax exemptions that benefit wealthier residents, which is in essence a tax hike. He also included scores of taxes and fees in his budget proposal critics call ridiculous and some even say may serve an ulterior motive.

"Are you proposing something this crazy as a way of inviting the legislature to impose the kind of income tax increase you claim you're opposed to?" asked E.J. McMahon, Empire Center for New York State Policy Executive Director.

Supporters of a wealthy tax hope that's exactly Paterson's plan, arguing that when you tax the rich, you're only taking from the part of their income they don't need to spend. And most New Yorkers, polled recently by the Siena Research Institute, say they'd prefer to see the rich taxed more. McMahon says broad based taxes at this time would be extremely damaging and the governor has said he agrees.

"If you're going to tax, you've got to do it at times when your revenue is growing,” Paterson said on Tuesday. “If you start taxing at times when they're receding, you will drive job creators out of the state."

Still, the governor says he hasn't completely ruled out taxing the rich. And the legislature, which has left Albany for now, still has months of negotiations ahead and years of bad spending habits to tackle.

"You gotta choose among a combination of bad or tough results and to me, it's not yet clear the legislature is ready to make that choice," McMahon said.

Legislative leaders have already said they're open to holding joint public hearings and conferences on the budget, but that process likely won't start until after session reconvenes on January 7th.