You can have the most attractive website of all your competitors. You can hire usability experts, professional photographers, and the greatest PHP developers money can buy. If you aren’t on the first page of Google, you might as well be from Mars. Sorry.
While it is true many people do not look past the first page some could be mislead by wasting time on usability... You don't say wasting time but imply it is the wrong focus.
Some reasons why you might want to consider usability important:
1) let even say you are on the first page of Google - just getting them to click on the results and visit your site is probably not the goal. You want them to do something. Not flee your site immediately.
2) Search results are not your only source of visitors. Taking care of those that visit is important (so don't think of search results as the only measure of your success).
3) Inbound links can certainly be impacted positively by having a usable site with valuable content (which will increase those visitors mentioned in item 2) and increase your search ranking.