Yes, site accessibility is an indirect ranking factor, but it can only negatively influence rankings. Site accessibility is part of the crawl budget: if a server if fast, Googlebot will crawl faster; if a server is slow, Googlebot will crawl slower.

If a page on the website responds with a 5xx error, Google will remove it from their indexed after trying to access it several times. Here’s a comment from John Mueller regarding the site accessibility:

“With regards to this situation where maybe 500 errors were showing or the server was down, that’s something that when we recrawl those pages, we’ll be able to take that into account again and index them and rank them as we did before.”

If Googlebot can’t access your website because of server availability, for extended periods of time, it will demote its rankings, and eventually drop it from their index. As Matt Cutts explains in this video, Google has to balance between short downtimes (i.e. minutes) and long downtimes (i.e. days or weeks).

“if that web server has just died and that domain is going away and is never coming back, well, clearly at some point you want to drop it from the index, and never show it again” source (Matt Cutts)