Yes, misleading buttons or links are a negative ranking factor, according to Google’s official documentation.

Have you ever clicked on a button that says something, but at click does something completely different, for example, clicking on a “add to cart” tries to download a file on your computer? This behavior is labeled as malicious and is penalized by Google.

Google describes this tactic as:

“Changing or manipulating the location of content on a page, so that when a user thinks they’re clicking on a particular link or button the click is actually registered by a different part of the page” source

It’s surprising that manipulating the browser’s back button to prevent searchers from returning to Google SEPRs is not considered malicious. Google’s John Mueller stated that:

“from a search point of view, I don’t think that would cause any problems.” source

I believe that this should be a negative factor because it creates a really bad user experience.