Avinash Kaushik has a typically detailed and clear-headed take on website success metrics: namely, the Key Performance Indicators that you select can either save or destroy your business, because what you choose to measure (and pay attention to) defines you as either a customer-centric or customer-hostile organization.
It's a good point and well worth reading, but it exposes a larger truth that I find equally perplexing: that most organizations have a very hard time defining success in the first place.
Let's say you're running a site and carefully gathering metrics. Let's say that you're also lucky or unlucky enough to be running an internal site, so you can loop in Active Directory and know, not only how many people accessed a page, but what department they work for, which office they're in, and other metrics that allow you to translate clicks and page views into concrete business scenarios. You're swimming in quality metrics, so you must certainly have a good handle on site success, right?
Not necessarily. Let's say there are two employees who sit next to each other: Bob and Sally. Bob thinks that, for any good intranet, the employee is the customer. Bob knows that employees come to the intranet to get their jobs done, so his success metrics focus on site paths to critical resources; the shorter the path the better the metric, since an employee who's clicking around looking for something is not a happy, productive employee. Sally, though, knows that the #1 customer in her world is the CCO, and the CCO's top priority is that employees understand what a great job the company is doing. Sally's success metrics will relate, first, to the number of articles that were published to the home page (getting the message out) and, second, the number of page views that those articles receive. The more page views the better the metric, since in Sally's world that translates into an audience actively engaged with her messaging.
Two definitions of success, both defensible, result in two success metrics that could not be more different.
Avinash underscores the need to choose performance indicators that orient your business in the proper direction, but in my experience a company only rarely will have a single definition of site success. There are many versions of success, some of which can vary from employee to employee, and that complicates the success scenario tremendously.
One truth is undeniable, though: you cannot pretend to build or run a site successfully unless you pull up your socks and engage in the hard work of defining your success metrics. What is your site home page trying to accomplish? What measurable action counts as a success within that scenario? If you get everyone in the room and lead them through that discussion, the various definitions of success have a chance to bubble up to the surface, where they can be reconciled or otherwise dealt with. Without the success metric conversation, the best you're doing is punching the clock and cashing your paycheck, which is good enough for some people but (I trust) not enough for you.