System Administration is the combination of system support and user support.
Any rule can be modified by the application of power and policy. By contrast, rules always are subordinate to laws.
System support is a subset of network support. Network support is a subset of system support.
There's a difference between system support and user support. There may be overlap in the two positions; sometimes both are done by the same person. But the two tasks are distinct, and sometimes have conflicting goals.
Great system support people often make lousy user support people and vice versa.
The person good enough to do both system support and user support will usually be hired away by a shop where the combined tasks are too large for a single person.
Application-to-application differences confuse everyone, especially users and support staff. Ditto UNIX-to-UNIX differences, etc. By contrast, complete consistency completely stifles improvement.
At any given site for any given application or feature, there's someone who knows more about it than the support staff. Finding that person is the first step to take to diagnose any given problem.
Time to diagnose and time to fix are fix are completely unrelated. Sometimes one approaches zero while the other approaches infinity. This is especially hard to deal with when the diagnostic person and the fix person are not the same.
One person's improved feature is another person's gratuitous change.
Users want applications and systems they can customize.
One user's customization is another user's gratuitous change.
The cost of customization is complexity. The cost of complexity is increased difficulty in administration and user support. The cost of increased difficulty in administration and user support is either lower quality of administration and user support, increased support staff, or both. Therefore increased customization means increased cost or lower quality of support or both.
It doesn't matter whether customization has actually been done. The mere fact that it's possible means you must check for it, thereby increasing the cost of problem diagnosis.
They're not users, they're clients.
-- Kevin Smallwood.
The user who says "Can X be done?" is usually really asking "Would someone please do X?". Make sure you answer both questions.
It's human to blame problems on outside causes. By contrast, an outsider will always suspect the insider as the cause.
The user who says "I didn't change anything" isn't always lying. Sometimes they're just ignorant or forgetful.
It's more important for users to do their job than to answer the needs of admins. Unless of course their job is to answer that need.
For every statement in "Users Are Human", change "user" to "admin" and vice-versa.
Cockpit error is the most common cause of problems. Everybody is a pilot.
One way of cutting costs without cutting development staff is by cutting overhead. System administration and user support are overhead.
User and system admin training are overhead. Not having them increases overhead. Go figure.
"Sure, we can do that. Here's what it'll cost you."
The situation at your site doesn't make you qualified to judge the situation at another site, and vice-versa.
Just because someone else's support staff does it mean your staff can do it. (This statement is subtler than it looks.)
The Summary: Be careful what you do in that vacuum. Nobody appointed you god. However, you can always be dis-appointed.
You can always incrementally add one more.
Sometimes the straw breaks the camels back. More often, the camel just goes slower and slower.
The difficulty of support does not grow linearly with the size of the site.
Eventually your site outstrips your methods, and you must bite the bullet and move to new methods.
Corollary: Nobody bites the bullet until there's not enough time to do the existing work. At that point there's not enough time to make the changes.
Adding a new kind of computer, operating system, application, peripheral, etc, has a much higher administrative cost than adding one more of what you've already got.
Author unknown.
