Common solutions used in data centers
How do data centers attempt to address these issues? The short answer is simple: brute labor and/or an attempt at in-house automation using manually written scripts.
There is usually a team of Unix administrators and another team of Windows administrators who are responsible for manually preparing each and every piece of hardware by installing the operating system and patching it to the required level.
These administrators are also responsible for resolving issues with the systems they provision, such as missing pieces in the installation or performance issues that may be due to improper setup of the operating system (wrong values supplied for OS properties, for example, network buffer properties).
There is another team of Database Administrators (DBAs). These DBAs may specialize in Oracle or DB2 or SQL Server, and frequently in companies that seek to combine multiple roles, may dabble in all of these. (Indeed in the DBA world, it was once considered a plus point to know as many databases as possible, until the realization dawned that a real expert in one main database was more of a valuable asset than a DBA who knew multiple databases and their nuances, but only superficially.)
These teams of Unix, Windows, database and also the middleware administrators are put into action in their brute numbers, and this is normally seen in the highly-populated countries in the world today where there are a great number of administrators in the job market. The admin labor is available at a low cost in such markets, and consequently more administrators can be hired.
Such administrators, in an effort to be extremely competitive against their peers, and to appear extremely loyal to their work, proudly say "we never sleep" (sacrificing their family happiness in the process) and make themselves available for tackling all the issues mentioned—albeit in a manual, uncontrolled, haphazard manner that would be prone to multiple and deadly mistakes.
However, brute force, by throwing reams of administrators at the manual tasks, does work at fighting fires and keeping them under control. This technique is employed by a number of companies to handle their data centers. But then, they get used to fighting fires every other day.
The other scenario is the company that prides itself on the thousands of reams of scripts running its data center. These countless scripts are used in an attempt to automate the manual steps of managing the data center. They are used for provisioning, to collect the configuration, for patching, for applying the changes to the schemas, for backing up, and for creating and monitoring the standby disaster recovery databases.
However, these scripts are not a magic bullet—there needs to be an effort to write and maintain these scripts. As technology changes, more and more complicated scripts need to be written. The scripts may be layered unnecessarily and may become quickly outdated—for example, an Oracle RMAN script used to back up an Oracle 9i database may still be used to back up an Oracle 10g database, without using the new features such as Block Change Tracking and Fast Incremental Backups, present in the later releases of RMAN.
This is the very problem with scripts—they stay static.
The languages are not easy, and require expertise to write scripts—which is somewhat rare. The writers of such scripts soon establish a position for themselves in the company as heroes. They are available to script everything.
And when these heroes leave the organization, there is chaos.
