How Are The Challenges Different For Internet DBAs?

Roger Schrag, Database Specialists, Inc.
http://www.dbspecialists.com

Abstract

I've been an internet DBA for five years, tuning databases and designing schemas for the likes of MSN LinkExchange, WebShopper.com, and NetGuide Live. In a previous life I administered databases and built applications for Fortune 500 companies like GE Capital and Oracle Corporation. If the internet is changing everything, it sure is changing the life of the DBA.

Being a DBA in the internet and e-commerce space is very different from being a DBA in traditional corporate America. For one thing, technical challenges facing internet DBAs are often quite different. Connection caching and replication servers for reporting become incredibly important. But the differences go deeper than that. The corporate culture and work environment is totally different when you are riding the crest of the internet tidal wave. Organizational charts give way to cowboys and idol worship. Plush corporate digs give way to Office Depot desks sagging under the weight of a hefty server that really should be in an air conditioned room, anyway. Yes, the internet is changing everything for DBAs, too.

Introduction

I began working with Oracle technology in 1989 as an application developer at Oracle Corporation in the applications division. If you had a copy of Oracle Financials release 8 or 9 lying around, you would find my name all over the scripts for Oracle Payables. From Oracle Corporation I went on to database administration at large companies headquartered in San Francisco, such as Genstar, a division of the behemoth General Electric Corporation. I had the usual DBA duties: architecture and design of new systems, production support, pager duty, performance tuning, upgrades, and so on.

In 1995 I joined the ranks of sweat-equity workers by administering Oracle databases for a fledgling internet startup in exchange for stock options and a miniscule salary. This venture didn't go far (actually, it went straight into chapter 11 bankruptcy), but I got my first taste of being an internet DBA. I had caught the bug, and there was no turning back.

Today I am a leader at Database Specialists, Inc. We are a group of senior-level Oracle DBAs who administer Oracle databases for many hot e-businesses and internet startups located in the San Francisco Bay Area--ground zero for the internet explosion. All of the DBAs in our group started out working on projects for Fortune 500 or traditional brick-and-mortar companies. We've all made the transition from traditional DBA to internet DBA, and each time somebody new joins our team and makes the transition, I realize just how much the internet is changing the world of database administration.

In this presentation we will first look at technological challenges facing internet DBAs and some basic tips for tackling them. We'll look at topics such as educating team members new to Oracle technology, true 24 by 7 operations, connectivity management, separate environments for development and data mining, and scalability issues. Then we will move on to the non-technical side of the house, considering topics like company culture at startups and the "coolness" factor, the effects of companies trying to do without an experienced DBA, and physical work environments.

The Technology Side

When looking at the technology requirements on our internet database projects, certain commonalities appear. The hottest internet companies today are defining the future of electronic business. These companies are inventing their own business models and creating new service offerings that would have been unimaginable ten years ago. Because most internet companies are breaking new ground, they can't simply purchase shrink-wrapped applications to drive their business. Most of the systems for these companies need to be custom built.

And what programming languages or tool sets are these companies using to build their applications? On the internet projects our group has participated in, Java, Perl, and C++ are all very popular. Some application framework products like NetDynamics or Cold Fusion are used as well. Oracle's development tools such as Oracle Developer and application servers such as Oracle Application Server have been notably absent.

The lack of Oracle tools and languages on these projects has not been our doing. All of our clients look to Oracle Corporation for a top-notch database server, while looking elsewhere for application servers and middleware. Many people in the internet world are leery of the way Oracle Corporation shifts directions on application development. They point to a litany of present and past Oracle products like Developer/2000, Power Objects, Sedona, Oracle Web Server, WebDB, and others. The internet focus of Oracle 8i and Oracle Corporation's embrace of Java might reverse this trend and encourage internet companies to look to Oracle for more than just its database server.

Internet DBA as Educator

Because many internet companies lack Oracle knowledge, an important job of the internet DBA is to educate members of the project team on the capabilities and strengths of the Oracle database server and related products. Internet DBAs need to be Oracle technology advocates, as well as plain old database advocates and even SQL advocates. At the same time, internet DBAs need to be balanced and work to choose the best solution for a problem and not necessarily the solution most dependent on Oracle technology.

At most of our internet and e-business clients, our DBAs are the only people on the project with more than a basic understanding of Oracle technology, and we are not usually brought in until the project is well underway. Internet DBAs are often "outnumbered" and frequently find themselves fighting for the best solution when developers are proposing complex application development to simply replicate functionality native to the Oracle database server.

For example, developers might know how to program in Java or C++ in their sleep, but know very little SQL and PL/SQL. Often the engineering folks know everything there is to know about their web servers and application framework tools, but at the same time are unfamiliar with basic features of the Oracle database server.

In such environments, there is often a tendency to do as much outside of Oracle as possible. This is probably just an aspect of human nature: when you are in a hurry and don't have time to learn something new, you stick with what you know. The unfortunate side effect is that you end up throwing together your own solution instead of leveraging Oracle technology and taking advantage of highly optimized code several years in the making.

The lack of Oracle expertise during the development cycle has led many e-commerce companies to develop their own LRU cache management code and sorting algorithms. Application-level caches can sometimes be useful, but often they could be avoided by exploiting the buffer cache and library cache in Oracle's SGA. Sorting routines, meanwhile, should be totally unnecessary because Oracle's native sorting mechanism is very efficient and scalable, and incredibly tunable.

Along these same lines, many internet companies have resisted the use of PL/SQL to some degree. It is a language that their developers do not know. Many people turn their noses up at PL/SQL because it isn't object oriented, isn't compiled into machine code, or any of a number of other reasons. While all of these claims may be true, PL/SQL can still be a valuable part of a database application because of its tight database integration and execution within the database server address space. PL/SQL is just another Oracle feature that many internet companies, left to their own devices, would not choose to leverage.

24 by 7 Operation

Many so-called 24 by 7 systems in the traditional corporate world can be taken offline occasionally for maintenance operations. For example, some shops designate a period every Saturday for systems to be taken offline. But the internet never closes. When sites like eBay or E*Trade go down, Wall Street talks about it for days. Internet DBAs, therefore, have to go to great lengths to minimize service disruption. This includes taking advantage of dynamic instance parameters, using space management techniques that minimize fragmentation, having foresight to anticipate future needs and plan for them up front, and being creative in order to resolve problems without taking the database down.

Dynamic Instance Parameters

In older releases of Oracle, all instance parameters were static. When you started an Oracle instance, the parameter file (initSID.ora) was read once and no parameters could be adjusted without editing the file and restarting the instance. Starting in Oracle 7.3, some parameters were made dynamic. You can change the settings of these parameters without restarting the entire instance by issuing an ALTER SYSTEM or ALTER SESSION command. In Oracle 8.0 and Oracle 8i, more parameters have been made dynamic.

For example, if you discover a performance problem with sorts, you may want to adjust the sort_area_size, sort_area_retained_size, and other initialization parameters relevant to sorts. To do this you would edit the parameter file. However, you would not need to restart the instance in order to put the new parameter values in effect--you could issue a series of ALTER SYSTEM commands instead.

Oracle 8i has over 80 parameters that can be changed dynamically. This allows you to tune sorts, log archiving, parallelism, and many other areas of database operation without disrupting services by restarting the entire instance.

Space Management Techniques Minimizing Fragmentation

In the old days, conventional DBA wisdom was to size every table and index so that it would fit into one extent. When segments reached more than perhaps 10 extents, they were said to be fragmented and it was time to "reorg" the tablespace by exporting tables, dropping them, and rebuilding them with larger extent sizes. Shortly after the introduction of Oracle 7.3, some interesting white papers came out of Oracle Corporation describing a new philosophy on space management. This strategy, in a nutshell, is to choose one uniform extent size per tablespace and use it consistently for all segments within the tablespace. The theory is that it isn't really a big deal for one segment to have several extents, but by using consistent extent sizes you eliminate free space fragmentation.

The "one extent size fits all" approach works very well in the internet world. There is never time to shut down the database for a reorg, and often when creating new tables or indexes you simply don't know how large they will grow. An initial and next extent size of 4 Mb works well for tables ranging in size from zero bytes up to about 2 Gb. That is a pretty broad range. A table bigger than that should probably be partitioned anyway.

Using a uniform extent size of 4 Mb (and pctfree of zero) allows internet DBAs to create objects without having to know necessarily how large they might grow, and avoids the need for costly reorgs down the road. Even if segments are repeatedly created and dropped or indexes are rebuilt, free space does not become fragmented.

Anticipating Future Needs

Foresight is very important to the internet DBA. During development and testing cycles it usually isn't too difficult to arrange a database down time. The moment an application goes live, however, you say goodbye forever to the luxury of being able to take the database down for maintenance. It's a "speak now or forever hold your peace" sort of arrangement. Internet DBAs must anticipate needs that may develop down the road, and prepare for them before restrictions on database down time go into effect.

Here is a simple example: One of our clients operates a family of web sites that each offers a different, although related, online service. Most of these sites have a separate Oracle database on the back end in order to offer some degree of autonomy. (Database links and replication, however, allow them to share common data and give a cohesive, integrated feel to the end user.) At one point a new service was rolled out that required a new database to be built on a new server. The backup strategy for this new database was to take hot backups across the dedicated network to a tape backup server, but the mechanism was not yet in place on launch day.

If we launched with the database in ARCHIVELOG mode, then the database would turn out archived redo logs and the archive log destination would quickly fill because we had no software in place yet to back up logs to tape and purge them from disk. If we launched in NOARCHIVELOG mode, then at some point in the future we would need to take the database down in order to switch to ARCHIVELOG mode. Sure, we would only need to take the database offline for a minute, but this would still amount to a service disruption.

In the end, we launched with the database in ARCHIVELOG mode and a silly daemon that ran once an hour and purged the archived redo logs. You could say this was a waste of CPU cycles, but we were able to implement proper backups a month later without any service disruption.

Avoiding Down Time By Being Creative

In Oracle, many maintenance operations can be performed with the database online. Some operations require a little bit of ingenuity in order to minimize service disruption. Suppose you want to move your rollback segments to a separate disk in order to better balance I/O. Can you do it without your end users being impacted? Sure. Create a new tablespace with data files in the new location. Create your new rollback segments and bring them online. Bring all of the existing rollback segments in the old rollback segment tablespace offline. You'll have to wait for pending transactions to complete, but eventually the old rollback segments will go offline. Then you can drop the old rollback segments and the now-empty tablespace. Of course, don't forget to update the instance parameter file if you changed the names of the rollback segments that are to be online at instance startup.

Along those same lines, indexes can be relocated while the database is online with the ALTER INDEX REBUILD command. Depending on the index type and the version of Oracle you are running, this could put an exclusive lock on the indexed table. However, index rebuilds can be fast since no sort is required. Having transactions that are delayed while waiting for a lock is usually better than taking the entire site offline for a period of time.

Connectivity and Session Management

Database connection and session management is another recurring issue for internet DBAs. Connecting to an Oracle database is relatively expensive: Typically names need to be resolved to IP addresses and port numbers. On Unix a separate process with a huge address space must be created and this process must attach to the SGA. The user must be authenticated.

Meanwhile, because of the stateless nature of the HTTP protocol, database sessions on internet-driven applications tend to be extremely brief. If applications were to create a new Oracle database connection for each HTTP request that requires database access, the system would not be very scalable and response time would be slowed by the overhead of establishing a new database connection.

Some application frameworks offer connection management, where a pool of persistent database connections is established up front and each database request is funneled through one of these connections. Oracle Application Server takes this one step further by providing a mechanism to allow one database transaction to span multiple HTTP requests. Unfortunately, many decision makers in the internet space seem to be unaware of Oracle Application Server or disinterested in what it does; only one of our current internet projects uses Oracle Application Server.

A surprising number of internet companies have chosen to develop their own database connection management systems in-house. Typically, they will use Java or Perl to encapsulate database access into a class or a module and will invent their own API. All application developers will use this Java class or Perl module to access the database.

Connection pooling systems, especially simplistic homegrown ones, can lead to some interesting problems. Consider a naughty application that sometimes forgets to close a cursor if it crashes. If a database connection is persistent for the life of the database instance and an application using the connection crashes without closing a cursor, a pile of forgotten open cursors will develop over time. Eventually the session will hit the maximum number of open cursors specified by the open_cursors parameter, or worse yet the instance might run out of enqueue resources or something even more obscure. A similar problem can occur with PL/SQL file handles, and with other limited resources.

It can be very hard to make application code bulletproof. If a reliable mechanism cannot be developed to free up all database resources when an exception occurs, then consider having the connection pooling agent cycle connections periodically so that Oracle can free up any resources that were leaked during the life of the session.

Other oddities can happen when applications share database connections. Suppose one application developer issues an ALTER SESSION SET NLS_DATE_FORMAT command. This session change will still be in effect when another application goes to use the database connection. Furthermore, the application that altered the session might find that its next SQL statement is processed using a different NLS date format because a different database connection was used.

Connection pooling can also make application tuning more difficult for the internet DBA. In some pooling mechanisms, two consecutive transactions from the same application could be processed by different database connections within the pool. This causes each database session to include a mish-mash of transactions from different applications. Isolating the activities of one application, or tracing database calls within a certain portion of code, can be difficult in such an environment.

Homegrown connection pooling middleware can also introduce limitations that can reduce performance or prevent internet DBAs from deploying performance boosting techniques. For example, one of our clients built their own connection pooler in Perl. Since the developer who wrote the code didn't know what bind variables were at the time, no support for bind variables was included in the connection pooling API. Today, dozens of distinct statements are parsed and executed each second on their database server that would be identical if bind variables could be used. Their production database server spends more time parsing SQL statements than executing them, but the developers are too busy to add bind variable support to their connection pooler.

Databases For Development, Testing, and Reporting

Volume testing can be hard with internet-based applications because it can be hard to simulate huge populations of concurrent users accessing the application via the world wide web. Not only can the randomness of a large user community be hard to simulate, but most internet applications are brand new, one-of-a-kind concepts and it can be hard to predict how the public will use them. Furthermore, it can be hard to predict just how many people will actually use the application.

"Robots" are not hard to write, but they usually follow very predictable patterns. One of our clients wanted to load test their database and application before production rollout. They wrote a robot that caused constant application hits at the rate of 30,000 per hour. However, the robots repeated the same series of requests over and over. After the first few hundred hits, all of the application SQL and data blocks were cached in the SGA and not one single disk I/O occurred after that. The performance results were flattering, but not realistic.

E-business systems, particularly at internet startups, often begin as very small, simple systems used by a very small number of people. Typically there will be developers and a few marketers showing off prototypes to potential investors. Perhaps this helps explain why so many e-businesses use one Oracle database for production, development, testing, and data mining. It is not uncommon for most developers to know all of the database passwords, and to manipulate data in the production database on a regular basis.

Using one database for so many purposes does keep the infrastructure simple and reduces the number of databases that the DBA needs to look after. However, the benefits end right there. As internet companies grow, they typically get burned by development efforts bringing down the production application, or data mining efforts consuming too much resource and slowing down the production site. After one or two really good burns, internet companies start to think about splitting out development and data mining onto separate databases from production.

Development and testing databases are usually built by cloning production on a periodic basis. Databases for data mining are a bit more tricky. The data needs to be complete and up to date. If not, people will go back to running their reports off of production. (Remember that passwords are often widely known.) For this reason, data replication of one form or another is often used to build reporting databases for data mining. This allows data analysis without impacting production performance.

Basic replication, using read-only snapshots or distributed transactions and custom PL/SQL code, is frequently used to populate a reporting database and keep it current. A new feature of Oracle 8i is that a standby database may be opened in read-only mode without losing standby capability. This would allow Oracle users to have a database serve as both a standby database for failover purposes, and a part-time reporting database.

Backups and Fault Tolerance

Backups and site failover are thorny subjects at internet companies. On the one hand the internet is always open and e-businesses cannot afford to have their site down. This leads to talk about standby databases, mirror sites, and failover mechanisms. On the other hand, disaster planning is preparing for the future and many internet companies focus on the present instead of the future because the internet is changing so fast.

We have been called in to participate on a surprising number of projects where no disaster recovery plan had been established and basic database backups were not being taken. If our clients are any indication, it seems like internet companies often push backups and fault tolerance to the back burner in favor of more immediately pressing issues. The challenge to the internet DBA becomes two-fold: First, convince others of the importance of backups, and second, devise a backup and recovery plan that is simple enough that it actually stands a chance of getting implemented.

Hot backups are a must for e-commerce databases, because down time for a cold backup is not possible and export files would take too long to recover and be too loss-prone. Because Oracle 7 did not include an integrated backup manager, most internet companies that do back up their Oracle7 databases use homegrown scripts to do so. Since Oracle 8.0 and Oracle 8i include Recovery Manager and Legato Storage Manager, it is now possible to have better managed backups without the need for homegrown backup scripts. However, many internet companies are not setting up Recovery Manager as part of their initial production database deployment.

Standby databases appeal to many internet companies. Now that Oracle 8i allows a standby database to be opened in read-only mode without losing standby capabilities, this appeal is even greater. A standby database represents a relatively inexpensive way to have a nearly up to date backup database that can be put into production very quickly in the event of a failure on the production database server.

The project manager on one of our e-commerce projects had the foresight to request a standby database from the get-go. One month after the e-business opened its doors on the world wide web, the power supply on the database server failed and left the production database dead in the water. The standby database was pressed into service and the site was back online quickly.

Oracle Parallel Server, or OPS, has a certain initial appeal to many internet projects as another potential disaster recovery mechanism. We'll have two database servers, the thinking goes, and that way the site will still be up even if one fails. While most of our clients have asked about Parallel Server at one point or another, none of our internet projects have actually deployed on OPS. With OPS the multiple database servers share one disk storage system, leaving a huge single point of failure. OPS is also a separately licensed product only available with the Enterprise Edition of Oracle. OPS administration is also a very specialized skill set. All of our clients have a difficult time finding qualified DBAs--the specter of finding a good DBA who also has OPS experience can be frightening. And finally, prior to Oracle 8i the "server pinging" problem caused performance degradation when multiple Oracle instances contended for the same data blocks. ("Cache fusion" in Oracle 8i is supposed to address this last problem.)

On many internet projects a standby database or other disaster recovery mechanism is seen as a "wish list" item which isn't tackled until well after the site is in production if at all. A surprising number of internet companies believe that using RAID storage and perhaps taking an occasional database export represents a sufficient backup strategy.

Scalability and Extensibility

System scalability can be mission critical in an internet based application. Indeed, one of the reasons internet companies choose Oracle technology is because of its scalability. Most internet applications start out simple, with small and lightly used databases behind them. However, an internet company only needs to sign one partnership deal in order to suddenly have huge amounts of traffic guided its way. Internet DBAs need to leverage Oracle technology in order to make systems scalable.

I helped an internet company migrate its database from MySQL to Oracle. MySQL is a free, open-source SQL database. These folks built their application using MySQL largely because of the price. The service provided by this application proved very popular and the web site was seeing a lot of hits. Unfortunately, MySQL does not have row-level locking or support for hot backups. These limitations among several others sharply stunted the potential growth of the free web-based service offered by the application. The company chose to migrate the application to Oracle in order to boost scalability.

Extensible designs have special meaning in internet applications. Because business models and strategic directions frequently change in the internet arena, applications and databases are often deployed in ways not originally intended. The object oriented nature of Java and the modularity of Perl can sometimes allow wholesale changes to be made to an application without a lot of effort. Working extensibility and flexibility into a database schema or server side code can be very helpful later on in the lifecycle of the system.

The Impact of Funding and Budgets on Technology Decisions

When it comes to finances, the internet companies we deal with can easily be divided into two categories: There are the young startups working on shoestring budgets, and there are the well-funded companies that are able to spend money in order to get things done right. The projects with minimal funding are usually being financed by the owners or their angels, or nervous venture capitalists who are minding the purse very closely. The projects that are well-funded tend to belong to either established companies that existed before the concept of e-business exploded, or companies that have been gobbled up by technology giants.

Obviously the better funded projects are in many ways more fun and challenging for the internet DBA. This is not to say that companies are blowing tons of money and we should all join in the spending frenzy. Rather, well-funded projects allow the internet DBA to leverage Oracle technology to the fullest, focus energy on real challenges instead of artificial barriers, and in general "do things right".

Oracle technology is not cheap. Setting aside the topics of price comparisons and whether or not Oracle products are a good value for the money, many internet companies find that Oracle licensing fees form a significant part of their budget. Companies with tighter budgets are more likely to try to get by without important database options or separately licensed products. One internet company paid contractors to write a data replication package so they would not need to purchase Advanced Replication (a separately licensed feature in Oracle 7.3). Another company wrote its own PL/SQL API to simulate data partitioning by separating statistics data into a separate table for each month.

Internet DBAs on poorly funded projects can spend a lot of their time fighting fires caused by inadequate equipment. Disk storage is cheap these days, but there are still a number of reasons why production database servers might be short on space. Of course, the ramifications of running out of disk space can be severe. DBAs can also find that their hands are tied when the computer on their desktop is not sufficient. Oracle 8i requires an X environment to install on Unix servers, for example, and this can be a thorn in the side for an internet DBA with a simple Windows box on their desk and a database server 200 miles away.

Beyond Technology

So far we have looked at how the internet impacts DBAs with regard to technology. However, the internet is changing more than just the technology that DBAs work with. It goes far beyond that. The internet DBA is faced with a new kind of corporate culture and many twists on the traditional working environment.

Corporate Culture

Internet companies are to some degree the wave of the future. What they lack in monetary resources they make up for in coolness, as in "This is a cool place to work." There is such a profound shortage of skilled IT workers in the San Francisco Bay Area (and probably most of the United States) that companies are in stiff competition to attract the best. Internet companies tend to be lean and mean, very agile and swift moving. They need to be this way to stay on the crest of the wave. Internet companies need very sharp technical people in order to maintain their edge. The coolness thing is part of the package to attract the best.

Free food and drink, a ping pong table in the lunchroom, an onsite masseuse, tickets to ball games and movies... all of this and more can be yours when you join an internet company. The freebies do a few things. For one, they help with the coolness factor. But perhaps more importantly, they help establish social relations between the employees. The folks at the office become more than just your coworkers, they become your best friends. Having your circle of friends at the office makes it easier to spend late nights and weekends at the job. It helps strengthen teamwork and encourage working longer hours.

All of this becomes fascinating when you look at how internet DBAs fit into the mix. In San Francisco there is such a shortage of experienced Oracle DBAs that salaries are outrageous. Experienced DBAs who have been around the block usually seem to prefer market rate pay over the perks of being employed at a cool company. This makes it extremely difficult for poorly funded internet companies to hire experienced Oracle DBAs.

Internet companies usually take one of three different paths with regard to the DBA debacle. Either they pay the piper and hire an experienced DBA at a high salary, or they hire a contractor while they continue the perpetual search for a good DBA who will work for less, or they go with a junior DBA. Companies will try different paths at one time or another. Many start with a junior DBA and solicit a contractor when they discover their system won't scale or otherwise hit a brick wall.

Many companies--and this is probably not limited to the internet space--try to convince themselves that they really don't need an experienced DBA. Internet companies only hire the brightest people, they figure, so why shouldn't their bright people be able to learn what it takes to manage an Oracle database? After all, databases are not rocket science.

Why is it a bad idea to throw the Oracle CD at one developer and crown them the de facto DBA? Part of the answer is time, and another part is breadth of experience. At an internet company, nobody has the time to read documentation or go through weeks of training in order to learn the best ways to do things. This puts pressure on people to merely make things work for now, and not necessarily deploy the best solution over the long haul. About experience, so much is learned about Oracle technology not by sitting in a classroom or reading a book, but by actually applying techniques on real systems. The developer who loaded up the Oracle CD for the first time will have no prior experience to fall back on.

It is not uncommon for internet companies to designate one application developer as "the database guy" or "database gal". This person starts out knowing nothing about Oracle technology, but is looked to for all of the answers. Learning is always a good thing, but some companies will take this to extremes. Some companies would rather see an employee struggle for days to resolve a problem than call in an expert who could resolve it in half an hour. The internet DBAs on our team are often called in after developers have bruised themselves badly by beating themselves against a wall. One client called us in after their project had been stalled for a week over what turned out to be a typographical error in a tnsnames.ora file. They simply didn't know where to look--but an experienced DBA had the problem nailed in five minutes.

Cowboy programming lives on at internet companies, and so does idol worship. Many internet systems are initially built by very small groups of developers, and it isn't uncommon to see one key person being the mastermind behind it all. When this cowboy (or cowgirl) is so highly regarded that everything they say is taken as gospel, this can pose challenges for the internet DBA if it turns out that this person actually has limited knowledge of Oracle technology. The personality issues can get dicey when you find yourself having to justify designs because the lead developer was ignorant (or worse, misinformed) about a basic Oracle concept.

Internet companies tend to be very laid back environments. Many don't have separate test environments or quality assurance teams. To a DBA with experience on projects for more conservative companies, the cavalier way in which untested code often gets thrown into production can be appalling. It's easy to get jaded, but you shouldn't let yourself fall into that trap. Internet DBAs, probably even more than traditional DBAs, need to be that voice of reason, the gatekeeper who asks, "Did you test this code first? What will happen if I put it into production and something goes wrong?"

The Work Environment

Physical work environments for internet DBAs can be quite strange. Many are ergonomic nightmares that can require some creativity to make livable. Phonebooks and outdated Oracle manuals make excellent monitor stands. In the last five years I've had the opportunity to sample bottom-of-the-line folding tables and chairs from all of the major discount office supply chains. And then there was the project in the office space that had no heat for two months in winter...

Modest office environments are often the result of scarce funding at internet startups. But even well-funded and successful internet companies often have overcrowded work spaces. Internet companies often go through explosive growth spurts, and frequently office space is in short supply.

Internet companies try to minimize the pain of substandard or overcrowded working environments by making them "cool" instead. Some people see concrete and steel beams as cool, others see it as rustic. One well-established and successful internet company had become a victim of its own success. The development team had doubled in size overnight and the office was very crowded. While they worked on securing a larger office space and planning a move, they painted the walls of the existing office garish colors in order to brighten up the work environment. Management actually sent an email to all employees identifying the fresh paint as an important step in improving the working environment.

All of this said, however, the uniqueness of the working environment in the internet space goes much deeper than the physical characteristics of the office space. The whole approach to project planning and implementation methodology has been twisted on end by the rate of change in the internet world.

The speed of change in the internet world, and the fact that most internet companies are breaking new ground with their business models or service offerings, create a dizzying pace and level of anarchy. Requirements change constantly. The planning horizon tends to be incredibly short--people think about tomorrow or perhaps next week more than they think about the next six months or year. The internet landscape is changing so fast that by the time you could put together a plan for the next twelve months, it would already be out of date.

This takes its toll on internet DBAs. In the internet world you need to be flexible and agile. You can't get too emotionally attached to today's design, because tomorrow it might be lining the bird cage. Being a good DBA in such a chaotic environment requires a good sense of balance. Nobody wants to implement a sloppy hack-job of a solution, but often the academically pure "best" solution is not feasible either. Internet DBAs need to choose their battles carefully--letting go on the minor issues and standing firm on the more important ones. Implement the most elegant, extensible, scalable solution you can given the constraints, but recognize that you may have to compromise in certain areas.

Another aspect of the work environment at internet companies is the surprisingly low level of access to other team members. An internet DBA needs to work closely with application developers, business analysts, and management. At small internet startups, high-level people are often too busy etching out a business model or meeting with potential alliance partners to go over requirements or plans with the internet DBA. Developers are sometimes hard to get ahold of because they are working against tight deadlines or perhaps they are "virtual" developers--working at home or 1000 miles away.

Internet companies are usually small and seem like they should have very open channels of communication. However, the furious pace of the internet and the flexibility of work schedules and locales can make face to face meetings surprisingly difficult. Internet DBAs must be excellent communicators who can make the most of limited audience time.

In Summary

The internet is changing everything, including what it means to be a database administrator. Internet DBAs face new technical challenges--such as true 24 by 7 operation and a development community that is not always as well versed in Oracle technology as it could be. But it goes deeper than that. The internet is changing the corporate culture and the complete work environment for the DBA. "Analysis and design go straight out the window," as one of my colleagues put it. Flexibility, extensibility, scalability, and leveraging existing technologies all become the names of the game.

The author of this presentation and his group at Database Specialists are senior consultants specializing in business solutions that use Oracle technology. They've provided database administration, application development, and architectural guidance to well over a dozen internet startups and e-businesses in the San Francisco SOMA and Silicon Valley areas. You may contact Roger Schrag at rschrag@dbspecialists.com, and you may visit the Database Specialists web site at http://www.dbspecialists.com.