1980 Called and it Wants Its Deliverables Back

I’ve been doing this data modeling stuff for a really long time.  So long that I just put "20+ years" on my slides…and I’m well past that 20.  However, having done this for a long time does not qualify me as an expert.  What qualifies me is that I know how to adapt my tools, methods and approaches as development tools and methods change.  What worked for projects in 1986 doesn’t necessarily work now. Not only do we have radically more advanced features in our tools, we have many more platforms to support.  We have a greater variety of development approaches.  Our architectures are much more distributed and much more complex than they were in the nineties.

Back in the mid-eighties I worked in the US Defense consulting business.  The somewhat serious joke was that consultants were paid $1 million for each inch of documentation they delivered.  It didn’t matter what was in the models or whether they were correct; it only mattered how many reams of paper were delivered.  That sort of "all documentation is great" mindset still exists well into 1999, long past its usefulness.   The world has changed and we data architects need to find a replacement for the publication fashion equivalent shoulder pads, thongs, leggings, skinny ties, and lace fingerless gloves.

Those differences mean that the deliverables we produce need to be radically different from what they were in 1999.  Our team members are no longer going to open up a 175-page prose document to find out how to use and understand the data model or database.  They don’t want all that great metadata trapped in a MS Word document or, worse, buried in an image.  The can’t easily search those and they can’t import that knowledge into their own models and tools. 

As much as I don’t want this to be true, no one wants to read or use massive narratives any longer.  Sure, if they are really, really stuck a quick write up might be helpful, but sometimes a quick video or screencast would be faster and easier to produce. If you are spending days or weeks writing big wordy documents, the sad truth is there is a high likelihood that absolutely no one except your mother is going to read or appreciate it…and she isn’t actually going to read it.

I’ve significantly changed what I produce when I release a data model.  I constantly monitor how these deliverables are used.  I look at server logs or general usage reports to continually verify that the time I’m spending on data-model related deliverables is adding value to the project and organization.  The main way I gauge usefulness of deliverables is by how urgently my team members start bugging me to get them up on the intranet where they are published. 

Here are my top 10 recommendations for approaching your data model deliverables:

1. Get formal, lab-based hands-on training for staff.  Or use staff that are already trained in the tools and version of the tools they are using. You may be missing out on features in the tools that make publishing data models much easier that the methods you are currently using.  I had a client who was struggling to support an elaborate custom-developed application that they didn’t really know how to use or maintain.  It used a deprecated data model format to build an HTML-based report of the data model.  Sound familiar? Almost all tools provide a feature to generate such reports in seconds. 
2. Move away from very large, manual documentation.   Think in terms of publishing data and models, not documents. Prose documents are costly to produce and maintain.  They do more harm than no documentation at all when they are not maintained.  The are difficult to search, share, and use.  This is not how the vast majority of IT staff want to consume information.  Team members want their data model data (metadata) in a format that is consumable, that can be used on a huge variety of platforms and that is interactive, not locked only in a PDF.
3. Know how long it takes to produce every deliverable.  Having this information makes it easier for you and your team to prioritize each deliverable.  I once had a project manager ask if cutting back the automatically generated reports could save time for getting data modeling completed. I could show her that the total time to put the documents on the intranet was only about 5 minutes.   My document production data also helps other modelers estimate how long a release will take to produce.
4. Stop exporting data for features that can be done right in the tool.  Move data model content that is locked in MS Word documents into the models or stop producing it.  Exporting diagrams as images and marking them up with more images means all that mark-up is unsearchable.  It also means that every change to the data model, even a trivial one, triggers a new requirement to recreate all those images.  Modern tools have drawing and mark-up features in them. Cost/benefit of exporting and annotating outside the modeling tool means you’ll always be spending more than your "earn".  You’re creating a data model deficit.
5. Stop producing deliverables that require a complete manual re-write every time there is a new release.  Unless, of course, these sorts of things are HIGHLY valued by your team and you have evidence that they are used.  I’m betting that while people will say that they love them, they don’t love them as much as getting half as much data architecture work done.
6. Focus on deliverables that are 100% generated from the tools.  Leave manually developed deliverables to short overviews and references to collaborative, user-driven content.   Wikis, knowledgebases, and FAQs are for capturing user-generated or user-initiated content.
7. Focus on delivering documentation that can be used by implementers of the model.  That would be Data Architects, DBAs, and Developers.  Probably in reverse order of that list in priority.  Yes, you want to ensure that there are good definitions and good material so that any data architect in your company can work with the model even after you’ve won the lottery, the largest number of people who will work with the models are business users, developers, and DBAs.  Think of them as your target audience.
8. Automate the generation of those deliverables via features right in the tools – APIs, macros, etc. Challenge every deliverable that will cost more to produce once than the value it will deliver to ever single member.
9. Move supporting content that is locked into MS Word documents (naming standards, modeling standards and such) to a collaborative environment like a wiki or knowledgebase.  Don’t delay releases just to deliver those items.  Theses formal deliverables are important, but from a relative value point of view, they should not hold up delivering data models.
10. Focus now on making a process that is more agile and less expensive to execute while also meeting the needs of the data model users.   If it is taking you more time to publish your data architectures than actually architecting them, you are doing it wrong.

While data modeling standards have stayed relatively stable over the last 10-20 years, our methods and tools haven’t.  If you are still modeling like it’s 1980, you stopped delivering value sometime around 1999.

Changing Tools? Here’s Some Free Advice That’s Paid For

Is your project suffering?  Are people struggling with their current toolset?  If they haven’t had training in the tool ("I’ve been modeling for 40 years; I don’t need training" or "There’s documentation; I’ll read it someday"), changing tools isn’t going to help solve the problem.

If your process is broken, if your resources are lacking training in the tool, let me do the math for you:

New Tool +
Broken Process +
Lack of Training =
________________

Old Problem with a New Notation

I should also mention that switching tools to try to fix a process, people, or training problem will lead to:

• Project delays
• Increased project risk
• Increased budgets
• Decreased productivity
• Decreased confidence in IT
• Decreased team morale

If you don’t fix your process, people, or training issues, all the tools in the world aren’t going to provide benefits you think you are going to get.

You are most welcome.

The Best Data Modeler is a Lazy Data Modeler – #tsql2sday Post

Special note: This post is part of TSQL Tuesday , a special blog posting monthly event based on a SQL Server/data topic chosen by one blogger.  I’d love to see more data architects be part of the blog conversations.

I frequently hear from project team members that they’ve never used the automation features of their data modeling tools.  Some of the reasons they give:

• I’m not a programmer.  It is not fair to expect me to to know how to program in order to use these tools.
• I have no idea how to use it and don’t have the time to find out
• Nothing I do can be automated; it’s all one-off tasks
• You can automate some of my work???

I think if you are giving these reasons without even trying, you a missing out on one of the niftiest features of your tools. 

I’m not a Programmer

Great! Neither am I.  The good news is that many tools don’t require you to have full blown programmer skills in order to automate data modeling tasks.  They have macro-like languages that require a bit of logic skills, but not much more.  For instance, Embarcadero ER/Studio XE uses a macro language called Sax Basic that is very similar to VBscript.  I’m lucky in that I used BASIC early in my career and am generally familiar with the language.  The toughest part is learning the functions, objects, and properties that are specific to ER/Studio, but thankfully there is a built in Help system that does a half decent job of helping you use them.

I Have No Idea How to Use It

That’s okay, because I don’t start with a blank macro when I go to automate a task; I just start with an existing one that is close to the same thing and tailor it to what I need to do.  I needed to export some meta data from my model to Excel, so I opened an existing macro that exported a bunch of data and tailored it to include the data I wanted, in the format I wanted it.

We host a mailing list/forum just for macros and automation of data modeling tools in our InfoAdvisors User Discussion Groups.  There are sample models that others have tailored and community members who are willing to help you through a tough part.  Also, most vendors have similar resources on their websites.

InfoAdvisors ER/Studio Macro

Nothing I Do Can Be Automated

I think you must be spending a huge amount of time clicking and waiting when you could be pushing a button and doing something else like grabbing a cup of coffee or answering a question on a forum…or even helping out a team member.  For instance, some of the macros that have been posted to our communities are:

ER/Studio Exporting All a Model’s Submodels as Images

ER/Studio Macro to Convert 1 Datatype to another Datatype

ER/Studio Macro to Add all Entities from a Model to It’s Submodel

ER/Studio Macro to Print PDFs of all Submodels Completely Unattended

ERwin Macro to Generate DDL for a DBMS not Currently Supported

Most of the macros I write tend to be to make some boring aspects of my job less boring by allowing me to do something else.  This means printing out my entire model, exporting images, making mass updates, etc.  If it has an algorithm that I can automate, I’m going to invest 15-20 minutes so that I don’t have to spend hours or even hundreds of hours over the course of a project doing those non-architecture tasks.

Another major use of automation I do is for setting properties for my DBAs.  Their standards and preferences should be automatable.  How FKs are named, how indexes should be names, which datatypes should be used, etc.  The fact that I can run a quick macro to do these and keep my DBAs happy, well, that’s priceless.  They love me for it <crickets, crickets>…well they should love me for it.

You can Automate Some of my Work???

Yes.  Not really the analytical parts, but some of the more mundane, boring, "change all these but not those" tasks.  Sure, finding and tailoring a macro takes time, but it is so worth it the next time you push a button, wander off to the coffee room to refill your 31 oz. Trenta cup with high-octane coffee.  Managers really don’t want you spending times on tasks that nearly anyone with mouse skills could do.  By using macros and APIs, you can add significant hours of productivity to your day.  Let’s also admit that computers are generally better at doing mass changes more accurately than we are. 

So let’s summarize:

1. Automating boring tasks makes you happier.
2. Happier Data Architects are better Data Architects
3. Automated recurring, boring tasks make bosses happier
4. Automating tasks makes for more accurate work
5. Saving time for you and your team members makes everyone happier.

A Lazy Data Modeler is a Better Data Modeler.