http://geek-and-poke.com/2012/06/no.html

NoSQL started out as meaning “No SQL” as in “No Relational Databases”.  It represented a movement to replace RDMBSs like SQL Server, DB2, Oracle, MySQL and Sybase.  Over time, the more adept architects realized that there’s a place for both relational and non-relational technologies. However, NoSQL was a brand at this point, so like KFC, they redefined this as meaning “Not Only SQL”.  This has been a nice turn of events, but it’s still a branding issue.

I believe the NoSQL community needs a new name – a new brand – to say what the community stands for.  In fact, I was on a closing panel at NoSQL Now! that discussed this need.  We didn’t really come up with an answer, but the general consensus was saying you aren’t isn’t nearly as helpful as saying what you are.

So what are those solutions that aren’t relational?  DocBlobGraphEVA?

This week was Canadian Thanksgiving so this post on metadata stuffing is timely.

Today Thomas LaRock (@sqlrockstar | blog ) posted a rant about our Database Design Throwdown topic on naming standards for tables and other database objects.  Tom is a fan of what I call “metadata stuffing” in object names.  That’s basically shoving as much additional information as one can into object names so that one does not have to go find out that information from its rightful place.

My reason for wanting to use prefixes is simple enough: I want to know if I am looking at a table or a view when reviewing code.

Karen doesn’t believe that anyone should be using object names as a place to store meta data about the objects themselves. I would like to agree with her but then we’d both be wrong.

He even created the nice graphic of my quote I use here. Thanks, Tom.

Object Confusion Abounds

That quote is indeed one I use during our debate.  It’s a snarky introduction to what I think the real problem is: our tools aren’t doing enough to help us with this potential confusion of tables and views.    When you are writing or looking at a query, the syntax for referencing a table or a view is exactly the same.

When a developer writes:

USE AdventureWorks;
GO
SELECT Name, ProductNumber, ListPrice AS Price
FROM Production.Product 
ORDER BY Name ASC;
GO

He has no idea for certain if Product is a table or a view.  The syntax is the same.  And if the developer is working in a text editor of some sort, or with some native tools, there’s no tooltip or other help that the he can use to check what type of object they are querying.

Why does this matter?  Tom has a great presentation where he shows the impacts of trying to make stuff work and you don’t know what you are looking at.  That’s why Tom wants to do this with his objects:

FROM Production.tbl_Product

or

FROM Production.view_Product

So that the object type is injected into the object name.  That seems so innocuous, doesn’t it?  What could go wrong?

I’m here to tell you that this is a slippery slope.  One of the most egregious examples of this sort of meta data stuffing I’ve run across is one that required all this meta data to be prefixed in front of every table name:

• tbl_ prefix
• Primary Systems that managed the table
• Primary subject area that the table belongs to in the data model
• Classification of the role the table plays in the database (Associative Entity, Domain Entity, Master Data, Reference Data, Log Data, etc.)
• Three letter login of the DBA responsible for administering this table (I KID YOU NOT).

So in this wonderful naming scheme, we’d get:

FROM Production.tbl_MFGR_ORDERPROC_DOM_KQL_Product

I have found that once an organization starts thinking of stuffing, their designs become turkeys really fast.  It’s ugly.  Think about the tools you use, with all those nifty object lists on the left side. To find a table you need to know all that great metadata as you scroll through the list, hoping that PRODUCT Is buried in there, somewhere.  And what the heck was the name of that guy that dressed funny who did all that data stuff for the company before he won the lottery?

All that metadata that should have been managed elsewhere, not prefixed in front of the “real” name of the table.  In fact, it was.  In the data model and in the system catalog. Every time any of that data changed (DBA assignment changes, DBA wins the lottery, whatever), we had to rename the table and change all the code and reports that referenced it.  Sure we could have isolated systems by this change by using views and or aliases but that is additional complexity for no performance gain, either.  Refactoring might have helped, but eventually we’d still have to change all the code and queries.

Why I Hate Metadata Stuffing

1. It’s redundant data.  Just like with business data, the reason we want to minimize redundant data is because we then have to worry about updating the data in multiple places. There’s cost and added risk for that.
2. It changes.  I don’t know about you, but I don’t have the luxury of taking down a production system just to update  a change in the name of the DBA’s favourite TV show or whatever lame naming scheme someone thought up. Sure, tables can’t change into views or indexes, but all the other type of stuffings will change.
3. It takes up real estate. I get all kinds of flack from developers and DBAs for the length of object names when I want the names to be meaningful.  It’s funny how spelling out CUSTOMER is unacceptable, but adding the exact same characters in front of every object of its type is A-OKAY.  What’s up with that?  Somehow optimizing names for developers is more important than loving your data?  Show me where it says that in the Project Charter.
4. tbl_ is a tell for bad database design.  I don’t know where this particular naming scheme originated, but when I do a database design review and I see this naming scheme, I know that the designer learned design in a one hour webinar “training course” and has not really mastered the complexities of enterprise database design and maintenance. The design will be less than best practice 90% of the time.  This naming scheme is prominent in programming books, introduction to database books, presentations by non-database people, and uninformed blog posts, by far.  It’s not popular with people who do professional database design. Sure, some products use this, too, but do you really want to take database design best practices from vendors? How many professional data architects do you think they have on staff?  I will most likely see a database design that is highly optimized to make development go faster.  Not for data integrity or loving data.
5. It’s not needed “for consistency”. One of Tom’s points is that if we are going to prefix views, we have to prefix tables to be consistent.  Actually, no, we don’t.   If we have to bite the bullet and prefix views because our tools let us down, we can choose not to clutter up tables names just to punish those objects, too.  I’m assuming that since Tom prefixes tables and views, he prefixes columns, too, right?  its just being consistent.  < snort>
6. It gets in the way of using the data. Tables and Column names are the most user-facing parts of a database design.  When we in IT insist on munging up these names with a bunch of systemese, we make it more difficult for business users to get at their data.  It shows that we have optimized the database design to help a relatively small number of technical users (developers, DBAs, ETL folks) over the needs of the business.  Ultimately, we build databases to manage data.  For the business.

What’s the Cost, Benefit and Risk?

One of my Splendid Truths is that all design decisions should assess cost, benefit and risks.  In the overall scheme of things, just prefixing “tbl_” in front of a table name isn’t that costly and it isn’t that risky.  Tom assesses his designs based only on potential for performance harm according to his post.  He “laughs” at my position.  I’m happy that my stance on metadata stuffing brings happiness to his day.  But performance is only one data point out of many for making a design decision.  Usability, clarity, business goal support are other factors that a database architect needs to consider when assigning a name to an object.  If we optimize something for a subsystem, we do it as the expense of other subsystems.

Our Tools Should Help Us More

Having said that, I feel the pain of people having to work with sub-standard tools or having to use tools that just refuse to help.  Tom showed how SQL Server Management Studio tooltips can help.  But all those command line “I don’t need any stinking help” aficionados are left on their own to know what they are looking at.

Oh, and Tom:

ProperNoun_Tom, Pronoun_you Verb_made Pronoun_me Verb_laugh Preposition_with Possessive_your Noun_post.

(See how all this stuffing gets in the way?)

One of my other Splendid Truths about database design is:

Your tools will impact your data models and database designs more than you can imagine.

We shouldn’t sit back and let that happen. Stuffing is great with Tofurky, not with databases.

This past weekend I attended SQL Saturday San Diego, AKA, #SQLSat157.  This was my first time speaking at this event and I want to give lots of thanks and kudos to the organizers for putting on a fine event.

Because I arrived in town early to meet with friends from both the space and data world, I was able to visit the San Diego Air and Space Museum.  It was fitting that it was the 50th anniversary of President Kennedy’s Rice University speech on space exploration:

There is no strife, no prejudice, no national conflict in outer space as yet. Its hazards are hostile to us all. Its conquest deserves the best of all mankind, and its opportunity for peaceful cooperation many never come again. But why, some say, the moon? Why choose this as our goal? And they may well ask why climb the highest mountain? Why, 35 years ago, fly the Atlantic? Why does Rice play Texas?

We choose to go to the moon. We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.

Not only is this fitting for motivating a generation to invest in space exploration, it’s fitting for professional development work, too.  We attend and speak at SQL Saturdays not because it’s easy, but because we need goals to serve to organize the best of our energies and skills. I can’t tell you how many times I’ve been inspired to learn something new because I saw a fellow community member demonstrate how it could help make life for end users or co-workers better.  And SQL Saturday gives me a full day of these sorts of workshops and demos…all for free.  How great is that?  It means giving up a Saturday and for those of us who travel to speak, 2-3 days plus expenses.  And yet every time I leave one, I think "That was so worth it".

Sessions

I spoke three times at this SQL Saturday: DB Design Throwdown, the Women in Technology Panel, and Career Management for Data Professionals.  Between those, I was able to see just a couple more sessions.  I really enjoyed Lynn Langit’s (@lynnlangit | blog)  NoSQL for the SQL Server Developer.   Lynn did a fabulous job explaining the differences between SQL and NoSQL technologies, as well as demoing MongoDB and cloud-based technologies.  You should spend some time on her blog; she has a lot of great stuff with plenty of videos and demos.

I also had the pleasure of being on the WIT panel with Lynn.  This panel, moderated by Tara Kizer, focused mostly on how we can energize the next generation of girls (and boys) to be interested in IT careers.  Lynn is doing some fabulous stuff over on http://teachingkidsprogramming.org, where she and her partner, Llewellyn Falco (@llewellynfalco | blog )  are building a framework for, well, teaching kids programming.  

I talked about the importance of talking with girls in your life, which is my usual homework assignment for attendees.  Having someone in the IT profession share the fact that the industry isn’t just about typing and programming can make a real difference to a girl who just needs to hear that IT professionals can make a difference in the world.  In fact, I have another blog post coming up soon on that topic.

Download the Database Design Throwdown: The Trailer presentation.

Download the Career Management for Data Professionals presentation.

http://peopletalkingtech.com/eposide-001-karen-lopez

My comments focused on how at the NoSQL professionals understand that it means "Not Only SQL" and can’t mean "No SQL" and have much of a future.  Using the right tool for the right job.  Cost, benefit, risk and all.

One of the things we talked about on the closing panel is "how do you find somebody that is a good architect who can tell you which types of technologies you can use for which use cases…"

Even though many people talk about NoSQL needing no architecture, we still need people to help choose when and what NoSQL technologies to use.  Seems to me that having experience working hands-on with relational and NoSQL technologies is going to be hugely valuable in the next couple of years.  If you have relational experience, now is the time to start learning about non-relational ones.

By the way, we talked a bit about database security.  Denny’s new edition of his book Securing SQL Server, Second Edition: Protecting Your Database from Attackers has recently been released.  Check it out.

The first is with co-presenter, Tom LaRock (@sqlrockstar | blog), where we debate, whine (Tom) and win (me)  several database design approaches and methods in front of a live audience (you!).  This is a warm up for our PASS Summit spotlight presentation.

Database Design Throw Down

Karen and Tom debate about the options and best practices of common and advanced design issues, such as: * Natural vs. Surrogate keys * NULL vs NOT NULL * Datatypes * Agile Database Design * Database Refactoring * Identity Crisis ? …and others. Bring your votes, your debates, and your opinions. Help us figure out who’s right and who wrong…or less right.

Session Level: Intermediate

My second presentation is on career management.

Career Management for Data Professionals

Career Success in Data Management during Turbulent Times: A workshop on issues and ideas that today’s data professionals can do to build their careers and networking skills with other data management professionals. Workshop topics will include: • Demonstrating your expertise • Building a portfolio of your success stories • Getting others to sell your skills and business value • Building & extending your data management skill set • 10 Steps to highlighting you and your work Bring your thoughts, ideas, and experiences.

Session Level: Beginner

There are many great speakers at this event and it’s FREE for a full day of learning. Registration is still open, but it is common for these events to sell out before the event. Register now!

…and, yes, this is quote from one of my former DBAs.

We will be judging victims candidates on the quality, completeness, professionalism and technical correctness of their solutions.  That part is just like in real life…except for cameras, producers and an entire professional film crew. And leeches, I think.  Just to make all more fun.

Anyway, the trailer for the reality show is up now.  Episodes will start going up on the Microsoft Learning website on 24 July. 

You Can Win, Too

Microsoft Learning is also running a prize contest for viewers.  You can win a free training class from New Horizons and certification exam vouchers from Prometric. This contest is open to nearly everyone, so not just US residents.  Register for a chance to increase your SQL Server skills. 

Stay tuned here for more exciting details, plus behind-the-scenes posts and photos. Seeing the contestants, knowing the judges…this is going to be one heck of a ride…

Ross Mistry (site | @RossMistry) and Stacia Misner (blog |@StaciaMisner) have written Introducing SQL Server 2012, a 288 page eBook and it’s free for download.  It’s currently available only as a PDF, but according to the publisher, .mobi and EPUB version will be available by 23 March. 

Ross covers the newest features and enhancements on the database administration side while Stacia describes what’s new with business intelligence, data quality, master data, analysis services, integration services and reporting services.

Once you get the book, you’re going to want to get your hands on the pre-release (RTM) version of SQL Server 2012.

So head over to http://www.microsoft.com/sqlserver/en/us/default.aspx , download the RTM of SQL Server 2012  and get started.  You should also check out the demos from the SQL Server 2012 Virtual Launch site for overviews of some of the newest features. 

To create a column with the IDENTITY property in SQL Server 2012:

CREATE TABLE 

    [ database_name . [ schema_name ] . | schema_name . ] table_name 

    [ AS FileTable ]

    ( { <column_definition> | <computed_column_definition> 

        | <column_set_definition> | [ <table_constraint> ] [ ,...n ] } )

    [ ON { partition_scheme_name ( partition_column_name ) | filegroup 

        | "default" } ] 

    [ { TEXTIMAGE_ON { filegroup | "default" } ] 

    [ FILESTREAM_ON { partition_scheme_name | filegroup 

        | "default" } ]

    [ WITH ( <table_option> [ ,...n ] ) ]

[ ; ]

<column_definition> ::=

column_name <data_type>

    [ FILESTREAM ]

    [ COLLATE collation_name ] 

    [ NULL | NOT NULL ]

    [ 

        [ CONSTRAINT constraint_name ] DEFAULT constant_expression ] 

      | [ IDENTITY [ ( seed ,increment ) ] [ NOT FOR REPLICATION ] 

    ]

    [ ROWGUIDCOL ] [ <column_constraint> [ ...n ] ] 

    [ SPARSE ] 

http://msdn.microsoft.com/en-us/library/ms174979(v=sql.110).aspx

I’ve highlighted the syntax that applies the IDENTITY to a column.  SQL Server will know which column has the identity property (and you can only have one of these columns in the same table).  Notice that there are only three options for an IDENTITY.

To create a SQUENCE in SQL Server 2012:

CREATE SEQUENCE [schema_name . ] sequence_name

    [ AS [ built_in_integer_type | user-defined_integer_type ] ]

    [ START WITH <constant> ]

    [ INCREMENT BY <constant> ]

    [ { MINVALUE [ <constant> ] } | { NO MINVALUE } ]

    [ { MAXVALUE [ <constant> ] } | { NO MAXVALUE } ]

    [ CYCLE | { NO CYCLE } ]

    [ { CACHE [ <constant> ] } | { NO CACHE } ]

    [ ; ]

http://msdn.microsoft.com/en-us/library/ff878091(v=sql.110).aspx

Notice that there is no TABLE or COLUMN referenced in that TSQL.  You are just creating a SEQUENCE.  SQL Server will have no idea what you are going to use it for or where it will be used.  In fact, you could create a SEQUENCE and never use it. Notice there are several more options for a SEQUENCE.

SEQUENCEs are separate objects in a database and can be used to populate more than one column even across several tables.  Because they aren’t bound to a table in any way, they can be used in ways that IDENTITY columns can’t be used.  For  instance:

• One SEQUENCE can be used across more than one table. For instance, a Control Number or Document Number on several tables.  I’ve seen this used in other DBMSs to generate document numbers across subtypes, say Purchase Orders, Shipping Notices, Return to Vendor Numbers, etc.  There’s a requirement that they be unique across all types of documents.  This could not be done with IDENTITY.
• Using a SEQUENCE allows tables to have more than one auto-generated number in multiple columns. Think of Order Number and Cancellation Number, both on the Order table. You can’t do that with IDENTITY.
• Since sequences are generated completely outside a table, you can chose to store the resulting value in a different format, say VARCHAR or CHAR. Or, heaven forbid, you might want to append data to it or do other kinds of evil to it before storing it.
• Since sequences are generated completely outside a table, you can store the results in a NULLable column.

I think the confusion sets in because we tend to think of auto-generated number as useful only as surrogate keys. But we have real data needs for generating numbers other than just row identifiers, even if that’s the primary use for this. In fact, one could have a table with a PK that uses IDENTITY and an unlimited number of columns that use a SEQUENCE to set their values.  Of course, SEQUENCEs can be used for surrogate primary keys, too.  The existence of SEQUENCEs gives us another option over IDENTITY.

So SEQUENCE is a feature that just generates numbers based on some characteristics. A designer has many ways to use the number once it is generated and tables are not dependent on that method to get that number, nor do they even “know” about how that value was provided.

This separation of the COLUMN from how the data is populated is a feature of using SEQUENCEs. How one uses this separation depends on the technical and business requirements for the data.

It’s free to register at https://www.ca.com/us/events/webcasts/na/Lets-Get-Down-to-Business-Help-your-Business-Love-its-Data-Models/02-07-2012.aspx

This is what I’ll be covering:

7 Feb 2012
1:00 p.m. – 2:00 p.m. EST

Duration 60 min.

There’s no one data model hiding in your modeling tool. There are actually thousands of them – not just multiple data model files, but different views and presentations of the same data model, each one ready to be used by different purposes and outcomes. In this session, Karen Lopez will discuss the steps in of preparing and presenting the “right” data model for the right audience, as well as making them accessible via the web. We will also cover the 10 tips for ensuring that your audience is happy they attended the data model presentation and looks forward to attending the next one.

As usual for my presentations, this will have a bit of snark and talk about good things to do in business data models an show some anti-patterns for modeling, too.  That’s probably where the snark will come it.  Can’t guarantee it, but it sounds about right.

Bring your ideas about tailoring the presentations  of data models or any type of design.  Oh, and if you have any action figures, bring those, too.

The ultimate inspiration is the deadline.
-Nolan Bushnell

I’ve been thinking about some experiences I’ve had working with other data architects and working with deadlines.  Usually as part of a team, we don’t get to set our own deadlines; they are set by a project manager or if we are lucky, via negotiation with the DBAs, developers and other architects on the team.  Our deliverables are typically the input into their deliverables, so their success is depends our getting our deliverables to them.  In my experience, though, it’s too common for data architects and data modelers to forget what deliverable those teams are waiting for or for them to make it available in the right format. 

In his post for #mememonday, Thomas LaRock (@sqlrockstar | blog ) give his best tip for working towards a deadline: work backwards.

What is Your Deliverable?

This is the most frustrating question.  It should be very apparent to a data architect what deliverables the team needs from them.  But time after time I see modelers focused 99.999% on the data models.  Yes, your data models should be beautiful.  They should be complete, correct and pretty.  Every single one of them should get a trophy. All the meta data you want to capture about them should be in the data model.  Do you think, though, that your development DBA and developers are tapping their feet waiting for the PDFs of the data models?  Or do you suppose that they just might want that alter script to run in their environment so that they can get to work on their deliverables? 

Ideally, the team wants both.  They want the model to be created and they want the scripts with all the changes needed for their next deliverable.  I can assure you, though, that the scripts are what they want first.  I can also assure you that no one will love the data models more than you do, Mr. Data Architect.  So embrace the love you feel for the data models and funnel it into getting the "real" deliverable done:  DDL.  Sure, it hurts a bit that they don’t love your models as much as you do, but they will learn to love them because they produce beautiful databases.

On most my projects (mostly development ones) I never lose sight of the fact that ultimately, the data models are a method to get to good quality databases and data.

What Makes for a Good Deliverable?

I worked with a data architect who was very passionate about her data models.  She spend days making them beautiful.  She filled them with all kinds of nifty meta data to help people understand their data better.  She wrote definitions that were paragraphs long, then added more notes to them just to make it even better.  She often missed review meetings because she was too busy making a more beautiful data model.  She missed deadlines for delivering DDL because she wasn’t quite sure on how to do that.  She researched data modeling layout techniques to make models more readable.  She demoed different layouts and took surveys about the alternatives to find just the right one.  And she got further and and further behind on delivering revisions to development, her only direct customer of her work.  The team had gone from every other day deliveries of fixes and enhancements to the development databases to every other week.

Our modeler didn’t have time to learn the features of our target DBMS because was busy making the model a perfect data model. Our sprints were failing because database changes were coming too slowly and were often done incorrectly.  The DBAs and developers had to spend days cleaning up the DDL she had produced because she didn’t know how to use those features of our modeling tool.  So not only did she miss the deadlines, she missed the deliverables.  She didn’t understand this, though. In her mind, the data model was all anyone needed.  In reality, what they really needed was working DDL scripts to apply to their environments.   That was what good meant to the team.

What Format?

I worked with another data architect who would spend months working on the data model, then publish it inside a PDF of a Word document.  The actual data model file was not shared.  No DDL was shared.  Nothing was delivered that teams could actually import into their tools or apply to their local development environment.  This also meant that it was nearly impossible to comment on the data model or easily provide corrections or feedback.  The modeler wanted corrections to come to him in emails and he would make them and generate the Word documents and PDFs months later.  Requests for the data model file itself were met with "You don’t need that, just retype the model in whatever tool you use".  We can’t build stuff on PDFs.  Produce them, sure.  But they aren’t the deliverables we are looking for.  Your team (your customers) can tell you what format works best for them.  Just ask.

Where Do You Deliver the Deliverables?

I work with many data architects who don’t use the same deliverable locations as the rest of the project team and I think that’s a huge failure.  Professional development teams use some sort of versioning or configuration control environment to share their work product.  Data architects should use those, too.  Yes, we have our repositories and model marts, but those are typically only accessible by people who use the modeling tools.  They are my versioning control for the models, but I also deliver to the teams in the same place they expect to find all the components of their systems.  Maybe it’s just a wiki, or an intranet location. Where ever it is, I’m delivering there.

I worked with a data modeler who refused to use or learn the team’s versioning system.  So he just emailed around files, with no version numbers and expected people to be able to search through their emails to find scripts to deploy them in production environments. He seemed to randomly include and exclude people in the distribution list.  He often just attached a bunch of files, then said "here they are" in the body of the message.  Email is the worst versioning system in the world.  Don’t use it for deliveries.

Another approach I find annoying is fairly new, though.  I work with a an architect that keeps all the files he works on for several clients on Dropbox. All in one giant folder.  When he gets his work done,  he sends out an email saying: it’s all on Dropbox.  We as a team have to try  to figure out what the files are, which is the right version and try to ignore all the other stuff that sitting there in the folder.  As afar as I’m concerned, he might as well randomly generate his data models.

Work Backwards

If you have the answers to those questions above, you can work backwards to meet your deadlines.  My inefficient team members mentioned  above don’t do this.  They start with a generic to-do list of how to produce data models, start at step one and work their way as far as they can before they hit the deadline.  They often miss deadlines because they haven’t started at the end and worked backwards.  Don’t be that guy (or gal).  Know what you are expected to deliver, when, where, and in what format.

I start with what my deliverable is, what format I need to produce it in, and what measures I should use to ensure it’s good enough.  Then I start there with the tasks that will get me closest to that completion.  It’s only when those are completed do I keep working backwards to fill in the other high priority tasks.  If I still have time left, I do more.  I make it prettier.  I make it more useful.  I make it beautiful.  I love it.

Not all data architecture is done on development projects.  I understand that.  If your duties include supporting development teams, though, you need to support them.  That means loving your data models, the data AND databases.  There’s no reason why you can’t have it all.  Just remember which parts you’re supposed to deliver first.

Have you worked with data architects or modelers who worked backwards and got the job done?  What about people like I’ve mentioned in this post?  What would you wished they had done instead?

You can get real experience with new SQL Server 2012 features such as:

• AlwaysOn, SQL Server’s new High Availability feature.  I have to say this is fun to play with.  Pull the plug and watch SQL Server gracefully fail over.  Like nothing happened at all.  Monitor the status of all the AlwaysOn components.  Very cool stuff here.
• Master Data Services, SQL Server’s new feature for managing reference and master data.
• Data Quality Services, a new feature for helping you Love Your Data even more.
• Sparklines and Data bars, new data visualization features.
• Columnstore Indexes, SQL Server’s new feature to make queries just fly.
• Spatial and Location Services for mashing your data up with location based services.

While you are running the VM set up, register for the SQL Server 2012 Launch events coming in March in the US.  I hope to make it to one or two of those.

I call these people "wonder candidates" and write about how they don’t exist in sufficient numbers to supply all the organizations in the world:

It would seem to make sense that if you were hiring a data professional you’d design a position that fills in the Data column, right?  No?  It turns out, though, that most people don’t think and work along a column.  In my experience, people aren’t passionate about tasks that span columns from top to bottom.  They normally aren’t skilled along the whole column, either.  Referring to the Zachman Framework, what sorts of skills and passions would this candidate need: planning, architecting, designing, building systems, building parts, keeping the systems up and running.

I thought about my rant in this area while reading a job posting on Dataversity for a Data Architect.  I’m sure the people at Miami Children’s Hospital do amazing things, probably with very limited budgets.  That’s why these hiring organizations tell me they have to fill their positions with Architect Designer Developer Implementer Operational Support Wonder Candidates.  I’m going to pick on this posting, so apologies to the hospital for using them as an anti-pattern for finding good data architects.  I’m sure they are nice people there and really want to get to successful database and data warehouse solutions.  You might even want to apply for that job.

“Designs and constructs very large relational databases for data warehousing. Develops data models and is responsible for data acquisition, access analysis and design, and archive/recovery/load design and implementation. Integrates new data with existing data data warehouses in design and planning.”

Right there we have the keywords design, constructs, develops, implementation.  These activities are done in different rows in the data column of the Zachman Framework.  There’s also this:

performance tuning, data retention policies, data classification, data security, and data acquisition….Data modeling experience. Database and application object management, including DDL, table constraints and triggers, clusters, object storage allocation and tuning, indexing options and tradeoffs, partitioning, etc., experience.”

Those activities are clearly down in the lower half of the Framework.  Yet data modeling, which exists along the entire data column, is not typically a strong skill set for people who work so far down in the Framework.   So my guess is that professional data architects and modelers will not be qualified to do the clustering/partitioning/indexing/performance tuning part of the job and that implementers who can won’t be qualified to prepare and maintain the data models they also want out of this role.

If I were interviewing for this type of position, I’d focus on why this organization wants data models but doesn’t seem to want to fund a data architect.  It’s sounds crazy, but I recommend that organizations not incur the costs of preparing and maintaining data models when they don’t want to work with professional data modelers.  They won’t see many of the benefits of having an active data model but will incur all the costs and the risks associated with preparing incorrect ones.

I realize that there are many successful IT professionals who can work along many rows and columns.  I’ve worked with these amazing people.  But staffing a team of these amazing people is costly: they are difficult to find, expensive to hire, and tough to keep around because:

There may be people who can do a lot of those things, but in my experience they aren’t passionate about all of them. New hires won’t be happy and the organization will not realize the economies that they think they will. 

I recommend that if organizations want to combine responsibilities that they do so across the columns in the same range of rows.  Combining positions where thought processes are similar (business and data analysts, DBAs and developers, etc.).  Analysts in general make for good analysts in other columns.  Operational people tend to think operationally, builders tend to think mostly of building, not planning well.  Let’s not drag people up or down the rows.

Go now and check your job postings.  Do they reflect the true nature of the job?  Or are they actually full of zombies ready to drag someone to an assignment that they don’t really want?

Do you work with any of these Zombies?  People who have been hired to fill several jobs, but don’t have the passion or skills to do all of them?  How is that working?

One of the features of not-only-SQL (NoSQL) data storage systems is the concept of eventual consistency (via Wikipedia):

Eventual Consistency… means that given a sufficiently long period of time over which no changes are sent, all updates can be expected to propagate eventually through the system and all the replicas will be consistent.

For those of us coming from a transactional system point of view, eventual consistency can be mind-boggling at first. Thinking about data being presented in an inconsistent manner is usually seen as a data quality failure — something to be avoided. But in non-transactional systems it’s worth the trade-off for speed and scalability. Think about your Facebook page for a minute: how bad would it be if one of your friend’s updates was not visible to you at the same time it was visible to someone else, but eventually you’d be able to see that update?

Paul Cannon has a great write up on using tools to estimate your eventual consistency with Cassandra:

"The best part is that they also provided the world with an interactive demo, which lets you fiddle with N, R, and W, as well as parameters defining your system’s read and write latency distributions, and gives you a nice graph showing what you can expect in terms of consistent reads after a given time.

See the interactive demo here.

This terrific tool actually runs thousands of Monte Carlo simulations per data point (turns out the math to create a full, precise formulaic solution was too hairy) to give a very reliable approximation of consistency for a range of times after a write."

Being able to plan your architecture to best fit the business need is what is important, not necessarily data purity at the cost of speed or reliability.  Again, that sounds weird to a profession that has focused on fighting to keep data integrity on the radar of management, but the best design decisions are made balancing cost, benefit and risk.  Those of us in the data world to understand that eventually consistent is often the best solution.  Even if it feels weird.

Having tools that help us understand how to best architect the trade-offs is the first step in delivering the right data consistency for what the business needs.

Related articles

• Eventually Consistent (queue.acm.org)

I had a fabulous time meeting DAMA Members in the US Midwest over the last couple of weeks.  I was worried most about making all these flights, but they all worked out well.  These DAMA Speaking Tours are something I look forward to every year.  If you are interested in booking one, contact chapters in your general area and propose some dates.

DAMA Minneapolis 

Career Success in the Data Profession

You’ve Just Inherited a Data Model, Now What? (Video)

DAMA Iowa

10 Physical Database Design Blunders

DAMA Chicago 

The Straw Poll Handouts

Career Success in the Data Profession

DAMA Day Nebraska

Data Modeling Contentious Issues