Einstein on Data

…can you read the formulae?

Podcast: NoSQL and PeopleTalkingTech

I recently talked with my good friend Denny Cherry (@mrdenny | blog) about my experience at the NoSQL Now! conference and working with NoSQL technologies.  Denny’s new podcast series is called People Talking Tech and he has other interesting topics and people coming up soon.

http://peopletalkingtech.com/eposide-001-karen-lopez

My comments focused on how at the NoSQL professionals understand that it means "Not Only SQL" and can’t mean "No SQL" and have much of a future.  Using the right tool for the right job.  Cost, benefit, risk and all.

One of the things we talked about on the closing panel is "how do you find somebody that is a good architect who can tell you which types of technologies you can use for which use cases…"

Even though many people talk about NoSQL needing no architecture, we still need people to help choose when and what NoSQL technologies to use.  Seems to me that having experience working hands-on with relational and NoSQL technologies is going to be hugely valuable in the next couple of years.  If you have relational experience, now is the time to start learning about non-relational ones.

By the way, we talked a bit about database security.  Denny’s new edition of his book Securing SQL Server, Second Edition: Protecting Your Database from Attackers has recently been released.  Check it out.

What to Do When Your Data’s All Wrong…

Please Clean Up…

Bad Data Models / Database Designs Kill – Under Modeling

Dr. Danielle Ofri has a frightening post about how data design can cause serious consequences.  She talks about running up against a 1000-character limit when trying to create a patient history for someone about to undergo a risky surgery:

I panic for a moment, fearful that the computer has frozen and that I’ve lost all my work — something that happens all too frequently. But I soon realize that this is not the case. Instead, I’ve come up against a word limit.

It turns out that in our electronic medical record system there is a 1,000-character maximum in the “assessment” field. While I’ve been typing, the character number has been counting backward from 1,000, and now I’ve hit zero. The computer will not permit me to say anything more about my patient.

If you’d done any database design, you know that even if you design a good, business-driven design, others who use the database might apply their own rules to the data on the way in or out of the database.

I remember designing a Point of Sale system database for an appliance retailer.  Our data model needed to support the selling of high-end appliances as well as bulk purchases for high-end appliances.  So our transaction amount columns were significantly large.  A developer on the application team thought our monetary field lengths were insanely large, so he enforced  a limit on a transaction of $9,999.99 for the total transaction.   To make matters worse, this system went all the way into production with that limit.  So on day one of the roll out, sales people couldn’t sell the most highest margin items such as a professional quality stove ($14,000) or sell to developers who were buying 100 low end stoves in one transaction.  Their orders had to be chunked up into tiny $9,000 mini-transactions.  Order completion was taking hours instead of minutes.  Credit cards were being rejected for too many large amount transactions back to back.  In other words, a nightmare deployment because organizations trying to spend tens of thousands of dollars were walking out and going to other retailers with “real systems” to make their purchase.

However, no lives were being lost (that I know of).  Some people may have gone longer without heat (furnaces) or with rotten food (freezers and fridges), but in the overall scheme of things the impact on customers was not life and death consequences.

If we get back to Dr. Ofri’s situation, though, she was faced with a terrible data dilemma: how to describe a complex patient history in 1000 characters.  This number probably sounded like a huge number when the project team was adding that “feature” to the system.  I’d even bet their own test data only went as high as 200 characters or so.

I’m also guessing that since this system is a fairly high-risk project that some expert user (or many) was responsible for approving the design of field lengths.  Perhaps he or she also thought that 1000 characters was enough.

In desperation, I call the help desk and voice my concerns. “Well, we can’t have the doctors rambling on forever,” the tech replies.

That response from IT (even if it a help desk tech who has no clue as to why there is a limit) makes me mad and afraid at the same time.  You’ve all heard it in your design reviews, haven’t you? 

• That’s way too long of a field. 
• It won’t fit on one window without scrolling
• No one has an e-mail address THAT long
• You are over modeling it
• That’s ridiculous.  No one is going to sit there and type that long
• 255 was good enough for the last 10 years, it’s good enough for the next 10 years
• The indexes on that column will be too large
• Go ahead and make the column that long; we’ll just truncate it in the application

The business users are frightened by the negative comments and agree that 25 is sufficient for e-mail address, not even realizing that some of their own e-mail addresses are longer than that. 

As I blogged recently about Over Modeling, it’s only over modeling if it doesn’t meet the business need.  Sure, we might make concessions to technical feasibility (“make every column 2000 characters, just in case”), but our designs should be business driven.

Dr. Ofri ends her story by saying:

I’ve finally condensed my patient’s complicated medical conditions to exactly 1,000 characters. I quickly hit “save” before I lose everything. I wish him good luck on his operation, wondering if his surgeons will have to condense the entire operative report to 1,000 characters as well. What happens if there are complications?

For my next medical evaluation, I think I will use haiku.

I don’t know about you, but I wouldn’t want to read that about my own patient record.

Next up: What could we have done differently on our project

The Perfect Data Model, Gone to Hell (MI) Due to Bad Web Form Design

Image via Wikipedia

I don’t normally work in the UX/UI design world, but I know enough from constantly filling out web forms that too many designs out there are destined for a special ring of data Hell.  If you’ve followed any of my web form rants on Twitter, you may have heard this before…but it should be repeated.  Seth Godin recently blogged about his frustrations with annoying web forms for data collection:

The problem with letting your web forms become annoying is that in terms of time spent interacting with your brand, they’re way up on the list. If someone is spending a minute or two or three or four cursing you out from their desk, it’s not going to be easily fixed with some clever advertising.

I realize that many web designers live in the US and hate the fact that they have to complicate their beautifully simple designs with all these weird non-US things like regions, postal codes, country lists, etc.  But if their organization does business with non-US customers or data, they need to realize that the design must support these wacky international requirements.  

One of my favourite resources for good web form design is a FREE eBook by Graham Rhind on names and addresses in web forms  Graham specializes in internationalization and address formats, so he is the go-to guy for these sorts of things.

It’s not just internationalization, though, that causes web form design to go all to Hell.

My pet peeve is referenced in Seth’s post: using drop downs to force a user to choose from a list of hundreds or thousands of values.  These are annoying because drop downs usually require acute mouse skills as well as waste time.  Developers love drop downs because they don’t have to do much data validation – if it’s in the list, it’s supposed to be good data.  However, optimizing a developer’s task isn’t always the the best for customers who have to use the form.   In fact, I’ve come to realize that the more we optimize development, the more we have to take from the end user.  It should not be that way, but I see it over and over again.

A typical frustration I’ll face is a form that collect address information.  It will have fields in the same order that we’d typically see a mailing label, something like:

• Name
• Address line one
• Address line two
• Apartment#
• City
• State
• Other
• ZIP
• Country

…with State and Country being a drop down of all the US States and Country being a list from somewhere on the web of a list of countries.  There might be some magic in the country list that then causes the list of states to change based on the country select.  The problem is that as one fills out the fields from top to bottom, he hits the State field before the country field.  He has to jump down a few fields to find the right country, then jump back to the drop down.  If he is very fortunate, this change in country does not require a complete refresh of the form so his data might still be there…or it might not.

Or, the web designer might think that we foreigners should use the Other field to fill our foreign state or province.  They might also beef up their data quality be requiring a State in the drop down, even if it only contains US states.  We users won’t know until we try to submit the form.

When I use forms that require me to pick a US State, I usually go with OH or OR since they are “close” to my Province Code of ON.  Sometimes I pick Hell, Michigan because it’s just as good of a place as any if web form constraints force me to enter bad data.  I’ve always wondered, though, how that impacts analytics for that data.

My other peeve is when Birth Date must be filled in via a drop down.  First one must pick from a list of months, then a list of dates 1-31, then a list of Years going back to the ice age….or somewhere near my birth year.  There are much better ways for web designers to collect and validate data.  I’d love to see business management sit down and enter a couple hundred addresses into their web forms to decide whether the forms are “good enough”.

When users find annoying forms, they are more likely to enter bad data.  Don’t ask me how I know this…let’s just say that there are many records of me out there, happily describing my nice home in Michigan, where I celebrate my 13th birthday ever year, in my apartment number 0000, which also shares a ZIP code with a popular TV series from the 1990s. 

Love your data; don’t torment it in the hands of end users before it even gets to you.