Monte Carlo-ing Your Eventual Consistency Bets

One of the features of not-only-SQL (NoSQL) data storage systems is the concept of eventual consistency (via Wikipedia):

Eventual Consistency… means that given a sufficiently long period of time over which no changes are sent, all updates can be expected to propagate eventually through the system and all the replicas will be consistent.

For those of us coming from a transactional system point of view, eventual consistency can be mind-boggling at first. Thinking about data being presented in an inconsistent manner is usually seen as a data quality failure — something to be avoided. But in non-transactional systems it’s worth the trade-off for speed and scalability. Think about your Facebook page for a minute: how bad would it be if one of your friend’s updates was not visible to you at the same time it was visible to someone else, but eventually you’d be able to see that update?

Paul Cannon has a great write up on using tools to estimate your eventual consistency with Cassandra:

"The best part is that they also provided the world with an interactive demo, which lets you fiddle with N, R, and W, as well as parameters defining your system’s read and write latency distributions, and gives you a nice graph showing what you can expect in terms of consistent reads after a given time.

See the interactive demo here.

This terrific tool actually runs thousands of Monte Carlo simulations per data point (turns out the math to create a full, precise formulaic solution was too hairy) to give a very reliable approximation of consistency for a range of times after a write."

Being able to plan your architecture to best fit the business need is what is important, not necessarily data purity at the cost of speed or reliability.  Again, that sounds weird to a profession that has focused on fighting to keep data integrity on the radar of management, but the best design decisions are made balancing cost, benefit and risk.  Those of us in the data world to understand that eventually consistent is often the best solution.  Even if it feels weird.

Having tools that help us understand how to best architect the trade-offs is the first step in delivering the right data consistency for what the business needs.

Related articles

• Eventually Consistent (queue.acm.org)

Handouts From My DAMA Speaking Tour

I had a fabulous time meeting DAMA Members in the US Midwest over the last couple of weeks.  I was worried most about making all these flights, but they all worked out well.  These DAMA Speaking Tours are something I look forward to every year.  If you are interested in booking one, contact chapters in your general area and propose some dates.

DAMA Minneapolis 

Career Success in the Data Profession

You’ve Just Inherited a Data Model, Now What? (Video)

DAMA Iowa

10 Physical Database Design Blunders

DAMA Chicago 

The Straw Poll Handouts

Career Success in the Data Profession

DAMA Day Nebraska

Data Modeling Contentious Issues

What Does #SQLFamily Mean to Me? #mememonday

This month’s Meme Monday assignment by Tom LaRock (@sqlrockstar | blog) is to write about the SQL Server community (#SQLFamily) and what it means to us. I’ve been blogging and Tweeting about my experiences as part of this community for a while and I sometimes get questions from those outside the SQL Server world about why I keep putting "SQL" in front of everything. It’s hard for me to explain because most of the time I don’t really mean SQL Server, but the SQL Server community people who have done so much for others and me.  There have been some amazing posts so far in this meme Monday. I’d love to see 100 blog posts about this topic.  Tell us what the #SQLFamily means to you.  If you don’t have a blog, this would be a great time to start one or you can send me your thoughts and I’ll guest blog them here.  Yes, there is still plenty of time.

I blogged recently about #SQLRun, a group of family members who ran the Portland Marathon, Half Marathon and 10k and raised a cloud of money for charities. This reminded me of one of my favourite quotes:

Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has.

- Margaret Mead

The SQL Server community isn’t just about having people offer up spare rooms, parking spots, or sofas for when I travel, helping each other out, or agreeing to sign a legal document for each other. The strength in our community is that we are working together to make the world a better place, often about data and databases, but sometimes about helping people get back to work or helping out with serious real-life problems. We don’t always agree on some of the things we are each trying to change (nulls aren’t evil, really), but we fight them out on Twitter, Facebook and LinkedIn, then head to an after party, hangout, or SQLSaturday to show support for each other when it is needed.

I’ve seen many instances of #SQLFamily changing the world:

• Like many IT organizations, PASS has a Women in IT chapter and program. I’m hoping the work we do there is making a difference in society, even if it causes only one person to go home and talk to their daughter about taking more math and science classes in school.
• I’ve seen ad hoc groups of #SQLFamily people get together to volunteer locally, to collect supplies for the less advantaged and to raise money for charities. Not huge things, but something that makes a difference. I hope to see more that at future PASS events.
• I have discussions and debates with other family members about balancing data quality with system performance, but that’s what the world needs. We need passionate people to figure out together the right balance.
• I’m always happy to see people muster up a series of job interviews for SQL Family members who are looking for a new project.
• I love the discussions about how to manage work-life balance, especially since this is a problem that impacts women staying in IT jobs.
• I’m thrilled when I see personal offerings of encouragement, even when we don’t know what else to do.  In my #SQLRUN blog post I wrote about our Scream Team of real-life and virtual encouragers. There is a lesson there (and another blog post to come).
• Thousands of community members spend time blogging wonderful information about what they know.  This sort of giving is something I don’t see as much of in other communities. Sadly, this is especially true in the data management professional community.

So many people blog on a regular basis, hoping to influence others to think harder about providing better data to the world. That touches me, even if it’s a post about data models, indexes, keys, normalization or virtualization. When someone takes the time to put their thoughts down in writing so that others can benefit, I see the power of this community. My tagline is "Love Your Data" because I want to influence the IT community to think beyond code and table structures. However, I need to be influenced just as much about the operational side of databases so that data can be available and reliably accessed. Having said that, some of the greatest lessons I’ve learned came from people in the community talking about collaboration, professional development, human dynamics and other "softer" subjects. All of this because virtually everyone in the community wants to make everyone else successful. That’s amazing stuff.

That’s when I realized we in the #SQLFamily aren’t just about databases. We are passionate about changing the world for the better.

That’s what #SQLFamily means to me.

Speaking: #SQLSat96 Washington, DC …well… Chevy Chase, MD 5 Nov

I’ll be presenting Model Driven Database Design on Saturday, 5 November at the SQL Saturday Washington DC. Which is being held in Chevy Chase, Maryland.  Yes, I know this is the greater DC area.  Last year it was held in Reston, Virginia. But these data things drive data architects crazy.  That’s why you really don’t want to live with a data architect.  Ever.

Anyway, I’ll be presenting a brand new presentation that covers:

Model-Driven Database Design

Model-Driven Database Development: Myths, Magic and Methods. In this presentation, Karen discusses data model-driven database development from the point of view of the Data Architect, the DBA, and the Developer. She will cover topics such as "Who does what?", "Why are we doing this?", "Do I have to Use a GUI?" and "Just who do you think you are?". Finally, 10 tips for making model-driven database development successful in your organization’s culture and environment.

Session Level: Beginner

Karen Lopez

Karen López is Sr. Project Manager and Architect at InfoAdvisors, Inc. Karen is a frequent speaker at conferences and user groups. She has 20+ years of experience in project and data management on large, multi-project programs. Karen specializes in the practical application of data management principles. Karen is also the ListMistress and moderator of the InfoAdvisors Discussion Groups at www.infoadvisors.com.

I’m up against some fairly big name speakers, so I’m really hoping to see you in my session.  Both of us can have a great time talking about data models.  See you there.

Datachick & SQLClippy in the #SQLPASS Day 12 Keynote – Column Store Indexes – Data Quality Services

On day 2 of the PASS Summit, I had a supporting role in Lara Rubbelke’s ( @SQLGal ) keynote demonstration of new features of SQL Server 2012. Lara is a Principal Program Manager for Microsoft. She demoed the new Column Store features and Data Quality Services that will be available in SQL Server 2012.

There’s my avatar next to a comment I made about slow reports and data quality:

SQLClippy ( @SQLClippy ) also had a role:

And in case you missed the faux comments from attendees at SQLPASS:

I think you can see that Lara picked the snarkiest of PASS members, don’t you? Love that she said that I’d comment on the database design.

What was the data problem she was showing?  Location data that left some businesses underwater:

You can watch the entire keynote at http://www.sqlpass.org/summit/2011/Live/LiveStreaming/LiveStreamingThursday.aspx (registration required).  I recommend downloading instead of live streaming, as that gives you the ability to rewind/fast forward during the video. Data Quality Services starts at about 38 minutes into the video.

Speaking: DAMA IA (Des Moines) Oct 18 – 10 Database Design Blunders

My next stop on the DAMA chapter speaking tour is Des Moines DAMA Chapter, where I’ll be presenting on 18 October.

10 Database Design Blunders

What’s going on in your physical data model? How many people can or will update it to match the reality of what’s going on in your databases? Who decides what goes into the physical model?

In this presentation we discuss 10 physical data modeling mistakes that cost you dearly. Will your physical design lead to performance snags, development delays, bugs and weakening of professional respect?
Data Architects are often tasked to prepare first cut physical data models, yet these skills usually overlap those of DBAs and Developers and this overlap can lead to contention, confusion, and complacency. With this presentation, you’ll learn about the 10 blunders, how to find them, plus 10 tips on how to avoid them.

More information should be available on their website soon about the location and timing.