Predicting Aging By Accelerated Testing

My dad, Fridgan Lev, wrote the below article in 1973. The ideas were quite powerful. They allowed predicting future. Unfortunately only for a single building, not for the whole humanity.

Predicting Aging By Accelerated Testing

Elephant Hunting for Pygmies

Here is an entertaining post by Tom Ilube, and Rob Styles. It compares introducing Big Data to a company to elephant hunting by pygmies.

The important angle to me was that a huge animal was hunted by my much smaller humans. They  stood on each other shoulders just to reach their target, and then ran like crazy not to be stomped out. 
Nowadays, we still need to work hard. However, using the right technology can replace the need for heroic efforts.

Definition of Big Data -- 1 TB

Amazon Relational Database Service allows databases up to 1 TB.
So there is a clear definition of Big Data according to Amazon -- any database bigger than 1 TB.

Recovery Strategy for Polyglot Persistence

Typically all data for a given application is stored in a single DBMS. Then the usual data maintenance tasks include data backups on a predefined schedule, and testing disaster recovery procedures.  The potential loss of data could be dealt with. The data could be recovered to a point in time right before the failure.  If not, it would be recovered from the most recent successful backup copy.  With the above measures it is always possible to tell the time period the data was lost for, and ask users to re-do their work.

The data administration gets more complicated for a heterogeneous persistence layer.  With multiple data stores, there is an additional concern of data inconsistency across data stores. 

Our application utilized three different data stores:

•  SQL Server tables
•  MongoDB collections
•  Uploaded files (pdf and such) on the file system

Data was interconnected pair-wise: SQL Server and files, SQL Server and MongoDB.

The dependency between SQL Server and files was simple and it was easy to overcome. The uploaded files were referenced from the SQL Server tables by path/name. I.e. the tables included pointers to the data files.  As files should be rarely deleted, the hard deletes were eliminated and replaced with soft deletes.  That meant that as long as the file backup was taken any time later than the SQL Server backup, the recovered data would be consistent.

The interdependency between SQL Server and MongoDB was more complicated.  For a given transaction data might get stored in SQL Server and overlapping data might go to MongoDB.  This poses problems for a possible recovery

• When recovering to a point in time right before the failure, the most recent transactions might not have been written to the MongoDB journal, but were written to the SQL Server log (and vice versa). So when recovered they might be somewhat off.
• When recovering from a backup snapshot it would actually require restoring from two backups: SQL Server backup and MongoDB backup.   The snapshots are run independently and might be a couple of minutes off.

MongoDB has good durability in a recommended configuration, i.e. when utilizing journaling and replication. In initial configuration the journaling was enabled, but not the replication. Without replica, the backups had a rolling problem effect: if data is changed over the course of a backup then the resulting snapshot may be in an inconsistent state that doesn't correspond to how the data looked in the MongoDB at any one moment.   So the first priority was enabling the replication. It added fail over capability, significantly reducing changes of a disaster.  And the backups used –oplog option for precise backup timing.

Yet another trick was used on the SQL Server side.  Its recovery procedures are more sophisticated.  And the database can be recovered to any point in time.  So whenever the database backups had slightly different times, the SQL Server recovery was tuned to match the MongoDB timing.

Two Roads to NoSQL: Why and Why Not

There are two common threads on the path to NoSQL database: the Why and the Why Not. The talks and presentations define objective drivers for the new technology. They painstakingly delineate all the reasons of Why it is preferred in particular situations. However the discussion boards are full of advice of where the new technology can be tried out, striking down the Why Not using it.

The Why reasons include:

• Big Data (scalability)
• Cloud grid (commodity hardware, high availability)
• Concurrency
• Connectedness (social media)
• Diversity (semi-structure, Web 2.0 decentralization of content generation)
• Low impedance mismatch between programming and data model

From my experience:

- A while ago I was involved in an application for analyzing network traffic. To keep the software price down open source stack with MySQL was used.  The application had great-looking screens and thought-out navigation flow, but failed measurably. The database could not even keep up with writing transactions, to say nothing about retrieving information. That would have been a good candidate for the NoSQL.
- A company moving its software from a shrink-wrapped version to the cloud platform must look at the highly-scalable NoSQL solutions.
The Why Not is looking from a different perspective. A new type of application brings its wave of the database innovation, some of which would survive and prosper. That was true for the data warehousing with multidimensional cubes and data warehouse appliances. Now it is the Web 2.0 turn with NoSQL.  You need to build up expertise in the new technology. You need to acquire confidence in the technology and your ability to execute. And it is easier to accomplish on a non-critical portion of applications: execution logs, configuration and metadata, redundant data, etc.

Multi-tenant database requirements

The following is a partial wish list for a multi-tenant DBMS:

• Distribute data across multiple nodes for scalability
• Restrict user access by account/tenant
• Backup multiple accounts/tenants at once
• Restore data for a single account/tenant
• Allow multiple schema versions
• Maintain multiple replicas for durability
• Meter and limit activities by account/tenant. NOTE: They should be measured in CPU cycles not in clock time (the query clock time limit as in SalesForce depends on the overall database load and adds unnecessary limitations)

Schema-less or schema-LOSS

One of my applications utilized MongoDB. Right from the start, we gained a lot of benefits from the choice. Data got stored exactly the way it was needed by the application. It all fit into memory, and performance was amazing. The MongoDB replica sets in the recommended 3-node configuration provided redundancy, and assured high availability.  MongoDB would get good marks from me; it is a solid, mature product.

And it’s schema-less. The database just stores objects from the application. There is no overhead of defining and maintaining a schema definition. The application goes to production. Whenever a problem arises, code is changed and quickly deployed without dealing with the hassle of database administrators. And then, after three emergency fixes, comes a realization. What was an initial advantage is now a big problem. After each fix, data was created in a new format, and old data was retained “as is”. There is data in four different formats. There coexist four different schemas, without a clear way to tell them apart. It’s no longer “schema-less”, now it’s SCHEMA-LOSS.

Before NoSQL fixes were accompanied by database management tasks.  Database structures were altered and old data was migrated to the new schema. Now there is a functionality gap caused by the disappearance of a schema. Unless some other tool fills the gap, maintenance becomes a nightmare.