Big Data Databases: the Essence

Big data databases store petabytes of unstructured, semi-structured and structured data without rigid schemas. They are mostly NoSQL (non-relational) databases built on a horizontal architecture, which enable quick and cost-effective processing of large volumes of big data as well as multiple concurrent queries.

Data

Schema

Scalability

Language

Transaction

Best for

Examples

Structured data is stored in tables

Supports rigid (pre-defined) data schema

Vertically scalable

Structured query language

ACID-compliant (atomicity, consistency, isolation, durability)

Complex queries, database transactions and routine data analysis

Amazon Redshift, Azure Synapse Analytics, Microsoft SQL Server, Oracle Database, MySQL, IBM DB2, etc.

Unstructured data is stored according to different models (key-value, document-oriented, graph, wide-column store, and multi-model)

Supports dynamic data schema

Horizontally scalable

Unstructured query language

CAP theorem (consistency, availability and partition tolerance), may be ACID-compliant

Storing and modeling structured, semi-structured and unstructured data

Amazon DynamoDB, Azure Cosmos DB, Amazon Keyspaces, Amazon DocumentDB, Oracle NoSQL database, etc.

Even though non-relational databases have proved to be better for high-performance and agile processing of data at scale, such solutions as Amazon Redshift and Azure Synapse Analytics are now optimized for querying massive data sets, which makes them sufficient when dealing with big data.

Best Big Data Database for Your Case

Answer some questions online about your needs (data nature, database usage, performance, and security requirements), and ScienceSoft's big data experts will recommend a database that is best for your specific case.

START

Big Data Architecture and the Place of Big Data Databases in It

Big data architecture may include the following components:

Data sources – relational databases, files (e.g., web server log files) produced by applications, real-time data produced by IoT devices.
Big data storage – NoSQL databases for storing high data volumes of different types before filtering, aggregating and preparing data for analysis.
Real-time message ingestion store – to capture and store real-time messages for stream processing.
Analytical data store – relational databases for preparing and structuring big data for further analytical querying.
Big data analytics and reporting, which may include OLAP cubes, ML tools, self-service BI tools, etc. – to provide big data insights to end users.

Big data architecture - ScienceSoft

Features of Big Data Databases

Data storage

Storing petabytes of data.
Storing unstructured, semi-structured and structured data.
Distributed schema-agnostic big data storage.

Data model options

Key-value.
Document-oriented.
Graph.
Wide-column store.
Multi-model.

Data querying

Support for multiple concurrent queries.
Batch and streaming/real-time big data loading/processing.
Support for analytical workloads.

Database performance

Horizontal scaling for elastic resource setup and provisioning.
Automatic big data replication across multiple servers for minimized latency and strong availability (up to 99.99%).
On-demand and provisioned capacity modes.
Automated deleting of expired data from tables.

Database security and reliability

Big data encryption in transit and at rest.
User authorization and authentication.
Continuous and on-demand backup and restore.
Point-in-time restore.
Compliance with national, regional, and industry-specific regulations GDPR (for the EU), PDPL (for Saudi Arabia), HIPAA (for the healthcare industry).

Best Big Data Databases for Comparison

Description

A leader among Big Data NoSQL databases in the Forrester Wave Report.

Support for key-value and document data models.
ACID (atomicity, consistency, isolation, durability) transactions.
Integrations with AWS S3, AWS EMR, Amazon Redshift.
Microsecond latency with DynamoDB Accelerator.
Real-time data processing with DynamoDB Streams.
On-demand and provisioned read/write capacity modes.
End-to-end big data encryption.
Point-in-time recovery and on-demand backup and restore.

best for

Operational workloads, IoT, social media, gaming, ecommerce apps.

Pricing

Database operations:

On-demand request units (RU): $1.25/million write RU and $0/25/million read RU.
Provisioned capacity unit (CU): $0.00065/write CU and $0.00013/read CU.

Storage: first 25 GB/month – free, $0.25/GB/month thereafter.

More details: AWS DynamoDB

Description

A leader among Big Data NoSQL databases in the Forrester Wave Report.

Support for the multi-model data schema.
Open-source APIs for SQL, MongoDB, Cassandra, Gremlin, etc.
Integration with Azure Synapse Analytics for real-time no-ETL analytics on operational data.
Support for ACID transactions.
On-demand and provisioned capacity modes.
Big data encryption (in transit and at rest) and access control.
99.999% availability.

best for

Operations management, ecommerce, gaming, IoT apps.

Pricing

Database operations:

Provisioned throughput: 100 request units/second, single-region write account - $0.012/hour (autoscale) and $0.008/hour (manual).
Provisioned throughput reserved capacity: up to 65% savings.
Serverless (bills for the request units (RU) used for each database operation) – $0.25 for 1,000,000 RU.

Storage: 1GB consumed transactional storage (row-oriented) – $0.25/month.

More details: Azure Cosmos DB

Description

Support for Apache CQL API code, Cassandra-licensed drivers and developer tools for running Cassandra workloads.
Big data encryption at rest and in transit.
On-demand and provisioned capacity modes.
Integration with Amazon CloudWatch for performance monitoring.
Continuous backup of table data with point-in-time recovery.
99.99% availability within AWS Regions.
Integration with AWS Identity and Access Management for database access control.

Best for

Fleet management, industrial maintenance apps.

Pricing

Database operations:

On-demand throughput: $1.45/million write RU, $0.29/million read RU.
Provisioned throughput: write RUs - $0.00075/hour, read RUs - $0.00015/hour.

Storage: $0.30/GB/month.

More details: Amazon Keyspaces

Description

MongoDB compatibility.
Support for the ACID transactions.
Migration support (e.g., MongoDB databases on-premises to Amazon DocumentDB) with AWS Database Migration Service.
Support for role-based access with built-in roles.
Network isolation.
Instance monitoring and repair.
Cluster snapshots.

Best for

User profiles, catalogs, and content management.

Pricing

On-demand instances: $0.277- $8.864/instance-hour consumed (Memory Optimized Instances Current Generation).
Database I/O: $0.20/1million request.
Database storage: $0.10/GB/month.
Backup storage: $0.021/GB/month.

More details: Amazon DocumentDB

Description

Flexible database management platform for big data querying with SQL, a leader of Gartner Magic Quadrant for Data Management Solutions for Analytics.
Automated infrastructure provisioning.
On-demand and provisioned capacity modes.
Amazon Redshift Spectrum to query big data in the data lake (Amazon S3).
Federated queries support for operational data querying.
Big data encryption (in transit and at rest).
Network isolation.
Row- and column-level security.

best for

BI and real-time operational analytics on business events.

Not suitable for Online Transaction Processing (OLTP) in milliseconds.

Pricing

On-demand pricing: $0.25/hour (dc2.large) - $13.04/hour (ra3.16xlarge).
Reserved instance pricing allows saving up to 75% over the on-demand option.
Managed storage pricing (for RA3 node types) –$0.024/GB/month.

More details: Amazon Redshift

Big Data Database Implementation by ScienceSoft

Big data consulting

We offer:

Big data storage, processing, and analytics needs analysis.
Big data solution architecture.
An outline of the optimal big data solution technology stack.
Recommendations on big data quality management and big data security.
Big data databases admin training.
Proof of concept (for complex projects).

Go for consulting

Big data database implementation

Our team takes on:

Big data storage and processing needs analysis
Big data solution architecture.
Big data database integration (integration with big data source systems, a data lake, DWH, ML software, big data analysis and reporting software, etc.).
Big data governance procedures setup (big data quality, security, etc.)
Admin and user training.
Big data database support (if required).

Go for implementation

ScienceSoft as a Big Data Consulting Partner

ScienceSoft's team proved their mastery in a vast range of big data technologies we required: Hadoop Distributed File System, Hadoop MapReduce, Apache Hive, Apache Ambari, Apache Oozie, Apache Spark, Apache ZooKeeper are just a couple of names.

ScienceSoft's team also showed themselves great consultants. Special thanks for supporting us during the transition period. Whenever a question arose, we got it answered almost instantly.

Kaiyang Liang Ph.D., Professor, Miami Dade College

Read original

About ScienceSoft

ScienceSoft is a global IT consulting and IT service provider headquartered in McKinney, TX, US. Since 2013, we offer a full range of big data services to help companies select suitable big data software, integrate it into the existing big data environment, and support big data analytics workflows. Being ISO 9001 and ISO 27001-certified, we rely on a mature quality management system and guarantee cooperation with us does not pose any risks to our customers’ data security.