Data Warehouse Implementation in 2021

Data Warehouse Implementation Steps, Costs, Trends - ScienceSoft

ScienceSoft has been providing data warehousing services since 2005.

The Gist of Data Warehouse (DWH) Implementation

Data warehouse implementation implies developing and deploying a data warehouse to gather and structure company’s data for analytical querying and reporting.

Time: From 6-9 months

Data warehouse implementation steps: Feasibility study, discovery, data warehouse conceptualization and platform selection, business planning, data warehouse system analysis and architecture design, development and launch, support and evolution.

Cost: Starts from $70,000

Team: A PM, a BA, a DWH system analyst, DWH solution architect, data engineer, QA and DevOps engineers.

Data Warehouse Solutions

Learn what an enterprise data warehouse (EDW) is and explore its typical architecture, features, and integrations. Find out what software we recommend for building it.

Learn about a typical architecture for a healthcare data warehouse, its features, valuable integrations, factors that model healthcare data warehouse success, major financial outcomes and platforms for building a robust healthcare data warehouse solution.

Data Warehouse Implementation Trends

Trend 1. Moving to the cloud

  • Scalability and flexibility of a cloud data warehouse.

The inherent scalability of a cloud data warehouse allows for easy adaptation to the changing amount of data and the required processing capacity. Thus, scaling the data volume up and down does not affect the performance of the data warehouse.

  • Flexible pricing options.

Cloud providers offer flexible pricing models (e.g., pay-as-you-go) and discount opportunities for provisioned resources to meet their clients’ technical needs and budgets.

  • Data availability.

Nearly all cloud data warehouses perform consistent backups automatically, which results in 99.99% data availability and fault tolerance.

Trend 2. Turning to Data Warehouse as a Service (DWaaS)

Minimizing data administration efforts.

When opting for DWaaS, you eliminate hardware and software acquisition, configuration and maintenance costs. As a DWaaS provider performs data warehouse administration and management, there is no need to hire an in-house team for managing the data storage infrastructure.

Trend 3. Big data integration into data warehouse

Combining historical business data with less structured data from big data sources (machine data, transactional data, public data, etc.) provides for uncovering hidden data patterns and correlations and getting insights that can drive business-improving actions, which is a huge step towards accurate forecasting and boosted profit.

Data Warehouse Implementation Plan

The process of implementing a data warehouse is closely bound to particular business needs and objectives, so the data warehouse implementation steps may differ or merge depending on the project specificity and scale.

1

Data warehouse feasibility study

2

Discovery

3

Data warehouse conceptualization and platform selection

4

Business planning

5

Data warehouse system analysis and architecture design

6

Development and stabilization

7

Data warehouse launch

8

Data warehouse support and evolution

Talents Required for Data Warehouse Implementation

Project manager

Defines and communicates data warehouse implementation project objectives, manages project scope, costs, timing and quality.

Business analyst

Elicits and documents data warehouse solution’s functional and non-functional requirements (including data warehouse solution’s building blocks, integrations with data source systems, etc.), technical limitations (if any).

Data warehouse system analyst

Analyses data sources (and their dependencies) and data analytics software (if any) to be integrated with the data warehouse solution. Reviews data loaded into the data warehouse for accuracy.

Data warehouse solution architect

Draws up data warehouse architecture requirements. Designs data warehouse architecture that supports high availability, performance, scalability, and security of the data warehouse solution.

Data engineer

Develops a data model and its structures, draws up data flows (based on the system analyst’s input). Develops, tests and maintains a data pipeline routing source data to the data warehouse. Builds the ETL/ELT process.

Quality assurance engineer

Conducts data warehouse solution’s requirements analysis, defines a test strategy, and designs an optimal test environment to simulate real-time data warehouse scenarios. Executes test cases to evaluate functional and non-functional aspects of the data warehouse system.

DevOps engineer

Sets up the data warehouse software development infrastructure, automates and streamlines development and release processes by introducing CI/CD pipelines, monitors data warehouse performance, availability, and security.

Sourcing Models

In-house data warehouse implementation

The company has full control over a data warehouse implementation project.

Caution: Not to delay or compromise the project, there should be the sufficient amount of resources and expertise.

Technical resources are partially outsourced

Augmenting the in-house tech team with a vendor’s resources to perform such activities as data warehouse design, implementation or support. The company has substantial control over the implementation project.

Caution: High requirements to in-house competencies. Additionally, there should be effective communication between all stakeholders to avoid project delays.

Technical resources are fully outsourced

Minimized risk of the resource overprovisioning after the project completion.

Caution: High requirements to in-house PM and BA competencies.

In-house project sponsor, everything else is outsourced

Minimized risk of data warehouse implementation project delays or failures due to resource unavailability. A vendor takes on full responsibility for the data warehouse implementation project and all related risks.

Caution: Increased vendor-related risks due to high vendor dependency.

Need Help to Implement a DWH?

With 15+ years in data warehousing, ScienceSoft is ready to advise on, implement and support your data warehouse to help you benefit from a cost-effective and high-performing data warehouse fully meeting your data storage, analytical and reporting needs.

Data Warehouse Software We Recommend

Amazon Redshift

Best for: petabyte-scale analytics

DESCRIPTION

  • Integration of structured, semi-structured, unstructured data types.
  • SQL data querying (including big data).
  • Integrations with the AWS ecosystem (including S3, AWS Glue, Amazon EMR) and third-party tools (Power BI, Tableau, Informatica, Qlik, Talend Cloud).
  • Automated infrastructure provisioning, backups and cluster health monitoring.
  • Federated query support and result caching.
  • ML-optimized performance under varying workloads.
  • Data encryption in transit and at rest and fine-grained access control.
  • Separate scaling of compute and storage.

Pricing

  • On-demand pricing: $0.25/hour (dc2.large) - $13.04/hour (ra3.16xlarge).
  • Reserved instance pricing can save up to 75% over the on-demand option (in a 3-year term).
  • Data storage (RA3 node types): $0.024/GB/month.

Azure Synapse Analytics

Best for: advanced data management

DESCRIPTION

  • SQL querying of structured, semi-structured, unstructured data types.
  • Multilanguage support (T-SQL, Python, Scala, Spark SQL, .Net).
  • Native integrations with Apache Spark, Power BI, Azure ML, Azure Stream Analytics, Azure Cosmos DB, etc.
  • Integration with third-party BI tools, including Tableau, SAS, Qlik, etc.
  • Result-set caching.
  • Automatic restore points and backups.
  • End-to-end data encryption, dynamic data masking, granular access control.

Pricing

  • Compute on-demand pricing: $1.20/hour (DW100c) - $360/hour (DW30000c).
  • Compute reserved instance pricing can save up to 65% over the on-demand option (in a 3-year term).
  • Data storage: $122.88/TB/month.

Oracle Autonomous Data Warehouse

Best for: high-speed query processing

Description

  • Querying across structured, semi-structured, unstructured data types.
  • Connection with custom applications and third-party products via SQL*Net, JDBC, ODBC.
  • Connectivity to Oracle Cloud Infrastructure Object Storage, Azure Blob Storage, Amazon S3.
  • Native integration with Oracle Analytics Desktop.
  • Deployment flexibility (Oracle public cloud (shared/dedicated infrastructure) or a customer’s data center).
  • Automated scaling, performance tuning, patching and upgrades, backups and recovery.
  • Independent storage and compute scaling.
  • Data encryption at rest and in transit.
  • Multifactor authentication.

Pricing

  • Compute: $1.3441/CPU/hour.
  • Data storage: $118.40/TB/month (in the public cloud).

Get Advice on Optimal DWH Software

ScienceSoft is ready to help you choose optimal data warehouse technologies to reduce data warehouse implementation and maintenance costs and maximize ROI.

Data Warehouse Implementation Costs

A data warehouse implementation project, which involves developing a 10GB data warehouse with data integration and data cleansing processes, may vary from $225,000 to $485,000 (excluding software licensing and other regular fees).

The major factors that influence data warehouse implementation costs are:

  • Number of data sources (ERP, CRM, SCM, etc.) and their complexity (whether a data model description exists or not; if the data model differs from the new data warehouse data model).
  • Data volume.
  • Complexity of data cleansing.
  • Number of data tables and columns used for analysis.
  • Required security level and policies.
  • Data warehouse velocity, scalability, and fault tolerance.

Consider Professional Services for Data Warehouse Implementation

ScienceSoft has been providing data warehouse services since 2005 and can help you build a data warehouse solution fully aligned with your business objectives with optimized investments involved.

Data warehouse implementation consulting

  • Data warehouse implementation feasibility study.
  • Data warehouse solution conceptualization and platform selection.
  • Data warehouse system analysis and architecture design.
  • Data warehouse solution implementation strategy.
  • Optimal data warehouse implementation sourcing model.

Data warehouse implementation outsourcing

  • Data warehouse implementation feasibility study.
  • Data warehouse solution conceptualization and platform selection.
  • Data warehouse system analysis and architecture design.
  • Data warehouse solution development.
  • Data warehouse quality assurance and launch.
  • Data warehouse support and evolution.

About ScienceSoft

ScienceSoft is a global IT consulting and software development company headquartered in McKinney, TX, US. Since 2005, we’ve been helping companies handle and benefit from data with a full range of data warehousing services, including data warehouse consulting, data warehouse implementation, data warehouse migration and support, and Data Warehouse as a Service (DWaaS). Being ISO 9001 and ISO 27001-certified, we rely on a mature quality management system and guarantee cooperation with us does not pose any risks to our customers’ data security.