Enterprise Data Management: Implementation Plan, Talents and Tools
ScienceSoft provides data management services to help companies ensure high data quality, safe and easy data access for informed decision-making across the company.
Enterprise Data Management: The Essence
Enterprise data management involves establishing data governance (standards and policies to organize and manage data access, collection, storage, analysis, security, etc.) and completing individual data management projects to implement data governance across the company.
Enterprise data management implementation timeframes: Full-scale data management implementation for a 5,000-employee company may take from 24 to 36 months.
Key steps: Enterprise data management strategic planning; planning, development and launch of constituent data management projects.
Enterprise data management team: Project manager, system analyst, solution architect, data engineer, DevOps engineer, QA engineer.
The suggested plan is based on ScienceSoft’s 33-year experience in providing data analytics and data management services. It features a sample procedure we follow when establishing enterprise data management for our customers.
Enterprise data management strategic planning
Step 1. Defining data management objectives
Data management objectives are based on the overall corporate objectives and used when establishing data governance policies and KPIs to assess data management implementation success. Common enterprise data management objectives usually include:
- Data is easily accessed, used properly and to its max value.
- Data is of acceptable quality.
- Data is secure.
- Data is compliant with standards and regulations (GDPR, HIPAA, etc.).
ScienceSoft’s best practice: When defining data management objectives, our consultants interview all stakeholders to collect their needs, goals, and vision of the successful data management project implementation. It helps understand their priorities, plan the development process accordingly and as a result - provide a satisfactory end product.
Step 2. Evaluating the current state of data management
It’s necessary to analyze:
- Existing data architecture and data flows.
- Data source systems (their number, connectivity between systems, who has access to them, etc.) and data stored there (data type and structure, volume, sensitivity level, etc.).
- Applied data security practices (data access and usage policies, who monitors compliance with data security policies, etc.).
- Data quality management practices (established data quality metrics, data auditing policies, etc.).
- Master and metadata management practices, etc.
Step 3. Establishing data governance
- Defining data governance policies and standards for each enterprise data management component:
System architecture diagrams, entity-relationship diagrams, data flow diagrams, etc., which provide an overview of how data is captured, ingested, transformed, modeled, and put to usage across the company.
Data integration policies and standards for consolidating data from disparate sources into a single dataset via ETL/ELT processes or data virtualization.
Data quality management
Rules for data quality (e.g., data profiling, cleansing, and matching) to ensure data is fit to serve its objectives.
Policies for controlling the data storage environment (stored data types, data location, recovery procedures, storage environment performance, etc.).
User authentication and authorization policies, data access policy, data audit policy, etc., to prevent unauthorized data access and its inappropriate usage.
Reference and master data management
Data deduplication and standardization policies to enable master and reference data consistency across transactional and business intelligence systems.
Business glossaries and data lineage traceability documents, which maintain information on data definition, relationships, movement, etc.
Data warehousing, analytics, and reporting
Policies for providing and overseeing data analytics processes and infrastructure.
Policies for storing, sharing, indexing and auditing corporate data stored outside relational databases for maintaining data integrity and enabling easy data access.
Implementation of data management projects
Once data governance is established, an enterprise can define an enterprise data management implementation plan and create implementation roadmaps for the entire program and then for each data management component.
ScienceSoft implements a data management program with the following steps for its elements:
Requirement engineering and architecture design
- Engineering requirements for a data management element.
- Reviewing technology stack currently used to enable the element.
- Outlining the optimal architecture and feature set for the future technical solution.
- Defining the optimal tech stack.
Planning the implementation of a data management project
- Determining a data management project scope, deliverables and timeframes.
- Defining a data management project team and designing collaboration workflows.
- Developing a project KPI suite based on the set data governance standards.
- Outlining possible project risks and defining a risk mitigation strategy and plan.
- Deciding on the relevant sourcing model.
- Estimating efforts, TCO and ROI for the data management project.
ScienceSoft’s tip: Our practice has shown that effective data management project planning can help reduce project time and budget by up to 30%.
Technical solution development and launch
- Implementing a technical solution for each data management component and integrating it into the existing IT environment.
- Deploying the tech solution in production.
- User acceptance testing.
- User training.
ScienceSoft’s best practice: To deliver a tech solution for each data management component in the shortest time possible, in our projects we opt for DevOps-driven iterative development – it assures the quickness and frequency of releases without sacrificing the solution’s quality.
Consider Professional Services for Enterprise Data Management Implementation
Enterprise data management consulting
- Defining data management objectives.
- Evaluating the existing data management maturity level.
- Creating a business case for a data management program.
- Establishing data governance.
- Planning out data management implementation.
- Defining the architecture and optimal tech stack for each data management component.
Enterprise data management implementation
- Defining data management objectives.
- Evaluating the existing data management maturity level.
- Establishing data governance.
- Architecture design and software selection for a technical solution/solutions.
- Development and stabilization of a tech solution for each data management program element.
- Tech solution launch.
ScienceSoft as a Trusted Data Management and Data Analytics Partner:
When we first contacted ScienceSoft, we needed expert advice on the creation of the centralized analytical solution to achieve company-wide transparent analytics and reporting.
The system created by ScienceSoft automates data integration from different sources, invoice generation, and provides visibility into the invoicing process. We have already engaged ScienceSoft in supporting the solution and would definitely consider ScienceSoft as an IT vendor in the future.
Heather Owen Nigl, Chief Financial Officer, Alta Resources
- Outlines the scope of the data management program.
- Determines the efforts, timelines, KPIs, and deliverables for the data management program and program elements.
- Assembles and helps establish the collaboration within the team involved in the enterprise data management program.
- Monitors and manages the costs of the program and program elements.
- Oversees the overall program execution.
Data management consultant
- Communicates with stakeholders to elicit data management needs and expectations.
- Analyzes the existing data management practices and environment (data architecture, data flows, etc.), data governance policies, etc.
- Helps conceptualize a data management program and develop an implementation strategy.
- Helps develop data governance policies and monitor compliance with them.
- Coordinates the creation of the overall enterprise data management project documentation.
- Conducts training for users (for example, on data governance policies and standards, on using new software, etc.).
- Assesses the existing data management environment, defines issues and optimal ways for improvement.
- Creates technical solution architecture based on the elicited business requirements.
- Recommends and implements appropriate tech approaches for enterprise data management, recommends a tech stack and assesses the business impact certain tech choices have.
- Provides supervision to development teams.
- Analyzes source systems, identifies data quality issues, their root causes, and steps to resolve them.
- Builds data flows, develops conceptual, physical and logical data models. Maintains data models along with corresponding metadata.
- Builds ETL/ELT processes to route source data to the destination.
- Manages master data, including creation, updates, and deletion.
- Introduces continuous integration/continuous delivery (CI/CD) pipelines to automate and streamline the development of technical solutions for data management components.
- Monitors the IT infrastructure state, handles ongoing maintenance.
- Designs a test strategy, create tests to evaluate the developed technical solutions for a data management program.
- Plans and coordinates testing activities, documents test results.
- Provides recommendations on data management software improvements.
Azure Synapse Analytics
Best for: enterprise data warehousing
Leader in the Gartner’s 2020 Magic Quadrant for Cloud Database Management Systems and the Forrester Wave for Cloud Data Warehouse reports. A leading data management platform for analytics, according to the Gartner’s Magic Quadrant report.
- 90+ pre-built data source integrations, including native integrations with Apache Spark, Power BI, Azure ML, Azure Stream Analytics, Azure Cognitive Services, Azure Cosmos DB, etc.
- SQL querying of structured, semi-structured, unstructured data types.
- Compatibility with Python, R, .Net, Scala, Spark SQL foe enabling analytics of various complexity.
- Integration with third-party BI tools, including Tableau, SAS, Qlik, etc.
- Result-set caching, workload isolation and clustered columnstore indexes.
- Automatic restore points and backups.
- End-to-end data encryption, dynamic data masking, granular access control.
- On-demand pricing - $1.20 - $360/hour.
- Reserved instance pricing offers saving up to 65% over the on-demand option (in a 3-year term).
Data storage: $122.88/TB/month.
Best for: big data warehousing
- Analytical querying over structured, semi-structured and unstructured data.
- Integrations with the AWS ecosystem (Amazon S3, Amazon Athena, Amazon SageMaker, Amazon EMR, etc.)
- Integration with third-party ETL/ELT, BI, data modeling, etc. tools (Power BI, Tableau, Informatica, Qlik, Talend Cloud, and more).
- Accommodating big data workloads with the Advanced Query Accelerator, result caching, materialized views and ML-based workload management
- Querying data from operational and relational databases on AWS with the federated query possibility. ML-optimized performance under varying workloads.
- Automated infrastructure provisioning, table design, backups and cluster health monitoring.
- Data encryption in transit and at rest.
- Fine-grained access control.
- Separate scaling of compute and storage.
- On-demand pricing: $0.25 - $13.04/hour.
- Reserved instance pricing can save up to 75% over the on-demand option (in a 3-year term).
Data storage (RA3 node types): $0.024/GB/month.
Microsoft SQL Server Integration Services (SSIS)
Best for: enterprise-level on-premises data integration and transformation
- A broad range of data management capabilities (data mining, cleansing, master data management, etc.)
- Using connection managers to link to a variety of data sources, both relational and non-relational.
- Intuitive UI and the drag-and-drop feature for creating data integration workflows code-free.
- Alerting in case some data integration activity has failed, so the administrator may immediately troubleshoot without re-running the activity.
- Comprehensive documentation and digital training resources.
SQL Server Integration Services (SSIS) is available under the SQL Server licensing.
- Express and Developer editions – free
- Standard – $3,586/core
- Enterprise - $13,748/core
- SSIS integration runtime on Microsoft Azure - $0.84 - $35.372/hour
ScienceSoft is an IT consulting and software development company headquartered in McKinney, Texas. We help enterprises implement complex data management initiatives to maximize data value and enhance decision-making. Being ISO 9001 and ISO 27001 certified, we provide data management services relying on a mature quality management system and guarantee cooperation with us does not pose any risks to our customers’ data security.