Data Integration

Need for Data Integration

• DI is necessary because of the way information systems have been architected and grown over the last 20-30 years - a situation exacerbated by rapid organizational expansion, tactical investment strategies and M&A activity. Typically, this has resulted in enterprise data being fragmented across multiple IT solutions and their corresponding data silos. At the same time, enterprises now want to use more data, and a more diverse range of data, for both operational and strategic purposes. This is driving the need for DI – particularly in larger organisations.
  
  

• What is data integration?

  In essence, DI involves combining different sets of data to provide a unified view of all the relevant data. It is a concept that is implemented through a combination of methodologies and technologies, and it encompasses database migration/upgrading, application migration/consolidation, operational DI and integration to support data analytic.
  
  

• What does data integration encompass?

  There are a number of approaches to delivering DI and these can be used individually or in combination, as appropriate:
  
  
  1. Data consolidation involves collecting data from multiple data sources and consolidating it in a persistent data store. This may involve, for example, migrating data from multiple existing datasets and consolidating these into a single persistent dataset as part of an application migration or consolidation initiative. A number of technologies can be used to assist with data consolidation, including extract-transform-load (ETL) tools and so-called 3G data migration tools.
     
     
  2. Data federation provides a unified view of an organisation’s data through a single interface, enabling disparate data sets to appear as a single homogeneous data set to the user. This is also known as enterprise information integration or EII.
     
     
  3. Data propagation involves replicating data from different sources in different locations and encompasses enterprise data replication (EDR), database log scrapers and change data capture (CDC) tools. Enterprise application integration (EAI) technology supports the integration of application systems, enabling them to exchange data using standard interfaces.
     
     
  4. Data access uses search capabilities to increase the accessibility of data. This is also known as enterprise information access or EIA.
     
     

• How does poor data integration drive up costs?

  Poor DI increases the costs for businesses in a number of ways (see Figure 1). These costs are either:
  
  
  1. Direct costs – for example, due to higher hardware and software licences, and the requirement for greater manual effort.
     
     
  2. Indirect costs – for example, opportunity costs, costs arising from poorer or slower business decision-making, or as a consequence of negative impacts on customers (resulting in higher rates of customer churn and the costs incurred from dealing with complaints)
     
     

• Using data integration to deliver commercial success

  1. Integrating business data creates a wide range of benefits for an organisation,helping to deliver commercial success and competitive differentiation.
     
     
1. In this section we look at three main areas that businesses can focus on to deliver the commercial benefits we have outlined and to reduce costs:
   
   
   1. Reducing the cost of the DI project itself
      
      
   2. Reducing the direct costs of poor data integration
      
      
   3. Reducing the indirect costs of poor data integration.
      
      

2. 5.1 Reducing the cost of the DI project can be achieved by

   1. Capturing business requirements accurately
      
      
   2. Paying attention to Data Quality
      
      
   3. Planning an achievable project
      
      
   4. Soft out the politics
      
      
   5. Review your use of DI technology and pay attention to license cost and other related tool cost.
      
      

3. 5.2 Reducing the direct costs of poor data integration
   

4. 
   5.2.1 Hardware cost

5. Reduction in Hardware csot can be achieved by
   
   
   1. Reduction in storage of duplicated data
      
      
   2. Reduction in aging hardware infrastructure
      
      
   3. Reduction of necessary use of high-end hardware
      
   4. Reduction in under-utilisation of hardware
      
6. Strategies such as migrating and consolidating data on a modern, lower-cost platform, employing visualisation and cloud computing are all being used to lower the cost of hardware for organisations. DI technology is an enabler of all of these initiatives.
   
   

7. 5.2.2 Software costs

8. Most organisations are paying too much for their software. A review often reveals:
   
9. software that is no longer being used or is underused, but is still being supported and maintained. Typically, 75% of the lifetime cost of software derives from maintenance costs, which means managing it and understanding lifetime TCO is of vital importance licensing costs that have risen exponentially due to the pricing model being used.
   
   
10. Many organisations still only consider the initial purchasing price rather than TCO and are shocked when they realise how much software is costing now their business has grown licensing costs that are inflated because the IT department bought more than it needed (eg to get volume discounts or because the organisation has subsequently contracted in size) inflated costs for licensed software where lower-cost viable alternatives exist.
    
    Many IT users buy into a brand. Where such software is delivering significant added value then there may be a business case for paying more; but sometimes the software is delivering no more added value than a lower cost or open source alternative departmental purchasing strategies that fail to leverage volume economies ‘bloatware’ costs – where IT departments buy far more functions than they need or are used. This also adds to hardware, training and maintenance costs.
    
    

11. 5.2.3 Manpower costs

12. Many companies are faced with higher than necessary operational costs (OPEX) because the legacy software/platforms they use require considerable manual intervention and the skills needed to maintain them are scarce and expensive.
    
    Reducing the direct costs of poor data integration can be achieved by
    
    
    1. Paying attention to licensing cost
       
       
    2. Application consolidation
       
       
    3. Pay attention to storage costs
       
       
    4. Automate as much as possible
       
       
    5. Choose a flexible DI tool
       
       

13. 5.3 Reducing the indirect costs of poor data integration by

    1. Understanding the total effect of Sub optimal DI
       
       
    2. Dont underestinate risk and compliance issues