Data Collection, Problems, and Quality

* Methods for collecting raw data

• Manually or by instruments and sensors
• Surveys
• Scanners

* Data problems
* Data quality

• Contextual DQ
• Intrinsic DQ
• Accessibility DQ
• Representation DQ

* Data integrity

• Uniformity
• Version
• Completeness check
• Conformity check
• Genealogy check or ‘drill down’

* Data access and integration

• Data integration software

Data Aquisition: The nature and sources of data

Data, Information, & Knowledge

• Data – items about things
• Information – data that have been ‘massaged’ (organised/maniupulated)
• Knowledge – information, experience, learning, expertise

Internal data

• Stored in more than one place
• About people, products, services, and processes
• Available via
• Intranet
• Other internal network

External data

• Many sources
• Usually irrelevant to specific MSS
• Needs to be monitored and captured in context to business needs and operations

Personal data

• Users’ own expertise, knowledge, opinions, interpretations

self assessment: Data Warehouse

Self Assessment Questions/Discussion Topics

1. Define a data warehouse, and list some of its characteristics.
2. What is the difference between a database and a data warehouse?
3. Describe OLAP.
4. Discuss the relationship between multiple sources of data, including external data, and the data warehouse.
5. Explain the relationship between SQL and a DBMS
6. Describe multidimensionality and explain its potential benefits for MSS/DSS.

On-Line Analytical Processing

Four Main Characteristics of OLAP

• Use multidimensional data analysis techniques
• Provide advanced database support
• Provide easy-to-use end user interfaces
• Support client/server architecture

OLAP Architecture

• OLAP Graphical User Interface (GUI)
• OLAP Analytical Processing Logic
• OLAP Data Processing Logic

OLAP systems are designed to use both operational and Data Warehouse data.

Multidimensional Analysis

Common decision maker requirements:

• Summarised information
• Ability to ‘slice and dice’ information
• Display information
• View information over time

Multidimensional Data Analysis

• Data viewed as part of a multidimensional structure
• Allows users to consolidate or aggregate data at different levels
• Allows business analyst to easily switch business perspectives

Additional Functions of Multidimensional Data Analysis

• Advanced data presentation functions
• Advanced data aggregation, consolidation, and classification functions
• Advanced computational functions
• Advanced data modeling functions

Twelve Rules That Define a Data Warehouse

1. Data Warehouse and operational environments are separated

2. data are integrated

3. contains historical data over a long time horizon

4. snapshot data captured at a given point in time

5. subject-oriented

6. mainly read-only

7. development is data driven; the classical approach is process driven

8. contains data with several levels of detail

9. characterized by read-only transactions to very large data sets

10. traces data resources, transformation, and storage

11. metadata are a critical component of this environment

12. contains a charge-back mechanism for resource usage

Characteristics of the Data Warehouse

The Data Warehouse is a database that provides support for decision making.

In simple terms, a data warehouse (DW) is a pool of data produced to support decision making; it is also a repository of current and historical data of potential interest to managers throughout the org.

Integrated

• Integration is closely related to subject orientation. Data warehouses must place data from from different sources into a consistent format. To do so they must deal with naming conflicts and discrepancies among units of measure. A data warehouse is assumed to be totally integrated.

Subject-Oriented

• Data are organised by detailed subject such as sales, products or customers, containing only information relevant for decision support. Enables users to determin not only how the business is performing but why. differs from an operation database in that most operationAL databases have a product orientation and are tuned to handle transactions that update the database. Comprehensive view of the organisation.

Time Variant (time series) (built-in time aspects)

• Maintains historical data. Does not necessarily provide current status (except in real time systems). They detect trends, deviations long-term relationships for forecating and comparrisons, leading to decision making. There is a temporal quality to every data warehouse. Time is the one important dimension that all data warehouses must support. Data for analysis from multiple sources contain multiple time points (daily, weekly, monthly).

Non-Volatile (can't be changed)

• After data are enetered into a data warehouse, users cannot change or update the data. Obsolete data are discarded, and changes are recorded as new data. Enables the data warehouse to be tuned almost exclusively for data access.

Summarized
Not normalized
Sources
Metadata (Data about data)

Strategic (DSS) Data and Operational Data

Three Main Areas in Which Strategic (DSS) Data Differ from Operational Data

• Time span
• Granularity
• Dimensionality

Decision Support Systems - Main Components :: page 92

Components of a DSS are:

• Data management
• Model management
• User Interface management
• Knowledge-based management

an arrangement of computerized tools used to assist managerial decision making

• requires extensive data “massaging” to produce information.
• used at all levels within an organization
• interactive and provides ad hoc query tools

• External data
• operational data
• business data
• data store
• data extracting and filtering
• end user query tool
• business model data
• end user presentation tool

Operational Data vs. Strategic (DSS) Data

• operational data are stored in a relational database
• data storage is optimized
• operational data capture daily business transactions
• Strategic data give tactical and strategic business meaning to the operational data

Data Visualization

Technologies supporting visualization and interpretation

• Digital imaging, GIS, GUI, tables, multidimensions, graphs, VR, 3D, animation
• Identify relationships and trends

Data manipulation allows real time look at performance data

Knowledge Discovery in Databases

Data mining used to find patterns in data

• Identification of data
• Preprocessing
• Transformation to common format
• Data mining through algorithms
• Evaluation

Tools and Techniques

Data mining

• Statistical methods
• Decision trees
• Case based reasoning
• Neural computing
• Intelligent agents
• Genetic algorithms

Text Mining

• Hidden content
• Group by themes
• Determine relationships

Data Mining

• Organizes and employs information and knowledge from databases
• Statistical, mathematical, artificial intelligence, and machine-learning techniques
• Automatic and fast
• Tools look for patterns

1. Simple models
2. Intermediate models
3. Complex Models

OLAP - Online analytical processing

Activities performed by end users in online systems

• Specific, open-ended query generation

1. SQL

• Ad hoc reports
• Statistical analysis
• Building DSS applications

Modeling and visualization capabilities

Special class of tools

• DSS/BI/BA front ends
• Data access front ends
• Database front ends
• Visual information access systems

Business Analytics

Business Analytics focuses on effective use of data and information to drive positive business actions. The body of knowledge for this area includes both business and technical topics, including concepts of performance management, definition and delivery of business metrics, data visualization, and deployment and use of technology solutions such as OLAP, dashboards, scorecards, analytic applications, and data mining.