What exactly is data?
We all have heard of data. We all generate it, use it, delete it (or so we believe), process it and share it. But, have you ever wondered what exactly data is? Well, I pondered on this question for a while and this is I think that describes the word data correctly: “Data is unorganized information that is processed to make it meaningful”.
Types of data?
Now, data can be categorized in many ways. But one way that is quite important for working with data is by it’s structure. Data can be:
Structured
Semi-structured
Unstructured
Structured Data
Structured data has a well-defined structure or adheres to a specified data model, can be stored in well-defined schemas such as databases, and in many cases can be represented in a tabular manner with rows and columns. Structured data is objective facts and numbers that can be collected, exported, stored, and organized in typical databases.
Some of the sources of structured data could include:
SQL databases and Online Transaction Processing (or OLTP) Systems that focus on business transactions.
Spreadsheets such as Excel and Google Spreadsheets.
Online forms
Sensors such as GPS and Radio Frequency Identification (or RFID) tags.
Network and Web server logs
You can typically store structured data in relational or SQL databases. You can also easily examine structured data with standard data analysis methods and tools.
Semi-Structured Data
Semi-structured data is data that has some organizational properties but lacks a fixed or rigid schema. Semi-structured data cannot be stored in the form of rows and columns as in databases. It contains tags and elements, or metadata, which is used to group data and organize it in a hierarchy. Some of the sources of semi-structured data could include:
E-mails, XML, and other markup languages.
Binary executables
TCP/IP packets
Zipped files
Integration of data from different sources.
XML and JSON allow users to define tags and attributes to store data in a hierarchical form and are used widely to store and exchange semi-structured data.
Unstructured Data
Unstructured data is data that does not have an easily identifiable structure and, therefore, cannot be organized in a mainstream relational database in the form of rows and columns. It does not follow any particular format, sequence, semantics, or rules. Unstructured data can deal with the heterogeneity of sources and has a variety of business intelligence and analytics applications. Some of the sources of unstructured data could include:
Web pages
Social Media feeds
Images in varied file formats (such as JPEG, GIF and PNG)
Video and audio files
Documents and PDF files
PowerPoint presentations
Media logs
Surveys
Unstructured data can be stored in files and documents (such as a Word doc) for manual analysis or in NoSQL databases that have their own analysis tools for examining this type of data.
To summarize, structured data is data that is well organized in formats that can be stored in databases and lends itself to standard data analysis methods and tools; Semi-structured data is data that is somewhat organized and relies on meta tags for grouping and hierarchy; and Unstructured data is data that is not conventionally organized in the form of rows and columns in a particular format.
Data Sources…
Data sources have never been as dynamic and diverse as they are today. Let’s look at these!
Relational Databases:
Typically, organizations have internal applications to support them in managing their day to day business activities, customer transactions, human resource activities, and their workflows. These systems use relational databases such as SQL Server, Oracle, MySQL, and IBM DB2, to store data in a structured way. Data stored in databases and data warehouses can be used as a source for analysis. For example, data from a retail transactions system can be used to analyze sales in different regions, and data from a customer relationship management system can be used for making sales projections.
Flat File and XML Datasheets:
External to the organization, there are other publicly and privately available datasets. For example, government organizations releasing demographic and economic datasets on an ongoing basis. Then there are companies that sell specific data, for example, Point-of-Sale data or Financial data, or Weather data, which businesses can use to define strategy, predict demand, and make decisions related to distribution or marketing promotions, among other things. Such data sets are typically made available as flat files, spreadsheet files, or XML documents.
Flat Files:
Store data in plain text format
Each line, or row, is one record
Each value is separated by a delimiter
All of the data in a flat file maps to a single table
Most common flat file maps to a single table
Most common flat file format is .CSV
Spreadsheet Files:
Special type of flat files
Organize data in a tabular format
Can contain multiple worksheets
.XLS or .XLSX are common spreadsheets formats
Other formats include Google Sheets, Apple Numbers, and LibreOffice Calc
XML Files:
Contain data values that are identified or marked up using tags
Can support complex data structures
Common uses include online surveys, bank statements, and other unstructured data sets
API and Web Services:
Many data providers and websites provide APIs, or Application Program Interfaces, and Web Services, which multiple users or applications can interact with and obtain data for processing or analysis. APIs and Web Services typically listen for incoming requests, which can be in the form of web requests from users or network requests from applications, and return data in plain text, XML, HTML, JSON, or media files. Let’s look at some popular examples of APIs being used as a data source for data analytics:
The use of Twitter and Facebook APIs to source data from tweets and posts for performing
tasks such as opinion mining or sentiment analysis—which is to summarize the amount of appreciation and criticism on a given subject, such as policies of a government, a product, a service, or customer satisfaction in general.
Stock Market APIs used for pulling data such as share and commodity prices, earnings per
share, and historical prices, for trading and analysis.
Data Lookup and Validation APIs, which can be very useful for Data Analysts for cleaning
and preparing data, as well as for co-relating data—for example, to check which city or
state a postal or zip code belongs to.
Web Scraping:
Web scraping is used to extract relevant data from unstructured sources. Also known as screen scraping, web harvesting, and web data extraction, web scraping makes it possible to download specific data from web pages based on defined parameters. Web scrapers can, among other things, extract text, contact information, images, videos, product items, and much more from a website. Some popular uses of web scraping include:
Collecting product details from retailers, manufacturers and eCommerce websites to provide price comparisons.
Generating sales leads through public data sources
Extracting data from posts and authors on various forums and communities
Collecting training and testing datasets for machine learning models
Some of the popular web scraping tools include BeautifulSoup, Scrapy, Pandas, and Selenium.
Data Streams:
Data streams are another widely used source for aggregating constant streams of data flowing from sources such as instruments, IoT devices, and applications, GPS data from cars, computer programs, websites, and social media posts. This data is generally timestamped and also geo-tagged for geographical identification. Some of the data streams and ways in which they can be leveraged include:
Stock and market tickers for financial trading
Retail transaction streams for predicting demand and supply chain management
Surveillance and video feeds for threat detection
Social media feeds for sentiment analysis
Sensor data feeds for monitoring industrial or farming machinery
Web click feeds for monitoring web performance and improving design
Real-time flight events for rebooking and rescheduling
Some popular applications used to process data streams include Apache Kafka, Apache Spark Streaming, and Apache Storm.
Really Simple Syndication (RSS):
RSS (or Really Simple Syndication) feeds, are another popular data source. These are typically used for capturing updated data from online forums and news sites where data is refreshed on an ongoing basis. Using a feed reader, which is an interface that converts RSS text files into a stream of updated data, updates are streamed to user devices.
