Mining for information

DON"T look now... You are being monitored...! Whenever you withdraw money from a bank, log on to the Internet, make a phone call or for that matter, even when you rent a video, the transa

tions are recorded and stored in computers. As a result, a huge mass of data is piling up in the electronic vaults of companies, research institutions and government offices.

But why should the data be stored? What is the use of this data? Till the 1990s, the answer was "not much". But now the society as a whole as well as all the companies are looking forward for 'data miners', who can use faster analysing tools that can help sift and analyse the stockpiles of data, turning up surprisingly valuable information.

What is data mining? And who are data miners?

Data mining can be defined as the exploration and analysis of large data sets, in order to discover meaningful patterns and rules. Staring at a huge spreadsheet is not a good way to analyse any data. The trick is to find effective ways to combine the computer's power to process data with the human eye's ability to detect patterns. The techniques of data mining are designed for, and work best with, large data sets.

And data miners are the people who ascertain the truth of the assumptions and the assertions made by explorers. They know how to efficiently process a lot of data. Data miners mine data, understand the hidden relationships, patterns and trends, and help their managers or decision makers make effective and actionable-decisions.

Data mining is a component of a wider process called `knowledge discovery from databases'. It involves scientists from a wide range of disciplines, including mathematicians, computer scientists and statisticians, as well as those working in fields such as machine learning, artificial intelligence, information retrieval and pattern recognition.

Many companies today are inundated with information whose magnitude is exponential. Subsequently, it is hard for decision makers to make sense of the data they have and arrive at actionable decisions. They know what they want, but they do not know what to ask for, more importantly how to ask. The time has come for user-managers to take this challenge in their own hands. The ability to handle data mining process independently has already become a necessity for managers to keep their businesses ahead of the game.

The 1990's saw the emergence of data-warehousing and data mart technologies. However, to derive meaningful information by way of relationship and patterns, statisticians employed elaborate methodologies. Soon after, machine-learning techniques based on human learning models emerged. Thus the convergence of database technologies, machine-learning techniques, and computational methodologies has lead to the development of current business data mining technology. Today, data mining is a joint undertaking of the statistical analysts and data-mining technologists (data miners). The technologists prepare the data, run the databases and the data mining software and statisticians identify the relevant variables, create models and interpret results. Once actionable decisions are implemented, often the data-mining department conducts impact analysis periodically.

Data mining has considerable commercial applications, but it can also be applied in many other fields. Abroad it has been used by law enforcement agencies to identify criminals - by looking for patterns and relationships in the texts of statements taken from dozens or hundreds of suspects. Data mining is also well suited to the analysis of scientific data, such as those amassed by astronomers.

Data Mining is a hot concept that it's showing up in non specialised tools. For example, high-end financial and statistical analysis, decision-support, and EIS vendors are adding DM capability to their products. Also, enterprise database vendors such as IBM and Hewlett-Packard are creating data- warehouse suites (IBM's Visual Warehouse) or virtual-warehouse frameworks (HP's Open Warehouse) that include DM tools. sectors like the banks and research institutes too look out for them.

In today's global corporate environment, data mining technology plays a crucial role as a decision support tool. Before long businesses will witness increasing dependence on powerful data mining tools for reliable and effective decision-making. This requires thorough, fast and easily accessible supporting information. The emergence of data miners, equipped with working knowledge and technological skills and sufficient business savvy are the immediate solutions to the critical problem of making effective decisions and of keeping today's corporations successful.

TINA MARIAM JACOB

tina.mas@cnkonline.com