{"id":397,"date":"2012-08-01T14:36:00","date_gmt":"2012-08-01T18:36:00","guid":{"rendered":""},"modified":"2015-11-30T04:57:26","modified_gmt":"2015-11-30T09:57:26","slug":"21st-century-statistical-systems","status":"publish","type":"post","link":"https:\/\/jeremy-wu.info\/?p=397","title":{"rendered":"21st Century Statistical Systems"},"content":{"rendered":"
\n
\n
\n
\n<\/p>\n

A combination of traditional censuses and the introduction of random surveys served to measure and infer on populations and economies in the 20th century.\u00a0 Both statistical approaches have been important in supporting decision- and policy-making worldwide, as well as informing the public.\u00a0 The 21st century has begun with the massive conversion to digital data and the explosive growth of Big Data around the globe, which in turn stimulates an insatiable demand for ever more timely and comprehensive response to information needs.\u00a0 Since conventional approaches to censuses and surveys are static and cross-sectional, they will not be able to meet these expanding dynamic requirements without fundamental changes.\u00a0 In the 21st century the defining characteristics of statistical systems and methods will be the sophisticated application of massive longitudinal data, integration of multiple data sources, and rapid and simple delivery of results, while still strictly protecting confidentiality and data security and assuring accuracy and reliability.\u00a0 Leaders of the statistical agencies from several nations including the United States have recently reiterated these needs and trends.\u00a0 Government agencies that can successfully overcome these issues will help their nations enjoy unique advantages in global competition; otherwise, they will face certain obsolesence.\u00a0 As a rapidly growing economic power whose statistics are receiving more attention and having greater impact in the world, China faces many of the same challenges.\u00a0 This article identifies some of the success stories emerging in the U.S. and other nations and discusses the needed changes in statistical paradigms to meet the challenges of dynamic, integrated data systems for the 21stcentury.<\/p>\n

The 20th Century Statistical Systems<\/strong><\/h4>

Taking a census, i.e. collecting data from every entity in a target population, has been the traditional statistical method to measure the profile and characteristics of a population for centuries.\u00a0 China reported its first population census more than 2,200 years ago [1].\u00a0 By the Western Han Dynasty around 2 A.D., available records [2,3,4,5] placed China\u2019s population at almost 58 million in over 12 million households. \u00a0The People\u2019s Republic of China enacted its first laws on the governance of statistics in 1983 [6].\u00a0 China has taken six population censuses since 1949, and every ten years since 1990 [7].\u00a0 Continuing for more than two centuries as required by its constitution, the United States (U.S.) conducted its 23rdand most recent decennial national population census in 2010 [8,9].<\/p>

Myriad other topical censuses, such as those on the economy, industry, and agriculture, are also commonly conducted in the U.S., China, and other nations.\u00a0 For example, the U.S. conducts an economic census of business activities every five years.\u00a0 The next economic census is scheduled to start in 2012 [10].\u00a0 The 2007 economic census covered 24 million businesses in the non-farm private economy, accounting for about 96% of the U.S. Gross Domestic Product [11].\u00a0 China conducted its last economic census in 2008 [12].\u00a0 Although each census may have different legal origins or motivations, the ultimate purpose is similar \u2013 to provide relevant, current, and reliable information for research, analysis, and ultimately decision- and policy-making.<\/p>

While the census has demonstrated its importance for many centuries, it has several well-known practical shortcomings.\u00a0 Most of all, human activities are continuous and dynamic over time, but a census can only provide a comprehensive snapshot on a designated census day or a defined period of time.\u00a0 Census results typically become outdated as soon as they are released.<\/p>

Dynamic human behavior and social, economic and political phenomena cannot be fully captured by a census taken at a single point in time. \u00a0The operation of a national census is typically so complex that multiple years are needed to design and collect data.\u00a0 More time is then spent to process, analyze, and report the results.\u00a0 The cost of a national census has become so prohibitively high that it is usually supplemented by smaller random surveys to provide more frequent results.<\/p>

After more than a decade of design, development, and testing, the American Community Survey (ACS) [13] began to implement \u201ccontinuous measurement\u201d on the characteristics of U.S. population and housing in 2005.\u00a0 About 3 million addresses are sampled per year (or 250,000 addresses per month) for a 5-year rolling cycle.\u00a0 The ACS produces estimates by aggregating data collected in the monthly surveys over a period of time so that they would be summarized annually based on the calendar year. \u00a0For local geographies with small populations, the ACS estimates may take up to five years to aggregate and become reportable [14].<\/p>

China reports its total population and structural changes in 2011 according to the National Sample Survey on Population Changes, which is described as a stratified, multi-stage, cluster, probability proportional to size sampling method.\u00a0 <\/span>Nearly 1.5 million persons in 31 provinces, 4,800 villages, 4,420 townships, and 2,133 counties were reportedly interviewed for the most recent update [15].<\/p>

Random sampling is a relatively new concept, introduced by the director of the Norwegian Statistics Bureau to the International Statistical Institute (ISI) in 1895 [16]. \u00a0<\/span>The international statistical community spent more than 30 years debating its merit before deciding that random sampling is an acceptable and sound scientific practice. \u00a0<\/span>During this period, theories and practices of today\u2019s mathematical statistics developed and grew to support the sampling approach.<\/p>

The first Department of Statistics in an arts and science colleges in the U.S. was established by the George Washington University in 1935 [17].\u00a0 <\/span>Academia would become the primary training ground for future statisticians.\u00a0 <\/span>According to the U.S. Census Bureau, statistical sampling methods were first used in its 1937 test survey of unemployment, partly in response to the need for more timely information about the scope of unemployment during the Great Depression [18].\u00a0 <\/span>Governments would become the primary employer of future statisticians.<\/p>

Supported by new theories and tested by applications in many fields, combined with the introduction of commercial computers in the 1950s and subsequent desktop computing, random surveys soon became the standard statistical practice to collect data and perform statistical analyses for making informed decisions. \u00a0<\/span>\u00a0<\/span>The foundation for today\u2019s statistical systems was built primarily from computing technologies developed in the 1970s, before the commercialization of the Internet ushered in the advanced information age in the 1990s.<\/p>

By the end of the 20th<\/sup> century, statistical systems including census and survey data were not only core governmental operations, but also the analytical foundation for market research, political predictions, agriculture and economic development planning, environmental management, public health, transportation planning, physical sciences, and other human and societal activities.\u00a0 <\/span>However, data must be collected according to statistical designs, including the application of probability principles, before they can be used for making statistical inference.\u00a0 <\/span>Large-scale statistical analyses were typically conducted by statisticians and subject-matter experts in either government or academia.<\/p>\n

21st\u00a0<\/sup>Century Information Needs and Trends<\/strong><\/h4>

The first decade of the 21st<\/sup>century was marked by the rapid conversion of data from analog to digital, as well as its quick acceptance and growth by the rapidly increasing Internet users, most of them are not statistical experts in academia or government.<\/p>

According to a study by the University of Southern California [19,20], digital storage surpassed non-digital for the first time in 2002, but at least 94 percent of all information on the planet was in digital form by 2007.<\/p>\n

\"Visualized:<\/a>
Visualized: One Zettabyte [21]<\/figcaption><\/figure>

The capacity to create and store digital data reportedly exceeded one zettabyte (1 ZB or 1021<\/sup> bytes) for the first time in 2010 [22,23], compared to about 0.29 ZB in 2007 and 0.00002 ZB in 1986 [19,20]. An industry executive declared that \u201c(e)very two days now we (human beings) create as much information as we did from the dawn of civilization up until 2003\u201d [24]. To illustrate the relative magnitude, the entire human genome [25] containing 3 billion chemical bases along the chromosomes of an individual can be captured in about 3 gigabytes (3 GB or 0.000000000003 ZB) of computer storage, relatively modest according to today\u2019s standards. In contrast, the Alpha Magnetic Spectrometer [26] records cosmic ray data at about 1GB per second.<\/p>

In practical terms, it means that paper records are becoming obsolete, the private sector is also generating large amounts of data, and billions of data consumers are not necessarily specialists.<\/p>

Complete sets of data are easily captured into electronic files for direct machine processing and computation without the need or consideration for sampling. The speed of this enormous change was concurrently matched by the spread of electronic data beyond political and geographical boundaries. Access to and use of information\u00a0technology is now pervasive if not common place in developed nations as well as in less developed nations.\u00a0 No matter where a computer is located in the world, it can be accessed as long as it is connected to the Internet.<\/p>

Big Data is a new and loosely defined term for large electronic datasets that may or may not be collected according to the structure and probability principles specified in the traditional statistical systems.\u00a0 <\/span>Administrative records, social media, barcode and radio frequency scanners, transportation sensors, energy and environmental monitors, online transactions, streaming videos, and satellite images have all contributed to the explosive growth of Big Data.\u00a0 <\/span>Most of these Big Data are not structured for conventional statistical analyses and inferences, nor are they simple or easy to use initially with current software and statistical systems.\u00a0\u00a0 <\/span>However, some contain important information that has not been available before for decision- and policy-making, especially when they are appropriately integrated into government data sources.<\/p>

The private sector has led the way in generating Big Data, integrating them with government statistics, and developing data mining techniques and methods to identify potential consumers, expand markets, test new products, and extract information for market and consumer research.\u00a0 <\/span>In some cases, they may even challenge traditional government functions. \u00a0<\/span>For example, certain search terms [27] in the social media may be good indicators of flu activity and can perform as well as the indicators produced by the public health agencies, if not actually better in terms of reduced lag time.\u00a0 <\/span><\/p>

Despite its diminished share in the ocean of available data, government statistics remain uniquely important in support of an increasingly global economy and expanding social needs of each nation.\u00a0 <\/span>However, in an era when search engines produce millions of results in seconds and international stock market data are reported in almost real-time around the clock, taking years and even months to collect, process, and release static results for limited coverage of geography, industry, or demographics is rapidly losing relevance.<\/p>

Most nations, even developed nations, are facing severe budgetary constraints.\u00a0 <\/span>The high costs and limited return with the current approach preclude the introduction of new censuses and surveys or the feasibility of any major expansion of the current census and sampling approach.\u00a0\u00a0 <\/span>Declining response rates worldwide compounds the problem.\u00a0 <\/span>For example, despite intense planning and efforts, the participation rate for the 2010 decennial census in the U.S. barely matched 74 percent achieved in 2000 [28].\u00a0 <\/span>Follow-up personal interviews would increase the average census cost to $56 per household [29], about 100 times the original mailing cost.<\/p>

In fact, the U.S. House of Representatives voted in May 2012 to terminate the American Community Survey, citing both confidentiality and budgetary concerns.\u00a0 <\/span>It is also uncertain now whether the 2012 economic census will be conducted in the U.S. as originally planned.<\/p>

The challenge to the national statistical agencies is real and daunting: the 20th<\/sup> century statistical systems can no longer adequately meet the needs of the 21st<\/sup>century.\u00a0 <\/span>Consumers of government statistics are rapidly increasing in number and breadth.\u00a0 <\/span>They require more comprehensive, dynamic, and timely data that can be accessed and understood easily, but the resources and time of development required by the existing methods are simply not available or affordable.\u00a0 <\/span>Governments are still expected to provide statistics that are accurate and reliable, while strictly protecting the confidentiality of the responding entities.<\/p>

Failing to meet these requirements, the Australian Bureau of Statistics is not sure that it \u201cwill remain at the heart of official information for societies\u201d [30]. \u00a0<\/span>The arrival of the Big Data era, along with the growing user requirements, is inevitable, and yet many governments and their statistical agencies are still unprepared in making the best use of Big Data.<\/p>

Fundamental change in the historical census and survey paradigm will be necessary to meet these challenges of the 21st<\/sup> century. \u00a0<\/span>Small evolutionary steps to tweak and tinker the edges of the current statistical systems built on knowledge and technologies grounded in the 1970s will simply not be adequate for the Big Data Revolution.<\/p>\n

Characteristics of the 21st<\/sup> Century Statistical Systems<\/b><\/h4>

The defining characteristics of the 21st<\/sup> century statistical systems will be the sophisticated application of massive longitudinal data, integration of multiple data sources, and rapid and simple delivery of results, while still strictly protecting confidentiality and data security and assuring accuracy and reliability.<\/p>

Longitudinal data refer to repeated observations of the same entity (such as a worker, a student, a household, a business, a school, or a hospital) over time. \u00a0<\/span>They provide unique measures of a baseline and change at the individual or business level.\u00a0 <\/span>These measures are not tracked and measured in the conventional cross-sectional studies, which collect data from multiple subjects at the same point in time.<\/p>

In particular, longitudinal administrative records are potential data sources for developing a comprehensive statistical system.\u00a0 <\/span>Statistics Canada defines administrative records simply as \u201cdata collected for the purpose of carrying out various non-statistical programs\u201d [31].\u00a0 <\/span>Other potential data sources include birth and death certificates, customs declarations, marriage and driver\u2019s licenses, individual and business taxes, unemployment insurance payments, social security, and medical prescriptions. \u00a0<\/span>There are a number of examples of massive amounts of longitudinal administrative records.\u00a0<\/span>\u00a0<\/span><\/span><\/p>\n