p. 1

blown to bits your life liberty and happiness after the digital explosion hal abelson ken ledeen harry lewis upper saddle river nj · boston · indianapolis · san francisco new york · toronto · montreal · london · munich · paris · madrid cape town · sydney · tokyo · singapore · mexico city

[close]

p. 2

many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks where those designations appear in this book and the publisher was aware of a trademark claim the designations have been printed with initial capital letters or in all capitals the authors and publisher have taken care in the preparation of this book but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions no liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein the publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales which may include electronic versions and/or custom covers and content particular to your business training goals marketing focus and branding interests for more information please contact u.s corporate and government sales 800 382-3419 corpsales@pearsontechgroup.com for sales outside the united states please contact international sales international@pearson.com visit us on the web www.informit.com/aw library of congress cataloging-in-publication data abelson harold blown to bits your life liberty and happiness after the digital explosion hal abelson ken ledeen harry lewis p cm isbn 0-13-713559-9 hardback alk paper 1 computers and civilization 2 information technology technological innovations 3 digital media i ledeen ken 1946 ii lewis harry r iii title qa76.9.c66a245 2008 303.48 33 dc22 2008005910 copyright © 2008 hal abelson ken ledeen and harry lewis this work is licensed under the creative commons attribution-noncommercial-share alike 3.0 united states license to view a copy of this license visit http creativecommons.org/licenses/by-nc-sa/3.0/us or send a letter to creative commons 171 second street suite 300 san francisco california 94105 usa for information regarding permissions write to pearson education inc rights and contracts department 501 boylston street suite 900 boston ma 02116 fax 617 671 3447

[close]

p. 3

isbn-13 978-0-13-713559-2 isbn-10 0-13-713559-9 text printed in the united states on recycled paper at rr donnelley in crawfordsville indiana third printing december 2008 this book is safari enabled the safari® enabled icon on the cover of your favorite technology book means the book is available through safari bookshelf when you buy this book you get free access to the online edition for 45 days safari bookshelf is an electronic reference library that lets you easily search thousands of technical books find code samples download chapters and access technical information whenever and wherever you need it to gain 45-day safari enabled access to this book · go to http www.informit.com/onlineedition · complete the brief registration form · enter the coupon code 9sd6-iqld-zdni-agec-ag6l if you have difficulty registering on safari bookshelf or accessing the online edition please e-mail customer-service@safaribooksonline.com editor in chief mark taub acquisitions editor greg doench development editor michael thurston managing editor gina kanouse senior project editor kristy hart copy editor water crest publishing inc indexer erika millen proofreader williams woods publishing services publishing coordinator michelle housley interior designer and composition nonie ratcliff cover designer chuti prasertsith

[close]

p. 4

chapter 1 digital explosion why is it happening and what is at stake on september 19 2007 while driving alone near seattle on her way to work tanya rider went off the road and crashed into a ravine for eight days she was trapped upside down in the wreckage of her car severely dehydrated and suffering from injuries to her leg and shoulder she nearly died of kidney failure fortunately rescuers ultimately found her she spent months recuperating in a medical facility happily she was able to go home for christmas tanya s story is not just about a woman an accident and a rescue it is a story about bits the zeroes and ones that make up all our cell phone conversations bank records and everything else that gets communicated or stored using modern electronics tanya was found because cell phone companies keep records of cell phone locations when you carry your cell phone it regularly sends out a digital ping a few bits conveying a here i am message your phone keeps pinging as long as it remains turned on nearby cell phone towers pick up the pings and send them on to your cellular service provider your cell phone company uses the pings to direct your incoming calls to the right cell phone towers tanya s cell phone company verizon still had a record of the last location of her cell phone even after the phone had gone dead that is how the police found her so why did it take more than a week if a woman disappears her husband can t just make the police find her by tracing her cell phone records she has a privacy right and maybe she has good reason to leave town without telling her husband where she is going in citations of facts and sources appear at the end of the book a page number and a phrase identify the passage 1

[close]

p. 5

2 blown to bits tanya s case her bank account showed some activity more bits after her disappearance and the police could not classify her as a missing person in fact that activity was by her husband through some misunderstanding the police thought he did not have access to the account only when the police suspected tanya s husband of involvement in her disappearance did they have legal access to the cell phone records had they continued to act on the true presumption that he was blameless tanya might never have been found new technologies interacted in an odd way with evolving standards of privacy telecommunications and criminal law the explosive combination almost cost tanya rider her life her story is dramatic but every day we encounter unexpected consequences of data flows that could not have happened a few years ago when you have finished reading this book you should see the world in a different way you should hear a story from a friend or on a newscast and say to yourself that s really a bits story even if no one mentions anything digital the movements of physical objects and the actions of flesh and blood human beings are only the surface to understand what is really going on you have to see the virtual world the eerie flow of bits steering the events of life this book is your guide to this new world the explosion of bits and everything else the world changed very suddenly almost everything is stored in a computer somewhere court records grocery purchases precious family photos pointless radio programs computers contain a lot of stuff that isn t useful today but somebody thinks might someday come in handy it is all being reduced to zeroes and ones bits the bits are stashed on disks of home computers and in the data centers of big corporations and government agencies the disks can hold so many bits that there is no need to pick and choose what gets remembered so much digital information misinformation data and garbage is being squirreled away that most of it will be seen only by computers never by human eyes and computers are getting better and better at extracting meaning from all those bits finding patterns that sometimes solve crimes and make useful suggestions and sometimes reveal things about us we did not expect others to know the march 2008 resignation of eliot spitzer as governor of new york is a bits story as well as a prostitution story under anti-money laundering aml rules banks must report transactions of more than $10,000 to federal regulators none of spitzer s alleged payments reached that threshold but his

[close]

p. 6

chapter 1 digital explosion 3 bank s computer found that transfers of smaller sums formed a suspicious pattern the aml rules exist to fight terrorism and organized crime but while the computer was monitoring small banking transactions in search of big-time crimes it exposed a simple payment for services rendered that brought down the governor once something is on a computer it can replicate and move around the world in a heartbeat making a million perfect copies takes but an instant copies of things we want everyone in the world to see and also copies of things that weren t meant to be copied at all the digital explosion is changing the world as much as printing once did and some of the changes are catching us unaware blowing to bits our assumptions about the way the world works when we observe the digital explosion at all it can seem benign amusing or even utopian instead of sending prints through the mail to grandma we put pictures of our children on a photo album web site such as flickr then not only can grandma see them so can grandma s friends and anyone else so what they are cute and harmless but suppose a tourist takes a vacation snapshot and you just happen to appear in the background at a restaurant where no one knew you were dining if the tourist uploads his photo the whole world could know where you were and when you were there data leaks credit card records are supposed to stay locked up in a data warehouse but escape into the hands of identity thieves and we sometimes give information away just because we get something back for doing so a company will give you free phone calls to anywhere in the world if you don t mind watching ads for the products its computers hear you talking about and those are merely things that are happening today the explosion and the social disruption it will create have barely begun we already live in a world in which there is enough memory just in digital cameras to store every word of every book in the library of congress a hundred times over so much email is being sent that it could transmit the full text of the library of congress in ten minutes digitized pictures and sounds take more space than words so emailing all the images movies and sounds might take a year but that is just today the explosive growth is still happening every year we can store more information move it more quickly and do far more ingenious things with it than we could the year before so much disk storage is being produced every year that it could be used to record a page of information every minute or two about you and every other human being on earth a remark made long ago can come back to haunt a political candidate and a letter jotted quickly can be a key discovery for a

[close]

p. 7

4 blown to bits biographer imagine what it would mean to record every word every human being speaks or writes in a lifetime the technological barrier to that has already been removed there is enough storage to remember it all should any social barrier stand in the way sometimes things seem to work both better and worse than they used to a public record is now very public before you get hired in nashville tennessee your employer can figure out if you were caught ten years ago taking an illegal left turn in lubbock texas the old notion of a sealed court record is mostly a fantasy in a world where any tidbit of information is duplicated cataloged and moved around endlessly with hundreds of tv and radio stations and millions of web sites americans love the variety of news sources but are still adjusting uncomfortably to the displacement of more authoritative sources in china the situation is reversed the technology creates greater government control of the information its citizens receive and better tools for monitoring their behavior this book is about how the digital explosion is changing everything it explains the technology itself why it creates so many surprises and why things often don t work the way we expect them to it is also about things the information explosion is destroying old assumptions about our privacy about our identity and about who is in control of our lives it s about how we got this way what we are losing and what remains that society still has a chance to put right the digital explosion is creating both opportunities and risks many of both will be gone in a decade settled one way or another governments corporations and other authorities are taking advantage of the chaos and most of us don t even see it happening yet we all have a stake in the outcome beyond the science the history the law and the politics this book is a wake-up call the forces shaping your future are digital and you need to understand them the koans of bits bits behave strangely they travel almost instantaneously and they take almost no space to store we have to use physical metaphors to make them understandable we liken them to dynamite exploding or water flowing we even use social metaphors for bits we talk about two computers agreeing on some bits and about people using burglary tools to steal bits getting the right metaphor is important but so is knowing the limitations of our metaphors an imperfect metaphor can mislead as much as an apt metaphor can illuminate.

[close]

p. 8

chapter 1 digital explosion 5 claude shannon claude shannon 1916­2001 is the undisputed founding figure of information and communication theory while working at bell telephone laboratories after the second world war he wrote the seminal paper a mathematical theory of communication which foreshadowed much of the subsequent development of digital technologies published in 1948 this paper gave birth to the now-universal realization that the bit is the natural unit of information and to the use of the term alcatel-lucent http:www.bell-labs.com/news/2001/february/26/shannon2_lg.jpeg we offer seven truths about bits we call them koans because they are paradoxes like the zen verbal puzzles that provoke meditation and enlightenment these koans are oversimplifications and over-generalizations they describe a world that is developing but hasn t yet fully emerged but even today they are truer than we often realize these themes will echo through our tales of the digital explosion koan 1 it s all just bits your computer successfully creates the illusion that it contains photographs letters songs and movies all it really contains is bits lots of them patterned in ways you can t see your computer was designed to store just bits all the files and folders and different kinds of data are illusions created by computer programmers when you send an email containing a photograph the computers that handle your message as it flows through the internet have no idea that what they are handling is part text and part graphic telephone calls are also just bits and that has helped create competition traditional phone companies cell phone companies cable tv companies and voice over ip voip service providers can just shuffle bits around to each other to complete calls the internet was designed to handle just bits not emails or attachments which are inventions of software engineers we couldn t live without those more intuitive concepts but they are artifices underneath it s all just bits this koan is more consequential than you might think consider the story of naral pro-choice america and verizon wireless naral wanted to form a

[close]

p. 9

6 blown to bits text messaging group to send alerts to its members verizon decided not to allow it citing the controversial or unsavory things the messages might contain text message alert groups for political candidates it would allow but not for political causes it deemed controversial had naral simply wanted telephone service or an 800 number verizon would have had no choice telephone companies were long ago declared common carriers like railroads phone companies are legally prohibited from picking and choosing customers from among those who want their services in the bits world there is no difference between a text message and a wireless phone call it s all just bits traveling through the air by radio waves but the law hasn t caught up to the technology it doesn t treat all bits the same and the common carriage rules for voice bits don t apply to text message bits verizon backed down in the case of naral but not on the principle a exclusive and rivalrous phone company can do whatever it thinks will maximize its profits in economists would say that bits deciding whose messages to distribunless controlled somehow tend to ute yet no sensible engineering disbe non-exclusive once a few peotinction can be drawn between text ple have them it is hard to keep messages phone calls and any other them from others and nonbits traveling through the digital airrivalrous when someone gets them waves from me i don t have any less in a letter he wrote about the nature of ideas thomas jefferson eloquently koan 2 perfection stated both properties if nature is normal has made any one thing less susceptible than all others of excluto err is human when books were sive property it is the action of the laboriously transcribed by hand in thinking power called an idea ancient scriptoria and medieval which an individual may exclumonasteries errors crept in with sively possess as long as he keeps every copy computers and networks it to himself but the moment it is work differently every copy is perdivulged it forces itself into the fect if you email a photograph to a possession of every one and the friend the friend won t receive a receiver cannot dispossess himself fuzzier version than the original the of it its peculiar character too is copy will be identical down to the that no one possesses the less level of details too small for the eye because every other possesses the to see whole of it computers do fail of course networks break down too if the

[close]

p. 10

chapter 1 digital explosion 7 power goes out nothing works at all so the statement that copies are normally perfect is only relatively true digital copies are perfect only to the extent that they can be communicated at all and yes it is possible in theory that a single bit of a big message will arrive incorrectly but networks don t just pass bits from one place to another they check to see if the bits seem to have been damaged in transit and correct them or retransmit them if they seem incorrect as a result of these error detection and correction mechanisms the odds of an actual error a character being wrong in an email for example are so low that we would be wiser to worry instead about a meteor hitting our computer improbable though precision meteor strikes may be the phenomenon of perfect copies has drastically changed the law a story told in chapter 6 balance toppled in the days when music was distributed on audio tape teenagers were not prosecuted for making copies of songs because the copies weren t as good as the originals and copies of copies would be even worse the reason that thousands of people are today receiving threats from the music and movie industries is that their copies are perfect not just as good as the original but identical to the original so that even the notion of original is meaningless the dislocations caused by file sharing are not over yet the buzzword of the day is intellectual property but bits are an odd kind of property once i release them everybody has them and if i give you my bits i don t have any fewer koan 3 there is want in the midst of plenty vast as world-wide data storage is today five years from now it will be ten times as large yet the information explosion means paradoxically the loss of information that is not online one of us recently saw a new doctor at a clinic he had been using for decades she showed him dense charts of his blood chemistry data transferred from his home medical device to the clinic s computer more data than any specialist could have had at her disposal five years ago the doctor then asked whether he had ever had a stress test and what the test had shown those records should be all there the patient explained in the medical file but it was in the paper file to which the doctor did not have access it wasn t in the computer s memory and the patient s memory was being used as a poor substitute the old data might as well not have existed at all since it wasn t digital even information that exists in digital form is useless if there are no devices to read it the rapid progress of storage engineering has meant that data stored on obsolete devices effectively ceases to exist in chapter 3 ghosts in the machine we shall see how a twentieth-century update of the

[close]

p. 11

8 blown to bits eleventh-century british domesday book was useless by the time it was only a sixtieth the age of the original or consider search the subject of chapter 4 needles in the haystack at first search engines such as google and yahoo were interesting conveniences which a few people used for special purposes the growth of the world wide web has put so much information online that search engines are for many people the first place to look for something before they look in books or ask friends in the process appearing prominently in search results has become a matter of life or death for businesses we may move on to purchase from a competitor if we can t find the site we wanted in the first page or two of results we may assume something didn t happen if we can t find it quickly in an online news source if it can t be found and found quickly it s just as though it doesn t exist at all koan 4 processing is power the speed of a computer is usually measured by the number of basic gordon moore founder of intel operations such as additions that corporation observed that the can be performed in one second the density of integrated circuits fastest computers available in the seemed to double every couple of early 1940s could perform about years this observation is referred five operations per second the to as moore s law of course it is fastest today can perform about a not a natural law like the law of trillion buyers of personal computgravity instead it is an empirical ers know that a machine that seems observation of the progress of fast today will seem slow in a year engineering and a challenge to or two engineers to continue their innovafor at least three decades the tion in 1965 moore predicted that increase in processor speeds was this exponential growth would exponential computers became continue for quite some time that twice as fast every couple of years it has continued for more than 40 these increases were one conseyears is one of the great marvels of quence of moore s law see sideengineering no other effort in hisbar tory has sustained anything like since 2001 processor speed has this growth rate not followed moore s law in fact processors have hardly grown faster at all but that doesn t mean that computers won t continue to get faster new chip designs include multiple processors on the same chip so the work can be split up and performed in parallel such design innovations promise to moore s law

[close]

p. 12

chapter 1 digital explosion 9 achieve the same effect as continued increases in raw processor speed and the same technology improvements that make computers faster also make them cheaper the rapid increase in processing power means that inventions move out of labs and into consumer goods very quickly robot vacuum cleaners and selfparking vehicles were possible in theory a decade ago but now they have become economically feasible tasks that today seem to require uniquely human skills are the subject of research projects in corporate or academic laboratories face recognition and voice recognition are poised to bring us new inventions such as telephones that know who is calling and surveillance cameras that don t need humans to watch them the power comes not just from the bits but from being able to do things with the bits koan 5 more of the same can be a whole new thing explosive growth is exponential growth doubling at a steady rate imagine earning 100 annual interest on your savings account in 10 years your money would have increased more than a thousandfold and in 20 years more than a millionfold a more reasonable interest rate of 5 will hit the same growth points just 14 times more slowly epidemics initially spread exponentially as each infected individual infects several others when something grows exponentially for a long time it may seem not to be changing at all if we don t watch it steadily it will seem as though something discontinuous and radical occurred while we weren t looking that is why epidemics at first go unnoticed no matter how catastrophic they may be when full-blown imagine one sick person infecting two healthy people and the next day each of those two infects two others and the next day after that each of those four infects two others and so on the number of newly infected each day grows from two to four to eight in a week 128 people come down with the disease in a single day and twice that number are now sick but in a population of ten million no one notices even after two weeks barely three people in a thousand are sick but after another week 40 of the population is sick and society collapses exponential growth is actually smooth and steady it just takes very little time to pass from unnoticeable change to highly visible exponential growth of anything can suddenly make the world look utterly different than it had been when that threshold is passed changes that are just quantitative can look qualitative another way of looking at the apparent abruptness of exponential growth its explosive force is to think about how little lead time we have to respond to it our hypothetical epidemic took three weeks to overwhelm the

[close]

p. 13

10 blown to bits population at what point was it only a half as devastating the answer is not a week and a half the answer is on the next to last day suppose it took a week to develop and administer a vaccine then noticing the epidemic after a week and a half would have left ample time to prevent the disaster but that would have required understanding that there was an epidemic when only 2,000 people out of ten million were infected the information story is full of examples of unperceived changes followed by dislocating explosions those with the foresight to notice the explosion just a little earlier than everyone else can reap huge benefits those who move a little too slowly may be overwhelmed by the time they try to respond take the case of digital photography in 1983 christmas shoppers could buy digital cameras to hook up to their ibm pc and apple ii home computers the potential was there for anyone to see it was not hidden in secret corporate laboratories but digital photography did not take off economically and practically it couldn t cameras were too bulky to put in your pocket and digital memories were too small to hold many images even 14 years later film photography was still a robust industry in early 1997 kodak stock hit a record price with a 22 increase in quarterly profit fueled by healthy film and paper sales and its motion picture film business according to a news report the company raised its dividend for the first time in eight years but by 2007 digital memories had become huge digital processors had become fast and compact and both were cheap as a result cameras had become little computers the company that was once synonymous with photography was a shadow of its former self kodak announced that its employee force would be cut to 30,000 barely a fifth the size it was during the good times of the late 1980s the move would cost the company more than $3 billion moore s law moved faster than kodak did in the rapidly changing world of bits it pays to notice even small changes and to do something about them koan 6 nothing goes away 2,000,000,000,000,000,000,000 that is the number of bits that were created and stored away in 2007 according to one industry estimate the capacity of disks has followed its own version of moore s law doubling every two or three years for the time being at least that makes it possible to save everything though recent projections suggest that by 2011 we may be producing more bits than we can store.

[close]

p. 14

chapter 1 digital explosion 11 in financial industries federal laws now require massive data retention to assist in audits and investigations of corruption in many other businesses economic competitiveness drives companies to save everything they collect and to seek out new data to retain wal-mart stores have tens of millions of transactions every day and every one of them is saved date time item store price who made the purchase and how credit debit cash or gift card such data is so valuable to planning the supply chain that stores will pay money to get more of it from their customers that is really what supermarket loyalty cards provide shoppers are supposed to think that the store is granting them a discount in appreciation for their steady business but actually the store is paying them for information about their buying patterns we might better think of a privacy tax we pay the regular price unless we want to keep information about our food alcohol and pharmaceutical purchases from the market to keep our habits to ourselves we pay extra the massive databases challenge our expectations about what will happen to the data about us take something as simple as a stay in a hotel when you check in you are given a keycard not a mechanical key because the keycards can be deactivated instantly there is no longer any great risk associated with losing your key as long as you report it missing quickly on the other hand the hotel now has a record accurate to the second of every time you entered your room used the gym or the business center or went in the back door after-hours the same database could identify every cocktail and steak you charged to the room which other rooms you phoned and when and the brands of tampons and laxatives you charged at the hotel s gift shop this data might be merged with billions like it analyzed and transferred to the parent company which owns restaurants and fitness centers as well as hotels it might also be lost or stolen or subpoenaed in a court case the ease of storing information has meant asking for more of it birth certificates used to include just the information about the child s and parents names birthplaces and birthdates plus the parents occupations now the electronic birth record includes how much the mother drank and smoked during her pregnancy whether she had genital herpes or a variety of other medical conditions and both parents social security numbers opportunities for research are plentiful and so are opportunities for mischief and catastrophic accidental data loss and the data will all be kept forever unless there are policies to get rid of it for the data will all be kept the time being at least the data sticks forever unless there are around and because databases are intenpolicies to get rid of it tionally duplicated backed up for security,

[close]

p. 15

12 blown to bits or shared while pursuing useful analyses it is far from certain that data can ever be permanently expunged even if we wish that to happen the internet consists of millions of interconnected computers once data gets out there is no getting it back victims of identity theft experience daily the distress of having to remove misinformation from the record it seems never to go away koan 7 bits move faster than thought the internet existed before there were personal computers it predates the fiber optic communication cables that now hold it together when it started around 1970 the arpanet as it was called was designed to connect a handful of university and military computers no one imagined a network connecting tens of millions of computers and shipping information around the world in the blink of an eye along with processing power and storage capacity networking has experienced its own exponential growth in number of computers interconnected and the rate at which data can be shipped over long distances from space to earth and from service providers into private homes the internet has caused drastic shifts in business practice customer service calls are outsourced to india today not just because labor costs are low there labor costs have always been low in india but international telephone calls used to be expensive calls about airline reservations and lingerie returns are answered in india today because it now takes almost no time and costs almost no money to send to india the bits representing your voice the same principle holds for professional services when you are x-rayed at your local hospital in iowa the radiologist who reads the x-ray may be half a world away the digital x-ray moves back and forth across the world faster than a physical x-ray could be moved between floors of the hospital when you place an order at a drive-through station at a fast food restaurant the person taking the order may be in another state she keys the order so it appears on a computer screen in the kitchen a few feet from your car and you are none the wiser such developments are causing massive changes to the global economy as industries figure out how to keep their workers in one place and ship their business as bits in the bits world in which messages flow instantaneously it sometimes seems that distance doesn t matter at all the consequences can be startling one of us while dean of an american college witnessed the shock of a father receiving condolences on his daughter s death the story was sad but familiar except that this version had a startling twist father and daughter were

[close]