Consider this: If we scan me, for example, and record the exact state, level and position of my every neurotransmitter, synapse, neural connection and other relevant details, and then reinstantiate this massive database into a neural computer, then who is the real me? If you ask the machine, it will vehemently claim to be the original Ray.
But there are strong arguments that this is really a different person. For one thing, old biological Ray (that's me) still exists. I'll still be here in my carbon, cell-based brain. Alas, I (the old biological Ray) will have to sit back and watch the new Ray succeed in endeavors that I could only dream of.
But New Ray will have some strong claims as well. He will say that while he is not absolutely identical to Old Ray, neither is the current version of Old Ray, since the particles making up my biological brain and body are constantly changing. It is the patterns of matter and energy that are semipermanent (that is, changing only gradually), while the actual material content changes constantly and very quickly.
One could scan my brain and reinstantiate the new Ray while I was sleeping, and I would not necessarily even know about it. If you then came to me, and said, "Good news, Ray, we've successfully reinstantiated your mind file so we won't be needing your old body and brain anymore," I may quickly realize the philosophical flaw in the argument that New Ray is a continuation of my consciousness. I may wish New Ray well, and realize that he shares my pattern, but I would nonetheless conclude that he is not me, because I'm still here.
What does this super technological intelligence mean for the future? There will certainly be grave dangers associated with 21st century technologies. Consider unrestrained nanobot replication. The technology requires billions or trillions of nanobots in order to be useful, and the most cost-effective way to reach such levels is through self-replication, essentially the same approach used in the biological world, by bacteria, for example. So in the same way that biological self-replication gone awry (i.e., cancer) results in biological destruction, a defect in the mechanism curtailing nanobot self-replication would endanger all physical entities, biological or otherwise.
Other salient questions are: Who is controlling the nanobots? Who else might the nanobots be talking to?
Organizations, including governments, extremist groups or even a clever individual, could put trillions of undetectable nanobots in the water or food supply of an entire population. These "spy" nanobots could then monitor, influence and even control our thoughts and actions. In addition, authorized nanobots could be influenced by software viruses and other hacking techniques. Just as technology poses dangers today, there will be a panoply of risks in the decades ahead.
Still, I regard the freeing of the human mind from its severe physical limitations as a necessary next step in evolution. Evolution, in my view, is the purpose of life, meaning that the purpose of life-and of our lives-is to evolve.
By the close of the next century, nonbiological intelligence will be ubiquitous. There will be few humans without some form of artificial intelligence, which is growing at a double exponential rate, whereas biological intelligence is basically at a standstill. Nonbiological thinking will be trillions of trillions of times more powerful than that of its biological progenitors, although it will be still of human origin.