In spite of the obvious syntactical assumptions herein, I would like you thoughts on recent debates surrounding what Ray Kurzweil and others call the coming Technological Singularity.
Recently I viewed a panel discussion where prominent members of the Artificial Intelligence research community debated the probability of creating an AI that would be "nice" to humans.
At the heart of the argument was the issue of unforeseen consequences. Eliezer Yudkowsky argued that through carefully stating the problems and through proper algorithms, we could be certain, dare I say, even able to predict, the outcome of the AI's actions, but not necessarily the actions themselves (i.e.: program an AI to make ice cream sundaes. Once it reaches a computational level trillions of times above that of human level intelligence, it may solve this problem--after all the ice cream runs out--by turning all matter in the universe into ice cream--again, carefully state your problem).

Hugo DeGaris, on the other hand, seemed to think that this approach, could be valid up to a certain threshold, beyond which (if it is truly "strong AI"--and not just, as in the
example above, a really adept ice cream making machine)the AI could theoretically rewrite it's own code (and my point is that if it can understand it's own blueprint <genetic engineering, anyone?>, realizing it's continuity depended on uninterrupted supply of energy, would likely "develop" a "survival instinct"--possibly to the exclusion of it's primary directives).

So, aside from philosophical debates over the nature of consciousness, Chinese room arguments or laws of accelerating returns...my question is this (and please bear with me--I am not a mathematician, so if the problem is not carefully stated, I hope you will get the gist of what I am saying)... and really, I think this is more or less a simple logic problem:

First, assume that we CAN create Artificial General Intelligence, in the conventional, popular definition.
Then, assume that it is built "bottom-up" using a hybrid approach, as proposed by Yudkowski, Goertzel et al, using a combination of (virtual) neural networks, evolutionary algorithms, Bayesian networks etc.
Is it safe to assume that, anthropomorphisms aside, our AI would be likely to rewrite its own code? In fact, isn't that kind of the whole point? Is it possible to prove, what the end result would be by, as Yudkowsky suggests, "looking at the source code?"
In my understanding, I see his point that, simply stated, 1+1=2--the old "garbage in, garbage out." I can see that, as with DNA, barring any mutation, it only does what it is programmed to do--I won’t grow gills suddenly.
However, since it will be an evolving and expanding AI, doesn’t it not (logically) stand to reason that it may eventually identify glaring omissions and limitations to its source code--and correct them? (by definition, it's at least tens of thousands times smarter, at this point, that the programmer that programmed it--and increasing in intelligence exponentially, according to the theory)

Growing gills would be a good corollary--for example, I could think of obvious military advantages to genetically engineering soldiers with switchable lungs/gills, or the ability to homogeneously make all essential vitamins--and I don't even have a PhD.

So, in the realm of logic, whose argument holds more weight?

Will the source code, properly designed and executed, always dictate the outcome--even in an evolving system?

My opinion, Piltdown men notwithstanding,
is that evolving systems by definition re-write their source code, hence the term
evolution.

If an ai becomes thousands of times smarter than humans, then it will conider humans inferior to it unless it is programmed properly. It cant alter its source code to escape usif altering its code in such a way isnt allowed. We would also have to program a series of fail safes. For example what if it found the best way to end human suffering was to end humanity altogether?

I can guarantee that if we create something more intelligent than us, it will take over. There really isn't any way to avoid it.

Unless, right from the start, it is forbidden to give an AI a body.
Any software or hardwired failsafe can be bypassed. We only have to look at computers to know how bad we are at creating them.