First, the Chinese language and Japanese language are different grammatically so using Kanjii to write down Japanese created some problems.
example: for Yama(mountain) we fit the Chinese kanji "山” and for Ue(above) we use "上”.
In Chinese its ok to say 山上 but Japanese we have to say 山[no]上 so how to deal with [no]?
Simply put, there were no Kanji for parts of the Japanese language so certain Kanji were written smaller than the others to denote that they are to be used phonetically(used just for their sound like in the casse of [no]). This meant that the inherent Chinese meaning of the character was thrown away. these gradually simplified into the two forms we know was Katakana(derived from Chinese print-type script) and Hiragana(derived from Chinese cursive script).
How Katakana evolved from Chinese Kanji
How Hiragana evolved from a cursive form of Chinese Kanji
A complete paper is found on Wikipedia.