Have you ever heard of the 大和言葉 (Yamato Kotoba)? Some may consider it a language in itself; others would describe it as “Ancient Japanese”. It refers to the language spoken in Japan during the Yamato period between the years 250-710 AD. Around that time, Japan only consisted of a variety of tribes. However, for the first time, one tribe became exceptionally powerful and soon ruled the country. This was the Yamato tribe. In order to differentiate between the predominant ethnic group from minorities, the word 大和民族 (“Yamato Minzoku”, Yamato People) was used often, though today it is considered racist. Their language was the 大和言葉 or 和語 (Wago) , which still is part of the modern language in Japan.
Just like many European languages are widely influenced by Latin, the 大和言葉 was subject to many changes due to the Chinese influence of that time. When the Yamato period ended, the Chinese culture became very popular. In order to emulate them, Japan even adapted its writing system using their Kanji characters. Due to that fact, Japanese became more and more Chinese. Nevertheless, 大和言葉still represents an important part in today’s language, e. g. in terms of grammar.
Despite that, the English influence in Japanese (like probably everywhere in the world) is getting wider and wider. Maybe, in a few hundred years, there might be just as many English words in Japanese vocabulary as there are ancient Chinese ones nowadays.
To cut this long story short: Today’s Japanese a mixture of 和語 or 大和言葉, 漢語 (Kango), meaning words derived from ancient Chinese and even Western languages, 外来語 (Gairaigo).
Even though this new insight might not necessarily cheer you up when studying all those many different Kanji readings or getting confused over the various counting systems, at least now you know the reason for it.