Music has its own internal logic. It is like the logic of a dream, clear in its own terms but not necessarily in everyday terms. Sometimes it expresses something you can describe in words, but not always.
If you remove tone from music, you have something, but what? I find it difficult to call it music.