There are two factors at play here.
The first thing to understand is that all aluminum has a thin clear layer of Aluminum Oxide on it, and when you scrape it away exposing pure aluminum it will very quickly reform, so it is almost always there.
Aluminum Oxide has a significant piezoelectric effect, strong enough that it is often used in science experiments specifically for this effect. So as you are scraping across the top of the metal you are breaking apart the bonds in the Aluminum oxide and as you do this it causes little jolts of voltage. You can even see this effect if you crush a sapphire in a dark room with a hammer, little sparks will form momentarily. A sapphire is just Aluminum Oxide more or less.
The piezoelectric effect I just described is likely the dominate effect.
There is a second effect that would be more than strong enough to show up as well, though I suspect less pronounced, that is what is called the Galvani potential of the two metals. When you have two dissimilar metals and they come in contact for a short time a small current flows and there is a difference in potential between them. In fact, the amount of voltage difference is determined not just by the metals but the temperature. This is exactly how thermocouples work, they are just two wires o different metals touching, we measure the voltage across them and this translates to a temperature.
Since the metal of your probe is not aluminum when it comes in contact with the aluminum a small voltage is produced. As you scrape over the rough surface the connection between your probe and the underlying aluminum isnt making great contact (remember there is a non-conductive oxide layer you are cutting though). On a microscopic level your probe is bouncing up and down making momentary contact with the aluminum as it scratches across very quickly. So this constant scratchiness between the two metals is likely causing the normal potential difference to vary and adding some noise to your plot as well.