I have spent two days trying to find a solution to this exact same problem. I have seen some overly complex solutions that probably work but trying to figure out the explanations is like trying to understand someone describe drag coefficients of rotary wing aircraft. While I am sure they work, they are way above my comprehension. I wanted an intuitive and straight forward solution for something that shouldn't require some arcane technique to fix.
What I did:
1) I used the Clone Stamp tool to paint over the spot on the wall I wanted to remove.
2) picked a more easily discernible point of reference for Tracking. In my case, I used a poster on the wall.
3) As per the instructions here, https://www.video2brain.com/en/lessons/removing-an-object-with-clone-stamp, I pasted the Attach Point data from the Tracker into Clone -> Transform - > Position
However, that didn't work for me as the object I was painting was not the object I was tracking. The solution from the video was simply moving my cloned spot to the tracked object instead of leaving it where I wanted it. So...
4) I then manually adjusted the Anchor Point of the Clone back to the original spot I wanted to cover.
I played footage and the cloned region successfully moved with the spot I was covering.
This is what I was able to figure out. I'm not an advanced user. I only use After Effects every couple of months when I need to fix minor things in footage, so I've only learned to use what techniques I need for the job at hand. No doubt, experienced users may shake their heads at my simplistic fix. I'm sure my technique would be wholly inadequate for situations far more complex than fixing spots on a gray wall, which is why my eyeballs cross when reading suggestions that seem way more involved than one would expect for something as seemingly minor as this. It's just a spot! It should be super easy to remove, right? Well, that's what I was looking for and that's what I came up with. I hope it helps.