The argument goes that in the EU we are swamped by lots of nasty foreign types. Under Brexit we can let in just who we want to let in.
In the EU we maintain full control over non-EU immigration and, even, entry. We can demand visas, we can prevent people visiting, we are free to set our own policy on residence, working permits and citizenship. There are no restrictions that a Brexit will lift.
As for EU citizens, at present, they all have a right to travel to the UK, work in the UK, live in the UK, claim benefits (with some restrictions) in the UK and, for most purposes act as if they were English!
A Brexit, though, is not sufficient to end this terrible state of affairs.
A full trade deal with the EU post Brexit would involve accepting freedom of movement. Norway, not in the EU, has accepted just that. You have a right to travel to Norway, work in Norway, live in Norway, claim benefits (with some restrictions) in Norway and, for most purposes act as if you are English! Not entering trade deal would put Britain’s ability to restrict EU citizens on the same basis as Britain’s ability to restrict non EY citizens. Not entering a trade deal would also reduce Britain’s income considerably.
In between there are a number of different trade offs between giving up the means to live and having to put up with icky foreigners. You may ask yourself how much of your income you are prepared to forgo to rid yourself of Vlad the Plasterer (a real person by the way. His name’s Vlad, he’s a plasterer and he drives round south London in a van with “Vlad the Plasterer” on the side).
But how much you, personally, would give up isn’t quite as important as how much most people would be prepared to give up. The answer is likely “not very much” and Britain would accept full freedom of movement to get that trade deal it needed to avoid an even sharper fall in national income.
A Brexit would, most likely, have no effect at all on immigration.