I don't think so.
Swiss Gruyère (not the French plastic-tasting thing they call Gruyère) has no holes, BTW - as most Swiss cheeses.
Swiss Emmental has.
In French on says "J'ai un trou de mémoire" = "I have a memory hole".
And "On ne va pas en faire un fromage" = "We won't make a cheese with it " = "We won't make a fuss about it".
But no direct link between Swiss cheese and memory.
Now, if some day we meet, I will cook you a true Swiss fondue, with delicious Gruyère and Vacherin, and I'll offer some good white wine with it.