Oh, what would this mean if the people they're forgetting had a major impact on their lives? Gaignun and Albedo were both very major figures in Rubedo's history, so his life would have gone very differently without them. Do his memories change to account for the missing information or will he just be left going 'hey, does anyone else find it weird that I ran the Kukai Foundation looking like a kid? And why was I Gaignun Kukai Jr. if there was no Gaignun Kukai?'

Not to mention the gaps it would leave for Albedo.