The Collapse of GPT – Communications of the ACM
cacm.acm.org
  • toastmeister@lemmy.caEnglish
    2·
    13 hours ago

    To make this more precise, we say that original data follows a normal distribution {isplaystyle X^{0}im {athcal {N}}(u ,igma ^{2})}, and we possess {isplaystyle M{0}} samples {isplaystyle X{j}^{0}} for {isplaystyle jn {{,1,ots ,M{0},{}}}}. Denoting a general sample {isplaystyle X{j}^{i}} as sample {isplaystyle jn {{,1,ots ,M{i},{}}}} at generation {isplaystyle i}, then the next generation model is estimated using the sample mean and variance:

    {isplaystyle u {i+1}={rac {1}{M{i}}}um {j}X{j}^{i};uad igma {i+1}^{2}={rac {1}{M{i}-1}}um {j}(X{j}^{i}-u {i+1})^{2}.}

    Leading to a conditionally normal next generation model isplaystyle X{j}^{i+1}u {i+1,;igma {i+1}im {athcal {N}}(u {i+1},igma {i+1}^{2})}. In theory, this is enough to calculate the full distribution of {isplaystyle X{j}^{i}}. However, even after the first generation, the full distribution is no longer normal: It follows a variance-gamma distribution.