• vrighter@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 year ago

    that is a studied, documented, surefire way to very quickly destroy your model. It just does not work that way. If you train an llm on the output of another llm (or itself) it will implode.

    • barsoap@lemm.ee
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      Also at best it’s an refinement, not a new sigmoid. So are new hardware/software designs for even faster dot products or advancements in network topology within the current framework. T3 networks would be a new sigmoid but so far all we know is why our stuff fundamentally doesn’t scale to the realm of AGI, and the wider industry (and even much of AI research going on in practice) absolutely doesn’t care as there’s still refinements to be had on the current sigmoid.