3 comments

  • noduerme 15 hours ago
    I love the idea, somehow, that these systems had to be built with domain knowledge integrated into the choices of layers and the order of algorithms, using rather brilliant discoveries for optimizing lines and edges, or whatever was relevant to the problem. That's an order of specificity that's lacking from our modern reliance on "one shot" solutions. Letting a one shot solver grind its brains against creating a custom solution to a specific set of transformations is kind if a closing of the circle when it comes to performance and resources for a certain task.

    I'm looking forward to the day we can have an optical device that runs only on light input, that looks like a piece of quartz, that you can place over a handwritten number (0-9) and on the top it displays the number in a clean font. Where the neural network to handle this can be translated into physical transformations once it's discovered, no inference processing power required.

    • andrewbutts 14 hours ago
      Here’s a 2024 project that built a denoiser out of passive optics! https://nips.cc/virtual/2024/poster/95165
    • somethingsome 6 hours ago
    • imtringued 11 hours ago
      How to defeat the bitter lesson 101: Use AI's domain knowledge as a substitute for human domain knowledge.

      The mistake was thinking that domain knowledge does not scale alongside the generalized scaling AI.

      • consensus1 11 hours ago
        Sounds like that is just another bitter lesson, not the refutation of it.
        • program_whiz 9 hours ago
          Or maybe the bitter lesson is a flawed over simplification. Computer vision is only possible because we used intuition about significant features in vision to make it tractable for GPUs with custom designed kernels. Tossing a generic model with more compute is pointless, it's the invariants and assumptions that make the models effective.

          Same for sequence modeling with attention, highly generic RNNs which can in theory model any sequence are just incapable, regardless of compute thrown at it. And a ton of engineered features like tokens are required to make LLMs work (just using the pure data is not scalable).

          Bitter lesson was only true back when the engineered features were hand made vs statistically derived, and then it stuck around making people think a bigger model is all we need to get to AGI, when the reality is the exact opposite.

          Technically an N-layer nnet can model "any function" but tossing a 10T plain nnet will not even approach a 10M param model with proper features, structure, and design. I call this the savory lesson (intuition and design matter)

  • agnosticmantis 16 hours ago
    Thanks for putting this together. Very nicely demystifies the weight-space programming phenomenon which I otherwise found quite magical and mysterious.

    It's not that the models magically "intuit" meaning from a blob of weights, but they methodically iterate on setting the weights with reasoning similar to a human engineer's reasoning.

    • consensus1 11 hours ago
      But how did it find the right neurons to iterate on in the massive multi terabyte weight space?
  • robrenaud 19 hours ago
    Would PPO learn quickly from the hand engineered solution? Is PPO getting a terrible init?

    It's interesting to see how tolerant of tedium the models are.

    • AshotSanamyan 14 hours ago
      Interesting question. My guess is it would learn faster at first, then drift away from the hand-built solution, unless you somehow hold it close for the first part of training. Would love to see someone try it.