A test of AI for Australia's corporate regulator found that the technology might actually make more work for people, not less.

cross-posted from: https://lemmy.world/post/19416727

Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

  • don@lemm.ee
    81·
    3 months ago

    Are you deliberately ignoring the point article? If AI is worse at summarizing than a human, then it hasn’t gotten to the point of summarizing better than a human. It’s gotten to the point of being able to fail at it worse than humans. It will have passed summarizing when it’s at least on par with the average human.

    I have seen many videos of robots trying to walk and often failing, while the average human baby consistently learned to do it faster than the robots, who still failed. Your position seems to be “Any progress AI programmers make means AI’s gonna overtake us really soon!”

    There’s such a thing as negative progress, and if these are your best examples of progress, then I don’t think you’re capable of giving an effective explanation of the concept to begin with.

    • Linktank
      111·
      3 months ago

      I don’t need to convince you of anything. I also don’t really care if you think it will happen soon or at all.

      I’m impressed with the progress so far and I believe that models will become available that will be able to do a far better job than an average human in a relatively short period of time. That being the next decade or two.

      Your defeatist whiney argumentative comments aside.