Science & validation

Measure quality. Understand cost. Choose on evidence.

A generative system should be assessed against the task it needs to solve, under its intended conditions of use. With GAIA©, we compare models and configurations to identify those that meet agreed quality, consistency and response-time criteria.

Science & validationBack to Algorithm G

Define

We establish the task, available information, boundaries and acceptance criteria. GAIA© software structures these requirements into a Genesis Prompt that provides the basis for system design.

Design

We build an architecture suited to the task. It can combine system instructions, retrieval, web calls, model selection and post-processing.

Validate

We compare configurations through automated tests and statistical analysis, with human review where needed. We assess performance using representative cases.

Measure value

We calculate the cost per task meeting the acceptance criteria, including failed attempts and retries. A lower-cost configuration is selected only if it also meets the defined requirements.

Research supported by high-performance computing.

Through the Centro Nacional de Computação Avançada (Portugal’s National Centre for Advanced Computing), our research and computational evaluation use MareNostrum 5 at the Barcelona Supercomputing Center. This infrastructure supports the exploration of alternatives and the study of trade-offs between quality, speed and cost.

Research infrastructure supports evaluation; it does not mean that every deployed solution runs on a supercomputer.

FCT
Centro Nacional de Computação Avançada
Barcelona Supercomputing Center

Cost per accepted task

Cost of evaluated executions ÷ tasks meeting the acceptance criteria

Comparisons state the cost scope, test conditions, reference configuration and pricing date.

Software & research references

GAIA© software has copyright registrations in the EU and the United States. These registrations concern software protection; performance is assessed through testing.

EU registration
3079/D/24
U.S. registration
TXu 2-557-783
Read the working paper on SSRN

What would you like to move forward?

Tell us about the task, the audience and what a useful result would look like.

Discuss an application