Comparing AI Models for Legal Information Retrieval

AI-task

The AI task examined is focused on improving the way consumers find legal information on webshops. The goal is to develop a prompt that simplifies finding essential legal details, such as terms and conditions, privacy policies, and refund conditions. By tackling challenges like scattered content, complex navigation, and poor website design, the AI prompt aims to create a more intuitive and efficient experience for users, ensuring users can easily locate their rights and responsibilities, thus enhancing trust.

Used prompt in research »

Methodology

We evaluated four common AI models - GPT-4o mini, GPT-4, Copilot, and Gemini - and evaluated their performance in retrieving legal information from various websites with our advanced prompt. The evaluation was conducted using a structured approach, focusing on nine criteria. For Windows Copilot, we use the more precise style option.

Results

1. Specificity and Detail
   • GPT-4o mini: Provided detailed steps and direct links to various policies, balancing detailed information with general guidance.
   • GPT-4: Emphasised updates and compliance, providing detailed descriptions of policies.
   • Copilot: Included specific update dates and detailed descriptions of content.
   • Gemini: Provided practical advice on using search functions and checking for updates.
2. Update and Relevance
   • GPT-4o mini: Advised checking the last update date, providing current and relevant information.
   • GPT-4: Emphasised the importance of periodic updates, providing highly relevant information.
   • Copilot: Provided specific update dates, offering detailed and current information.
   • Gemini: Suggested checking for updates, offering practical steps and recommendations for non-native speakers.
3. User Experience and Navigation
   • GPT-4o mini: Clear and detailed steps for accessing legal information with high usability.
   • GPT-4: Concise and clear instructions, making steps easy to follow.
   • Copilot: Detailed steps with specific content descriptions, ensuring high usability.
   • Gemini: Practical and clear steps with additional tips, enhancing usability.
4. Accuracy and Completeness
   • GPT-4o mini: Accurate links and comprehensive coverage of all relevant legal information.
   • GPT-4: Accurate and detailed steps with precise links, emphasising regular updates.
   • Copilot: Accurate links and specific update dates, providing detailed descriptions.
   • Gemini: Accurate steps and methods are suggested, offering general coverage with practical tips.
5. Legal and Language Considerations
   • GPT-4o mini: Consistently accurate and appropriate legal language with clear terminology.
   • GPT-4: Clear and precise legal language with a focus on compliance and periodic updates.
   • Copilot: Detailed and accurate legal language across all websites.
   • Gemini: Emphasises reviewing terms in the user's native language, recommending translation tools.
6. Real-world Application
   • GPT-4o mini: Practical steps leading directly to legal information, with accurate and useful direct links.
   • GPT-4: Concise, practical steps leading directly to legal information, with emphasis on regular updates.
   • Copilot: Detailed steps and direct links, adding practical value with content descriptions.
   • Gemini: Practical steps and direct links, emphasising search functions and understanding terms in the native language.
7. Language and Presentation Quality
   • GPT-4o mini: Clear and straightforward language, professionally presented with structured headings and bullet points.
   • GPT-4: Concise and easy-to-understand language, professionally presented with direct links.
   • Copilot: Clear, detailed language with specific content descriptions, maintaining a formal and user-friendly tone.
   • Gemini: Simple, clear language, professionally presented with practical tips.
8. Cross-Platform Consistency
   • GPT-4o mini: High level of detail and accuracy consistently across all websites.
   • GPT-4: Consistently detailed and accurate, with clear steps and emphasis on updates.
   • Copilot: High consistency in providing detailed and accurate information.
   • Gemini: Maintains consistency in providing practical steps and relevant information.
9. Support and Contact Information
   • GPT-4o mini: It is generally suggested to contact customer support for further assistance.
   • GPT-4: Suggested looking for contact details on the website and emphasised customer support.
   • Copilot: It is recommended to contact the respective websites directly for any questions or clarification.
   • Gemini: Suggested using contact forms or e-mail addresses available on the websites, emphasising practical steps to find further details.

Overall Conclusions

• Best Overall Performance: GPT-4o mini and GPT-4
  • Both models provide comprehensive, accurate, and detailed guidance, with a strong focus on clarity, usability, and regular updates.
• Most Detailed and Practical Guidance: Copilot
  • Copilot excels in providing specific update dates, detailed descriptions of content, and practical steps, making it highly user-friendly and practical.
• Best for Multilingual and Practical Tips: Gemini
  • Gemini offers practical steps and emphasises understanding terms in the user's native language, making it particularly useful for non-native speakers.

Content Optimisation

1. Update Dates: GPT-4o mini and Gemini should include specific update dates to enhance the timeliness and relevance of the information.
2. Contact Information: All models should consistently provide specific contact information for further assistance to improve user support.
3. Balance of Detail and Practicality: Gemini could benefit from adding more detailed content descriptions, while Copilot could include more general practical tips.

Select Your Model and Find Your Legal Information

As AI models continue to evolve, it is essential to leverage their capabilities to access accurate and comprehensive legal information. We encourage users to try these models and provide feedback to help improve their performance. By doing so, we can collectively enhance the tools available for legal research, making this information more accessible and reliable for everyone.