Software Developer – AI Coding Assistant Evaluation

Company Description

An enterprise client is seeking experienced software developers to participate in an AI training project focused on testing and evaluating AI coding assistants.

The project aims to improve AI coding tools by gathering feedback from developers as they work on their own software projects and compare different AI-generated solutions.

Job Description

We are looking for developers who have a software project they personally own, such as a side project, personal tool, or open-source library.

In this project, you will use an AI coding assistant to work on your existing software project. This may involve building new features, fixing bugs, refactoring code, or writing documentation. As you work, you will be presented with two versions of the AI assistant’s response and asked to select the one that works better, along with a short explanation of your choice.

What You’ll Do

• Bring and work on a software project that you personally own and maintain in a Git repository.
• Use an AI coding assistant to build features, fix bugs, refactor code, and improve documentation.
• Review and compare two versions of AI-generated coding responses.
• Evaluate which response is more useful, accurate, and effective for your project.
• Provide clear, honest feedback explaining why you selected one response over the other.
• Work comfortably with terminal commands and Git during project sessions.
• Follow project guidelines and ensure that no passwords, API keys, or other people’s personal information are included in your sessions.

Qualifications

Requirements

• Experienced software developer with a personal software project that you own and maintain in a Git repository.
• Comfortable working with terminal commands and Git.
• Has a computer running macOS, Linux, or Windows with WSL2.
• Able to review and evaluate AI-generated code and coding suggestions.
• Strong written communication skills, with the ability to explain decisions clearly and briefly.
• Able to provide independent and honest feedback without using AI assistance to write evaluation responses.

Nice to Haves

• Experience maintaining open-source software projects.
• Experience building personal tools, applications, libraries, or other software projects.
• Familiarity with AI coding assistants and developer productivity tools.
• Experience with software debugging, feature development, code refactoring, and documentation.

Additional Information

• Fully remote freelance opportunity.
• Worldwide applicants are welcome, subject to project eligibility.
• Rate: Up to $100 per hour.
• Expected Workload: Approximately 10–39 hours per week, depending on project needs.
• Work volume may fluctuate throughout the contract.
• Payments will be released weekly as a lump sum for work completed during the previous week.
• AI usage during the project will be covered, so candidates will not need to pay for an AI subscription or API usage.

Project Eligibility and Ownership Requirements

• Candidates must bring a software project they personally and solely own.
• Open-source projects must have an approved license. Projects with multiple maintainers may require additional confirmation.
• Projects owned by an employer, client, university, or startup are not eligible.
• Projects covered by an NDA are not eligible.
• Projects involving AI model training or evaluation, machine learning infrastructure, security exploit tooling, or the processing of other people’s personal data are not eligible.
• The client’s team must review and approve the project before work begins.
• Selected candidates must sign contributor terms before starting.

Important Privacy and Participation Notes

• Transcripts from project sessions will be used to improve AI models. Once submitted, these transcripts cannot be withdrawn.
• Only project session transcripts will be shared, not the rest of the candidate’s software project.
• Ratings and written feedback provided by candidates will become part of the project’s deliverables.
• Candidates must not include passwords, API keys, or other people’s personal information in their sessions.
• All ratings and written evaluations must be completed independently, without AI assistance.

Important: An active Upwork contract is required before candidates can receive onboarding instructions, access to the evaluation platform, or payment for the project. Candidates should accept the contract offer once received.

Due to the high volume of applicants being contacted, the team may not be able to respond to every individual message. Thank you for your understanding.