Hands-On with Amazon Q Developer in GitHub: Building an AWS SAM App from Scratch

Introduction In mid-April, I blogged about leveraging cloud and generative AI to maximize developer productivity. Among other areas, I highlighted the potential of AI in the PR Review/Gitflow process, suggesting tools like Evolua.io or CodeRabbit to offload routine code reviews and low-value fixes to AI assistants. Today, I'm excited to explore Amazon's latest offering in this space: Amazon Q Developer in GitHub, which has just been released in preview. As an AWS Community Builder, I've had early access to this tool for over a week, and I wanted to put it through its paces with a real-world test: Can Amazon Q Developer build a complete AWS SAM application from scratch, based solely on requirements provided in GitHub issues? How Amazon Q Developer for GitHub Works Before diving into my experiment, let's quickly review how Amazon Q Developer integrates with GitHub: Issue-based development: Create a GitHub issue describing your feature requirements, add the "Amazon Q development agent" label, and Q Developer will generate a complete implementation and submit it as a pull request. PR-based collaboration: Add the Q Developer label to an existing PR, and it will review your code, suggest improvements, or address requested changes. In both cases, Amazon Q analyzes the context of your repository and comments, then either generates new code or modifies existing code. The AWS blog post explains the current capabilities and limitations of this preview version in detail. My Experiment: Building an AWS SAM App from a Blank Repository To test Amazon Q Developer's capabilities, I created a blank repository with just a README file. My goal was to see if Amazon Q could build a serverless photo application with AWS SAM, complete with GitHub Actions pipeline, solely based on requirements provided in issues. Lesson #1: Crafting Effective Prompts is Critical Following best practices for working with generative AI, I spent time crafting a detailed issue description that specified all the requirements for my AWS SAM application. This included: Application architecture details Required AWS resources Code structure Testing requirements GitHub Actions pipeline specifications I submitted this as Issue #1, added the Amazon Q development agent label, and waited. Lesson #2: Mind the Context Window Within minutes, I received my first lesson in the limitations of Amazon Q Developer: ⚠️ The issue description is too long. I will only look at the first 1000 words. Like most LLM-based tools, Amazon Q Developer has a context window limitation. My detailed prompt exceeded this limit, meaning it would only process part of my requirements. Lesson #3: Concise Requirements Are More Effective Taking this feedback, I used an LLM to help condense my requirements into a more focused version. I submitted this as Issue #2, focusing on just the essential specifications: Core application functionality (photo upload/download API) Required AWS resources (S3, DynamoDB, Lambda, API Gateway) Basic structure and pipeline requirements The tool acknowledged receipt and began working on the implementation: ⏳ I'm working on generating code for this issue. I'll update this issue with a comment and open a pull request when I'm done. However, after waiting for a while, both Issue #1 and Issue #2 experienced minor failures. Despite these setbacks, I'm not giving up and plan to keep trying with different approaches. Challenges with the Preview Version These minor failures aren't entirely surprising for a preview service. They could be due to several factors: Service capacity: Since Amazon Q Developer is currently free to use during preview, AWS may be experiencing high demand leading to temporary resource constraints. Task complexity: Building a complete AWS SAM application with CI/CD pipeline from scratch is quite ambitious for an AI assistant, especially one in preview. Preview limitations: As with any preview service, there are likely still bugs being worked out and capabilities being refined. Key Lessons from My Initial Testing While my experiment hit some snags, I've already uncovered several valuable insights: Master the art of concise prompting - Your issue descriptions must be detailed enough to guide Q Developer effectively while staying under 1000 words. This constraint forces you to prioritize the most critical requirements. Start with focused, smaller tasks - Rather than requesting a complete application, begin with individual components or features to maximize success rate and learn the tool's capabilities. Embrace an iterative workflow - Amazon Q Developer shines as a collaborative partner in an iterative development process, not as a one-shot solution. Be prepared to refine requirements and review generated code carefully. Amazon Q Developer: Production-Ready or Preview Promise? Despite being in preview, Amazon Q Developer

May 5, 2025 - 23:42

Hands-On with Amazon Q Developer in GitHub: Building an AWS SAM App from Scratch

Introduction

In mid-April, I blogged about leveraging cloud and generative AI to maximize developer productivity. Among other areas, I highlighted the potential of AI in the PR Review/Gitflow process, suggesting tools like Evolua.io or CodeRabbit to offload routine code reviews and low-value fixes to AI assistants.

Today, I'm excited to explore Amazon's latest offering in this space: Amazon Q Developer in GitHub, which has just been released in preview. As an AWS Community Builder, I've had early access to this tool for over a week, and I wanted to put it through its paces with a real-world test:

Can Amazon Q Developer build a complete AWS SAM application from scratch, based solely on requirements provided in GitHub issues?

How Amazon Q Developer for GitHub Works

Before diving into my experiment, let's quickly review how Amazon Q Developer integrates with GitHub:

Issue-based development: Create a GitHub issue describing your feature requirements, add the "Amazon Q development agent" label, and Q Developer will generate a complete implementation and submit it as a pull request.
PR-based collaboration: Add the Q Developer label to an existing PR, and it will review your code, suggest improvements, or address requested changes.

In both cases, Amazon Q analyzes the context of your repository and comments, then either generates new code or modifies existing code. The AWS blog post explains the current capabilities and limitations of this preview version in detail.

My Experiment: Building an AWS SAM App from a Blank Repository

To test Amazon Q Developer's capabilities, I created a blank repository with just a README file. My goal was to see if Amazon Q could build a serverless photo application with AWS SAM, complete with GitHub Actions pipeline, solely based on requirements provided in issues.

Lesson #1: Crafting Effective Prompts is Critical

Following best practices for working with generative AI, I spent time crafting a detailed issue description that specified all the requirements for my AWS SAM application. This included:

Application architecture details
Required AWS resources
Code structure
Testing requirements
GitHub Actions pipeline specifications

I submitted this as Issue #1, added the Amazon Q development agent label, and waited.

Lesson #2: Mind the Context Window

Within minutes, I received my first lesson in the limitations of Amazon Q Developer:

⚠️ The issue description is too long. I will only look at the first 1000 words.

Like most LLM-based tools, Amazon Q Developer has a context window limitation. My detailed prompt exceeded this limit, meaning it would only process part of my requirements.

Lesson #3: Concise Requirements Are More Effective

Taking this feedback, I used an LLM to help condense my requirements into a more focused version. I submitted this as Issue #2, focusing on just the essential specifications:

Core application functionality (photo upload/download API)
Required AWS resources (S3, DynamoDB, Lambda, API Gateway)
Basic structure and pipeline requirements

The tool acknowledged receipt and began working on the implementation:

⏳ I'm working on generating code for this issue. I'll update this issue with a comment and open a pull request when I'm done.

However, after waiting for a while, both Issue #1 and Issue #2 experienced minor failures. Despite these setbacks, I'm not giving up and plan to keep trying with different approaches.

Challenges with the Preview Version

These minor failures aren't entirely surprising for a preview service. They could be due to several factors:

Service capacity: Since Amazon Q Developer is currently free to use during preview, AWS may be experiencing high demand leading to temporary resource constraints.
Task complexity: Building a complete AWS SAM application with CI/CD pipeline from scratch is quite ambitious for an AI assistant, especially one in preview.
Preview limitations: As with any preview service, there are likely still bugs being worked out and capabilities being refined.

Key Lessons from My Initial Testing

While my experiment hit some snags, I've already uncovered several valuable insights:

Master the art of concise prompting - Your issue descriptions must be detailed enough to guide Q Developer effectively while staying under 1000 words. This constraint forces you to prioritize the most critical requirements.
Start with focused, smaller tasks - Rather than requesting a complete application, begin with individual components or features to maximize success rate and learn the tool's capabilities.
Embrace an iterative workflow - Amazon Q Developer shines as a collaborative partner in an iterative development process, not as a one-shot solution. Be prepared to refine requirements and review generated code carefully.

Amazon Q Developer: Production-Ready or Preview Promise?

Despite being in preview, Amazon Q Developer shows remarkable potential for transforming development workflows. The GitHub integration feels natural, and the ability to generate code directly from issue descriptions could dramatically accelerate development cycles for teams willing to adapt their processes.

That said, be prepared for the expected preview-stage limitations. Beyond the context window restriction, you might encounter occasional bugs or inconsistent results. The service may also experience capacity constraints since it's currently free to use—AWS might be throttling resources during peak times, leading to slower responses or temporary unavailability.

I Strongly Recommend Trying It Today

I enthusiastically encourage you to experiment with Amazon Q Developer in your GitHub repositories. It costs nothing during the preview period and requires no AWS account setup, making it the perfect time to explore its capabilities and limitations.