I Built aictl to Review Code and Articles with Generative AI, and Had It Review My Blog Posts

Having an editor who reviews your PRs


Posted on Tue, Oct 1, 2024
Tags generative-ai, githubactions
generative-ai, githubactions
📝 This article is a translation of the original Japanese post. View original

Automating Blog Post Reviews with Generative AI

I’ve been writing blog posts for about 8 years now, but since I write them on my own, nobody reviews them. So I developed go-zen-chu/aictl to have articles reviewed via GitHub Actions when I open a PR.

There are products and tools like Copilot that do code review with generative AI, but there don’t seem to be many tools for having blog posts reviewed, and it looked easy to build using the OpenAI client, so I made one — also as a learning exercise.

How to Set It Up

The setup is described in the README of go-zen-chu/aictl, but in short:

  1. Place the yaml below at .github/workflows/check-pr.yml
  2. Set AICTL_OPENAI_API_KEY in the repository secrets
    1. For the API key, see API Reference - OpenAI API

and setup is done (secrets.GITHUB_TOKEN appears in the workflow, but it exists by default when GitHub Actions runs, so no configuration is needed).

 1name: check-pr
 2on:
 3  pull_request_target:
 4    types: [opened, synchronize]
 5jobs:
 6  aictl-review:
 7    runs-on: ubuntu-latest
 8    permissions:
 9      pull-requests: write
10    steps:
11      - name: Get number of commits for fetch depth
12        run: echo "fetch_depth=$(( commits + 1 ))" >> $GITHUB_ENV
13        env:
14          commits: ${{ github.event.pull_request.commits }}
15      - name: Git checkout until fetch-depth
16        uses: actions/checkout@v4
17        with:
18          fetch-depth: ${{ env.fetch_depth }}
19          # need to specify ref to fetch PR head, otherwise you get main branch with no HEAD
20          ref: ${{ github.event.pull_request.head.sha }}
21      - name: Fetch base branch as origin
22        run: git fetch origin ${{ github.event.pull_request.base.ref }}
23      - name: Get the diff between PR and origin branch and get commit message too
24        id: git-diff
25        run: |
26          # If fetch-depth is not specified in checkout, HEAD cannot be found
27          # Make sure to filter out deleted file in `git diff` since you cannot review deleted files
28          diff_files=$(git diff --name-only --diff-filter=ACMR origin/${{ github.event.pull_request.base.ref }}..HEAD | tr '\n' ',' | sed 's/,$/\n/')
29          echo "diff_files: ${diff_files}"
30          echo "diff-files=${diff_files}" >> "$GITHUB_OUTPUT"
31
32          # In PR, you cannot get `github.event.head_commit.message` so you need to get commit message via git log
33          commit_msg=$(git log --format=%B -n 1 ${{ github.event.pull_request.head.sha }})
34          echo "commit_msg: ${commit_msg}"
35          echo "commit-msg=${commit_msg}" >> "$GITHUB_OUTPUT" 
36      - name: Run aictl for code review
37        id: aictl-review
38        uses: go-zen-chu/aictl@main
39        if: ${{ steps.git-diff.outputs.diff-files != '' && !contains(steps.git-diff.outputs.commit-msg, '[skip ai]') }}
40        with:
41          query: |
42            The Markdown files contain technical articles about programming, life hack articles, book reviews, and so on.
43            Please review them from the perspective of making them easier to read and reaching more readers.
44
45            The other files are the source code of the blog system itself.
46            Please review them from the perspective of code quality and maintainability so that a better blog system can be built.
47          language: 日本語
48          text-files: ${{ steps.git-diff.outputs.diff-files }}
49        env:
50          AICTL_OPENAI_API_KEY: ${{ secrets.AICTL_OPENAI_API_KEY }}
51      - name: Post aictl review result to PR comment
52        if: steps.aictl-review.outcome != 'skipped'
53        run: |
54          # make sure to checkout to pr branch and resolve detached HEAD state
55          git checkout ${{ github.ref_name }}
56          # TIPS: surrounding with single quote, you can ignore `code` string in outputs
57          cat <<'AICTL_REVIEW_EOF' > response.md
58          ${{ steps.aictl-review.outputs.response }}
59          AICTL_REVIEW_EOF
60          gh pr comment --body-file response.md "${URL}"
61        env:
62          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
63          URL: ${{ github.event.pull_request.html_url }}

Results

To explain the above, the flow is:

  1. It runs when a PR is created or updated
  2. It fetches all commits in the PR and produces the file diff (git-diff)
  3. It sends the diff from step 2 to the generative AI via aictl. The contents of query are up to you
    • Putting [skip ai] in the commit message skips execution
  4. It posts aictl’s output to the PR

Here’s an image of this very article being reviewed in a PR.

I’d say the reviews it gives are pretty good. If you want more detailed reviews, you can give more specific instructions about what to review in the query of aictl-review.

Limitations and Future Plans

It’s reasonably practical at this point, but here’s what I’d like to implement in aictl going forward.

  1. Right now it’s forced to the cheap GPT-4o-Mini model, so let the model be chosen freely
  2. Support other generative AI providers as well (this one looks time-consuming)
  3. Let it remember what was submitted in past PRs so it can understand context

I intend to keep developing it as I find the time.

Share


See also