I Built aictl to Review Code and Articles with Generative AI, and Had It Review My Blog Posts

Having an editor who reviews your PRs


Posted on Tue, Oct 1, 2024
Tags generative-ai, githubactions
generative-ai, githubactions
📝 This article is a translation of the original Japanese post. View original

Automating Blog Post Reviews with Generative AI

I’ve been writing blog posts for about 8 years now, but since I write them on my own, nobody reviews them. So I developed go-zen-chu/aictl to have articles reviewed via GitHub Actions when I open a PR.

There are products and tools like Copilot that do code review with generative AI, but there don’t seem to be many tools for having blog posts reviewed, and it looked easy to build using the OpenAI client, so I made one — also as a learning exercise.

How to Set It Up

The setup is described in the README of go-zen-chu/aictl, but in short:

  1. Place the yaml below at .github/workflows/check-pr.yml
  2. Set AICTL_OPENAI_API_KEY in the repository secrets
    1. For the API key, see API Reference - OpenAI API

and setup is done (secrets.GITHUB_TOKEN appears in the workflow, but it exists by default when GitHub Actions runs, so no configuration is needed).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
name: check-pr
on:
  pull_request_target:
    types: [opened, synchronize]
jobs:
  aictl-review:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - name: Get number of commits for fetch depth
        run: echo "fetch_depth=$(( commits + 1 ))" >> $GITHUB_ENV
        env:
          commits: ${{ github.event.pull_request.commits }}
      - name: Git checkout until fetch-depth
        uses: actions/checkout@v4
        with:
          fetch-depth: ${{ env.fetch_depth }}
          # need to specify ref to fetch PR head, otherwise you get main branch with no HEAD
          ref: ${{ github.event.pull_request.head.sha }}
      - name: Fetch base branch as origin
        run: git fetch origin ${{ github.event.pull_request.base.ref }}
      - name: Get the diff between PR and origin branch and get commit message too
        id: git-diff
        run: |
          # If fetch-depth is not specified in checkout, HEAD cannot be found
          # Make sure to filter out deleted file in `git diff` since you cannot review deleted files
          diff_files=$(git diff --name-only --diff-filter=ACMR origin/${{ github.event.pull_request.base.ref }}..HEAD | tr '\n' ',' | sed 's/,$/\n/')
          echo "diff_files: ${diff_files}"
          echo "diff-files=${diff_files}" >> "$GITHUB_OUTPUT"

          # In PR, you cannot get `github.event.head_commit.message` so you need to get commit message via git log
          commit_msg=$(git log --format=%B -n 1 ${{ github.event.pull_request.head.sha }})
          echo "commit_msg: ${commit_msg}"
          echo "commit-msg=${commit_msg}" >> "$GITHUB_OUTPUT" 
      - name: Run aictl for code review
        id: aictl-review
        uses: go-zen-chu/aictl@main
        if: ${{ steps.git-diff.outputs.diff-files != '' && !contains(steps.git-diff.outputs.commit-msg, '[skip ai]') }}
        with:
          query: |
            The Markdown files contain technical articles about programming, life hack articles, book reviews, and so on.
            Please review them from the perspective of making them easier to read and reaching more readers.

            The other files are the source code of the blog system itself.
            Please review them from the perspective of code quality and maintainability so that a better blog system can be built.
          language: 日本語
          text-files: ${{ steps.git-diff.outputs.diff-files }}
        env:
          AICTL_OPENAI_API_KEY: ${{ secrets.AICTL_OPENAI_API_KEY }}
      - name: Post aictl review result to PR comment
        if: steps.aictl-review.outcome != 'skipped'
        run: |
          # make sure to checkout to pr branch and resolve detached HEAD state
          git checkout ${{ github.ref_name }}
          # TIPS: surrounding with single quote, you can ignore `code` string in outputs
          cat <<'AICTL_REVIEW_EOF' > response.md
          ${{ steps.aictl-review.outputs.response }}
          AICTL_REVIEW_EOF
          gh pr comment --body-file response.md "${URL}"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          URL: ${{ github.event.pull_request.html_url }}

Results

To explain the above, the flow is:

  1. It runs when a PR is created or updated
  2. It fetches all commits in the PR and produces the file diff (git-diff)
  3. It sends the diff from step 2 to the generative AI via aictl. The contents of query are up to you
    • Putting [skip ai] in the commit message skips execution
  4. It posts aictl’s output to the PR

Here’s an image of this very article being reviewed in a PR.

I’d say the reviews it gives are pretty good. If you want more detailed reviews, you can give more specific instructions about what to review in the query of aictl-review.

Limitations and Future Plans

It’s reasonably practical at this point, but here’s what I’d like to implement in aictl going forward.

  1. Right now it’s forced to the cheap GPT-4o-Mini model, so let the model be chosen freely
  2. Support other generative AI providers as well (this one looks time-consuming)
  3. Let it remember what was submitted in past PRs so it can understand context

I intend to keep developing it as I find the time.

Share


See also