Automating Blog Post Reviews with Generative AI
I’ve been writing blog posts for about 8 years now, but since I write them on my own, nobody reviews them. So I developed go-zen-chu/aictl to have articles reviewed via GitHub Actions when I open a PR.
There are products and tools like Copilot that do code review with generative AI, but there don’t seem to be many tools for having blog posts reviewed, and it looked easy to build using the OpenAI client, so I made one — also as a learning exercise.
How to Set It Up
The setup is described in the README of go-zen-chu/aictl, but in short:
- Place the yaml below at
.github/workflows/check-pr.yml - Set AICTL_OPENAI_API_KEY in the repository secrets
- For the API key, see API Reference - OpenAI API
and setup is done (secrets.GITHUB_TOKEN appears in the workflow, but it exists by default when GitHub Actions runs, so no configuration is needed).
1name: check-pr
2on:
3 pull_request_target:
4 types: [opened, synchronize]
5jobs:
6 aictl-review:
7 runs-on: ubuntu-latest
8 permissions:
9 pull-requests: write
10 steps:
11 - name: Get number of commits for fetch depth
12 run: echo "fetch_depth=$(( commits + 1 ))" >> $GITHUB_ENV
13 env:
14 commits: ${{ github.event.pull_request.commits }}
15 - name: Git checkout until fetch-depth
16 uses: actions/checkout@v4
17 with:
18 fetch-depth: ${{ env.fetch_depth }}
19 # need to specify ref to fetch PR head, otherwise you get main branch with no HEAD
20 ref: ${{ github.event.pull_request.head.sha }}
21 - name: Fetch base branch as origin
22 run: git fetch origin ${{ github.event.pull_request.base.ref }}
23 - name: Get the diff between PR and origin branch and get commit message too
24 id: git-diff
25 run: |
26 # If fetch-depth is not specified in checkout, HEAD cannot be found
27 # Make sure to filter out deleted file in `git diff` since you cannot review deleted files
28 diff_files=$(git diff --name-only --diff-filter=ACMR origin/${{ github.event.pull_request.base.ref }}..HEAD | tr '\n' ',' | sed 's/,$/\n/')
29 echo "diff_files: ${diff_files}"
30 echo "diff-files=${diff_files}" >> "$GITHUB_OUTPUT"
31
32 # In PR, you cannot get `github.event.head_commit.message` so you need to get commit message via git log
33 commit_msg=$(git log --format=%B -n 1 ${{ github.event.pull_request.head.sha }})
34 echo "commit_msg: ${commit_msg}"
35 echo "commit-msg=${commit_msg}" >> "$GITHUB_OUTPUT"
36 - name: Run aictl for code review
37 id: aictl-review
38 uses: go-zen-chu/aictl@main
39 if: ${{ steps.git-diff.outputs.diff-files != '' && !contains(steps.git-diff.outputs.commit-msg, '[skip ai]') }}
40 with:
41 query: |
42 The Markdown files contain technical articles about programming, life hack articles, book reviews, and so on.
43 Please review them from the perspective of making them easier to read and reaching more readers.
44
45 The other files are the source code of the blog system itself.
46 Please review them from the perspective of code quality and maintainability so that a better blog system can be built.
47 language: 日本語
48 text-files: ${{ steps.git-diff.outputs.diff-files }}
49 env:
50 AICTL_OPENAI_API_KEY: ${{ secrets.AICTL_OPENAI_API_KEY }}
51 - name: Post aictl review result to PR comment
52 if: steps.aictl-review.outcome != 'skipped'
53 run: |
54 # make sure to checkout to pr branch and resolve detached HEAD state
55 git checkout ${{ github.ref_name }}
56 # TIPS: surrounding with single quote, you can ignore `code` string in outputs
57 cat <<'AICTL_REVIEW_EOF' > response.md
58 ${{ steps.aictl-review.outputs.response }}
59 AICTL_REVIEW_EOF
60 gh pr comment --body-file response.md "${URL}"
61 env:
62 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
63 URL: ${{ github.event.pull_request.html_url }}
Results
To explain the above, the flow is:
- It runs when a PR is created or updated
- It fetches all commits in the PR and produces the file diff (git-diff)
- It sends the diff from step 2 to the generative AI via aictl. The contents of
queryare up to you- Putting
[skip ai]in the commit message skips execution
- Putting
- It posts aictl’s output to the PR
Here’s an image of this very article being reviewed in a PR.
I’d say the reviews it gives are pretty good. If you want more detailed reviews, you can give more specific instructions about what to review in the query of aictl-review.
Limitations and Future Plans
It’s reasonably practical at this point, but here’s what I’d like to implement in aictl going forward.
- Right now it’s forced to the cheap GPT-4o-Mini model, so let the model be chosen freely
- Support other generative AI providers as well (this one looks time-consuming)
- Let it remember what was submitted in past PRs so it can understand context
I intend to keep developing it as I find the time.