Git not using the right SSH key when pushing to github

So I have two different github accounts and I've created two different SSH keys for them id_me and id_work, however when I try to make a git push from to my work account, it authenticates me with my personal SSH key instead.

This what I've done:

Added each SSH key to its respective github account. Then followed the steps to add them to my SSH agent:

ssh-add -K id_me
ssh-add -K id_work

And set up my ~/.ssh/config like this:

# Personal github account
Host github.com
  HostName github.com
  User git
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_me

# Work github account
Host github.com-work
  HostName github.com
  User git
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_work

I'm trying to make some changes to a repo on my work account. First, I make sure to clone it using the proper command:

git clone [email protected]:test.git

However, when I push the changes back up, it throws a permission denied error msg:

ERROR: Permission to test.git denied to doctopus.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Also, I notice that when I run ssh -T github.com-work, it authenticates me on my personal account:

Hi doctopus! You've successfully authenticated, but GitHub does not provide shell access.

So the issue looks like git is not using the right SSH key when it pushes to github.