So I figured out a solution. Might not be the best answer, but it's what I got. I added my user to group called dialout and then added that group to /dev/mem and /dev/gpiomem. I used the commands below. I also had to do the same for the SPI and I2C interfaces in /dev/ too but you may not need that.
sudo usermod -aG dialout [your username]
sudo chown root.dialout /dev/mem
sudo chmod g+rw /dev/mem
sudo chown root.dialout /dev/gpiomem
sudo chmod g+rw /dev/gpiomem
If anyone has a better answer, please let me know.