My son is now five, he went out 5-6 time last year, mostly in the afternoon. Took him out once in the morning, I don't pack that much gear for a week as i did for him. He slept through the morning shooting, wakes ups and asks for breakfast. Hot pancakes in the blind, still talks about it.
It's different hunting with kids, but no less rewarding. Keep taking him out.
There is room for all of God's creatures, right next to the potatos and gravy.