I , too, have spent a great deal of time in this construct. As you said, many have, and many have arguments for and against God based on this concept alone. The story of Job seriously disturbs me, for example, and if you allow that metaphor, humans have indeed been thinking on this a long time. Some have even become more convinced of a God, such as Viktor Frankl, Man's Search for Meaning, and CS Lewis, Surprised by Joy... both a serious study of human and animals suffering, and the role of evil.
It is not my intent to persuade you. I love that you're grappling with this. We are where we are, that's it, but your perspective from living in a desert would not invalidate the person's perspective that lives on the sea. It suggests to me that the core of you would like less suffering, and if you can figure that out, even profit on it in the market, bringing peace to the world... God or not, I wish you the greatest success in this endeavor. Grapple on! Keep wrestling with God, my friend Jacob. Buddhists technically don't believe in God, but they do offer a path out of suffering. I would, though, like to suggest several caveats that might result in a different conclusion from a meta analysis of suffering in general.
The easiest one that does not require any spiritual rumination goes as follows. Assume for a moment that REM dreaming is a natural function of existence. For sure humans dream, some have nightmares. We believe animals dream, and there is recent evidence that at least octopuses have nightmares, and PTSD. Animal suffering seems well established. The average person will have X number of dreams and Y number of nightmares. In dreams, sometimes animals, other entities, and even the dreamer themself will suffer. Should I conclude the dreamer doesn't exist? Should I conclude that the dreamer is evil because there is suffering of others and self within in the dream?
Consider horror movies. Not only did an author contemplate ways in which people can suffer, they put it in a form of media in which they can share that vision. Are they increasing suffering or minimizing suffering? Did you know some countries have outlawed graphic scenes in movies in which humans harm animals? That would be an argument for a greater construct of good and evil in the collective unconsciousness, in which an ideal is absolutely necessary to in order to even have the question you're grappling with. Ideal, archetype, or real entity, or nothing, that increases the likelihood of a God, either as a real thing or a construct shared. I even allow there might be a being superior to human but nothing like anything humans have imagined. I suspect humans are not the epitome of all life in the universe, as evidence by all the suffering we create for ourselves and others.
We do, more often than not, suffer by our own hands.
Nietzsche said 'God is dead.' He was actually arguing that there is a God and society is killing Him. In doing so suffering increases upon the world, both for humans and nature, by the hand of humans who have fallen for the lie that there is no soul, it's just matter, and consciousness itself is an illusion, which leads to suffering is an illusion.
I am not arguing suffering is an illusion. Suffering is real, though subjective, greatly affected by belief. Again, continue with the horror genre for a moment.
Some folks actually pay to see horror. The audience is no more evil than the author; they are just people contemplating evil and how not to succumb to it! Also, and fair enough, for sure, no one is really being murdered or tortured in these fictions, and so perhaps the point is mute...
Exception, hypothetically, if humans are souls first having a human experience, then we can argue that just as normal souls who are seeking to understand and contemplate nature through the consumption of the dark arts, we in turn created for ourselves a 'sand box' of duality in which the greatest contrast between light and dark, negativity versus positivity, or good versus bad allows us the greatest experiential insight. We can better understand the fiction of life and vicariously apply that learning only after we have taken our blows. I am okay with person calling all spiritual concepts myths. I would rather have a lesson learned vicariously than by having direct suffering. That said, people learn more by experience than they do from an academic study. You can read all the books on chess that you want, but to be a grand master you must have a series of worthy opponent, bosses that knock you on your ass, and a significant number of losses.
No boxer ever rose to the top without a single blow to their face.
There is no story yet created in which protagonist did not suffer adversity. This does not mean we should endeavor to create adversities for people to overcome, but if you are Dorothy and you find yourself in Oz, the only path to you is the Golden Path. On the Journey, Dorothy found other suffering and sought to ease their pain. That is 'right.' That's the golden rule. There is no morality without the hierarchy of lesser to greater, and it is our own suffering that leads to the discovery of others suffering. If you stop there, you have not gone far enough to discover the way home. The golden rule and the yellow brick road is not an accidental metaphor. Glenda, the good witch, did not liberate Dorothy from all suffering. She could have, and had she done so, Dorothy could have saved herself and left a world in suffering. Dorothy had to confront and grapple with the suffering of others first to get out of herself, and then, only then, to have the realization that she always had the power to go home, as well as raising all of Oz out of suffering.
She had the power to stop suffering. Suffering lead to the journey that lead to the resolution.
Most the time, when we have everything we want the learning curb to optimum maturity is flat. The farm, the garden, surrounded by family is the ideal. Dorothy's parents are absent, the same way father god and mother god are absent from the context of our human life. We are truly orphans! We're like nothing else in nature... We are like everything in nature.
The most perfect, painless existence is soul life, the life between physical lives. Allow that as my construct, even if you can only abstract it. Many people will not leave the nest, or the comfort zone. As children, God didn't do evil to us, but rather we wanted to play, we created a sandbox, and many souls have made this box rather unpleasant, but not because they are bad actors but because they are undisciplined, testing the boundaries of suffering and pleasure.
Suffering is part of the consensual reality on this planet. There will be suffering until we consensually agree to end suffering. And if you think suffering will end when animals stop eating plants and other animals, then you have failed to consider how animals and plants and everyone suffer if populations get out of control, unchecked by a death force. There is a death force. Every cell in your body is schedule to die. If this wasn't true, you would have webbed fingers, a tail, and fur. These features are observed in the development of the fetus. Those cell often die before birth. Some still get born with them. You have trillions of cells. Imagine life if you kept every cells alive that was ever a part of your body. Seriously, how many cells do you lose a day? That person who experience no cell death would be one giant mass of cells, unable to move on land. That would be suffering.
You can have no free will without an unconditional source of love. You can't free will and love without contrast, suffering. You clearly are a loving being, as you care about suffering. I love you! You're awesome. Suffering is the consequence for getting out of balance. If in your endeavor to end all suffering you become a helicopter parent, you will increase suffering because the child will never develop the callouses to play the guitar. You need pain, joy, callouses, persistence and discipline to play the blues.
I dare say BB King know some pain and joy, and through suffering brought other joy by diminishing their pain. All acts of charity and kindness lead one to the conclusion there is good without invalidating there is evil. You argued for no God, but did you argue for no evil? Is it okay that animals get eaten? And if animals eat, does that mean they are evil? There is joy in eating, for sure. How much suffering would there be if allowed no predator to eat?
All animals are also evolving, their souls are evolving, and so the rules of the game are just as poignant. There is a space where even the souls of animals go for respite. There is also evidence that animals have sometimes employed compassion by helping a being outside of their species, even beings they would normally prey upon, which means they, too, sometimes contemplate kindness.
Your question and observations are valid because you recognize animals have souls. No soul, no suffering. Do robots suffer? If you were a neural scientist, you would say consciousness is an illusion, therefore no suffering exists- humans are meat zombies. Book 'Who's in Charge,' by Michael Gazzaniga, a neural scientist champions that idea. You might say fish don't feel pain, neither do insects... But the most recent studies show fish do feel pain, and they show gratitude when you remove the hook and let them go. Even insects suffer. A study flies reveals as the number of dead flies on the windowsill goes up, the anxiety of the fly hitting the window goes up. It is in the literature even plants suffer. They make sounds when being damaged that are suggest they in are in distress.
Pain and pleasure do motivate us first selfishly. Selfish people increase suffering until they become aware of other suffering. By first suffering, and then extension becoming aware of suffering, we learn empathy. We learn there is an 'other,' both real and archetypal. You clearly have idea of the 'evil.' By recognizing that, the is a necessary opposite and equal archetypal manifestation of that, in which human employ to minimize suffering. Abstract or real, it is real in its affect on society. Call it or name it anything you like.
We learn to walk more kindly on the planet only in recognition of other.
Now, add the caveat that 'some' scientist are wondering if we live in simulation. Once again, scientists are suggesting suffering is an illusion. Simulated suffering, though, is still suffering, even if you signed up to play the game. You got to suffer losses by the bosses to evolve your competency! I hope all the video games in which the only way to level up is death to the lesser beings is not causing avatars to suffer... but we can go there by extension of the dream metaphor. Especially if we are god's dream, or nightmare...
I wish you well on your journey, fellow traveler.