Mitigating Overconfidence in Large Language Models (LLMs) in Production Environments