So here's what I'm trying to do
Given I changed a configuration in the load balancer
And I added that to my terraform declaration
When I run a plan there are zero changes which is expected
Do I need to refresh at this point to match my hardware state before applying?
Or when I run an apply this would just update the state?
If you've changed the settings outside of Terraform and you've updated the Terraform configuration to match then indeed there's no extra step to run here: terraform plan should report that it detected the value changed outside of Terraform (assuming you're using Terraform v1.0.0 or later) but then report that it doesn't need to make any changes to match with the configuration.
Note also that in recent Terraform the terraform refresh command is still available but no longer recommended. Instead, you can use terraform apply -refresh-only to get a similar effect but with the opportunity to review the detected changes before creating a new state snapshot. In the situation you've described, a refresh-only apply like this will also allow you to commit the detected change as a new state snapshot so that future terraform plan won't re-report that it detected a change made outside of Terraform, which might avoid your coworkers being confused by this message when they make a later change.