Fetching updates every Salt highstate without failing highstate on error, instead reverting to defaults

Question

I have a systemd service evil, with state described in init.sls. This service requires the presence of a file /etc/evil/wicked.txt. This file pulls in updates from a python function fetch_darkness that contacts a server, the response of which changes very infrequently.

evil:
  pkg.installed:
    - name: {{ package_name }}
    - watch:
      - file: /etc/evil/wicked.txt
      - pkg: evil

/etc/evil/wicked.txt:
  file.managed:
    - contents: {{ salt.evil.fetch_darkness(grains['id'], pillar.node.hostname }} # this changes infrequently
    - show_changes: false
    - allow_empty: false
    - user: evil
    - group: evil
    - mode: 600
    - makedirs: true
    - require:
      - user: evil
    - watch_in:
      - service: evil

This creates a problem, because every highstate, it causes the service evil to be restarted, even if the actual contents of the file /etc/evil/wicked.txt haven't changed, which is 99% of the time.

So the next solution was creating a temporary file /etc/evil/wicked-temp.txt that wasn't watched by evil. /etc/evil/wicked.txt pulls its contents from wicked-temp.txt when that is changed, this way evil is only restarted when the file actually updates instead of every highstate:

evil:
  pkg.installed:
    - name: {{ package_name }}
    - watch:
      - file: /etc/evil/wicked.txt
      - pkg: evil

/etc/evil/wicked-temp.txt:
  file.managed:
    - contents: {{ salt.evil.fetch_darkness(grains['id'], pillar.node.hostname }} # this changes infrequently
    - show_changes: false
    - allow_empty: false
    - user: evil
    - group: evil
    - mode: 600
    - makedirs: true
    - require:
      - user: evil

/etc/evil/wicked.txt:
  file.managed:
    - source: /etc/evil/wicked-temp.txt
    - show_changes: false
    - allow_empty: false
    - user: evil
    - group: evil
    - mode: 600
    - makedirs: true
    - onchanges:
      - file: /etc/evil/wicked-temp.txt
    - require:
      - user: evil
    - watch_in:
      - service: evil

"rm /etc/evil/wicked-temp.txt":
  cmd.run:
    - onfail:
      - file: /etc/evil/wicked.txt

However, now the issue is, there are several servers that each get highstated, and there's many times that fetch_darkness isn't able to reach the main server, causing the highstate to fail as wicked.txt/wicked-temp.txt can't get populated.

This isn't great, because >99% of the time, the contents of the server response doesn't change. If wicked.txt doesn't exist, then sure, the highstate should fail. But if it does, then I'd like to keep using whatever already exists on the file wicked.txt and not have a failed highstate.

Is there a way to solve this problem:

1. Service must not restart unless file contents change
2. File must pull updates
3. Highstate must not fail if file already exists, but is unable to pull updates

EDIT: I should've mentioned that the file's contents do change every time. The file is a randomly generated key that is fetched from the central server, with a certain label attached, so something like 1-xxxxxxx, or 2-yyyyyy. However, 1-xxxxx is identical to 1-zzzzzz, even if the actual contents differ, which is why evil should not restart when 1-xxxxx changes to 1-zzzzz, but only when it changes to 2-yyyyyyy.

Answer 1

you already have a custom module that does the fetch, why not create a custom state module that handles the rest? this way you can determine if the file actually changed, handle it when it does and correct for errors when the fetch doesn't work.

in the first example however it shouldn't do anything that triggers that the file changed unless something with the file actually is changing. you might want to double check what is changing in the file.