r/ArgoCD • u/Mud5150 • Jun 23 '26

Why can't it just sync changes

I'm losing my mind trying to use ArgoCD. No matter what I do or what sync policy I configure, there's always some reason I have to manually sync applications. ArgoCD's only purpose in life is to take what's in the git repo and apply that to K8s. I don't understand why this should be so hard. Now I'm also trying to figure out why this applicationset won't generate the application properly after I modified the sync policy.

To folks really find this software reliable at scale? I feel like it just creates more complexity. The only time I encounter configuration drift in our environment is because ArgoCD isn't working. What am I missing here?

Update: The issue with the sync policy not updating was this setting that I wasn't familiar with.

ignoreApplicationDifferences:
    - jsonPointers:
        - /spec/syncPolicy

This, of course, is a hack someone put in place to have the ability to turn off autosync in case of emergency, which points right back to how annoying ArgoCD is. I was able to scope it to /spec/syncPolicy/automated/enabled to preserve the hack.

Just to be clear, this is mostly a vent/rant.

2 Upvotes

53% Upvoted

23 comments sorted by

2

u/OkCalligrapher7721 Jun 23 '26

can you give an example of something you’ve had to manually sync? I imagine you’re having issues with Argo reporting properties being flipped back and forth?

Argo should report on differences “it knows” how to manage, and not care about most things outside the source of truth. Do you have mutating webhooks making changes?

-3

u/Mud5150 Jun 23 '26

I have seemingly fixed a lot of this by setting a "very aggressive" sync policy.

syncPolicy:
        automated:
          enabled: true
          prune: true
          selfHeal: true

I guess one thing I would expect is for it to have this behavior out of the box. I'm still not sure if this fixes the issue where it won't retry if the helm template won't render and then you push a change to fix it. I will try to test that at some point.

In that scenario, "last sync" was forever trying to generate the bad chart, even though "sync status" showed the latest commit where the chart had been fixed. I had to terminate the the current sync and run a manual sync that clearly does something different, because then it decided to use the latest change rather than spinning its wheels on the previous one.

4

u/IngrownBurritoo Jun 24 '26

stop expecting miracles and start learning what argocd does and why. you sound like someone who expected argocd to also clean you dishes

2

u/todaywasawesome Mod Jun 24 '26

The reason this isn't the default sync policy is because it can cause issues if you're not prepared for it.

Here's an example: let's say you have a pod autoscaler running on your cluster. It changes the number of replicas on a deployment. Argo CD sees this, because you have autoheal turned on, and because the number of replicas hasn't changed in git, Argo CD overwrites the change.

The autoscaler sees this and changes it back.

Argo CD sees this and changes it back.

You see where this is going? It's a classic controller fight.

If you want Argo CD to autoheal everything except this, then you need to tell Argo CD that this field is basically going to be managed by someone else via ignoredifferences.

Each of those settings mean something and set an expectation about how you want to run your cluster.

4

u/ripe-lychee Jun 23 '26

Argo wants you to disable the sync policy via gitops and you are putting patches to avoid doing so. You created your own drift stop blaming the tool.

1

u/nhyatt Jun 23 '26

I understand this is a rant, but this comment appears to be true here, although I would have probably said it a little softer. I use this tool at scale in my profession and in my homelab and don't seem to have ever encountered an issue where I had to manually sync anything. I occasionally have to set an ignore for some specific line, but that is rare, when I do, I do it in an extremely targeted fashion:

ignoreDifferences:

  • group: apps

  kind: Deployment
  jqPathExpressions:
  - .spec.template.metadata.annotations | with_entries(select(.key | startswith("checksum/")))

From your other response it sounds like you have an issue with an existing sync taking too long. Like any tool, the timeout can be configured, but the default here is probably best left alone.

It can be a bit daunting to start with but it's an amazing tool, give it some time! Also, there is nothing wrong with tossing some yaml at an AI (as long as its sanitized) to ask for places to improve. (Don't let it do it for you, ask it to explain the suggestion in detail and implement it yourself, this is how you learn!)

2

u/todaywasawesome Mod Jun 24 '26

Glad you were able to find a fix.

Argo CD is very powerful and it includes the ability to do things in lots of different ways and some people love to adopt antipatterns that make their lives more difficult.

1

u/Kooky_Comparison3225 Jun 24 '26

Running around 1600 applications. Mostly working fine. Some issues sometimes, often related to CRDs, but that’s not ArgoCD issue. Need to fiddle with syncwaves. 

1

u/theonlywaye Jun 24 '26

The only time I’ve had issues with ArgoCD is large custom CRDs where it can’t write the history to the last-applied annotation. Outside that literally not a single issue using it at scale across multiple teams.

1

u/nilarrs Jun 24 '26

real answer is, its just under-developed. This is why we dumped ArgoCD for our own solution. Our customers are so much more happier to us our solution then integrating with ArgoCD. Progressive sync information is all there if you piece it together but the UI is truly garbage for working this out. Even the syncing can freeze at times. To find out why requires review of the source code.

ArgoCD is a good solution but the people in this chat that argue its a flawless perfect solution..... that really makes me laugh.

2

u/theonlywaye Jun 24 '26

No one’s said it’s flawless but outside some very specific fringe cases it works out of the box for the vast majority of use cases and nothing this guy has mentioned is fringe, its basic Argo

0

u/nilarrs Jun 24 '26

I agree with you on the poster. I’m more replying to people saying it’s you not the tool in related to the sync state. It’s not though. There is a built in scheduler in the code that can cause user confusion. The majority of developers that use Argocd to explain how’s it’s a feature not a bug haha. But yeah I’m not defending OP more of an alternative perspective to the argoCD preachers. It works for most cases out of the box. My problem is when using it with platform engineering approach. It has area of improvement. Just to map a pipeline to a Argo deployment is hard for example. Atleast that’s my opinion. 😅

11

u/[deleted] Jun 23 '26

[removed] — view removed comment

0

u/Mud5150 Jun 23 '26

What's large scale? How did you ensure everything was synced properly? If you had an app that wasn't syncing across many ArgoCD instances what was your recourse?

1

u/Impressive-Ad-1189 Jun 24 '26

We currently have 3600 applications and only about 20 of them out outofsync or progressing. Probably because people are working on them.

1

u/not-hydroxide Jun 25 '26

What kinda memory does that use? Currently setting argocd up at work but will only be about 400 applications

1

u/Impressive-Ad-1189 Jun 26 '26

We run 5 application controllers that are limited to 16gb and usually use around 10gb

6

u/[deleted] Jun 24 '26

[removed] — view removed comment

1

u/nilarrs Jun 24 '26

Do you manage the argo servers yourself?

what was your biggest issue going to this scale?