Introduction:
All IaC engineers working on relatively large infrastructures with Terraform know that a growing number of resources defined in the same Terraform module slows down the "apply" and "plan" operations. Terraform needs to communicate with the API server to check if all the properties of resources in the state file have been modified. In conjunction with the ideas presented here, we propose another idea to speed up this process by using resource hashes to track changes.
The Main Idea:
This idea is not new, and it does not belong to us. We suggest using the same approach as an experiment. We’ve previously discussed lock files used to track dependency changes. Terraform is no exception; it uses the .terraform.lock.hcl file to track provider changes. When a resource defined in HCL is created, we can calculate the SHA-256 hash of its representation and store it in the state file alongside other YAML keys. We store these hash sums to compare them with others. So, where do we get these sums?
We can implement a standalone service or a Kubernetes operator that subscribes to, for example, CloudTrail and recalculates the SHA-256 sum of a resource when a change is logged for it. The recalculated sums can be stored in a key/value store. When Terraform needs to check if the resource properties have changed, it pulls the recalculated and stored hash sum and compares it with the value in the state file. If the hash sums are the same, Terraform avoids initiating API calls for that particular resource, thus saving valuable time.
Implementation:
As previously mentioned, the best place to store the hash sums of successfully created or updated resources is the state file from the client side. It is better to keep it self-contained at the core technology level, which, in our case, is Terraform. The next question is how to store and update the hash sums on the server side. Let's take AWS as an example. Suppose we have an EC2 instance; we can get its representation in YAML, calculate the SHA-256 sum, and store it on an AWS service or an independent key-value store.
All is good so far, but in the next step, Terraform won’t pull our hash sum, it won’t compare anything, and it will start sending its API requests to AWS as usual, wasting time. Thus, the first idea to solve this problem might involve modifying Terraform's core functionality. Good luck getting the budget for that.
It may be possible to implement a wrapper like Terragrunt or a provider. Please share your ideas on that.
Conclusion:
Implementing a wrapper that uses hash sums to modify the --target lists for each application and using Terraform's --target option might be the easiest solution. However, it is unclear how a provider could help solve these problems. The best solution would be to modify Terraform's core functionality of state verification.
Acknowledgements:
We started thinking of Terraform's performance improvements with Vahagn Martirosyan when he invited us to discuss his proposals. After a conversation with Soren Martius, we created a LinkedIn post with Terramate and Terraform change requests in the comments (vote up on GitHub), which triggered this discussion with Hayk Gaziyants.
Comentários