-1

at my company we are using AWS Cloudformation to manage our infrastructure. The templates are written in JSON. The problem is that the templates are getting bigger by the day 5k Lines. Company policy is to write certain projects in one template (i.e. I'm not able to divide up the code into separate templates). And as a result the templates are getting really huge.

I've been playing with yaml, json and cloudformation generators. I found some nice yaml generator (Spiff). Using it I can reduce our templates by almost 90% in some cases.

Do you think that it is a good idea to generate cloudformation Templates on the fly and work mainly with Spiff yaml generators? Or there will be a dire cost that I'll pay within the next couple of months. Please keep in mind that Spiff is a bit cryptic and not pure Yaml, so there's a learning curve for people maintaining the templates after me.

What do you think?

Moataz Elmasry
  • 179
  • 3

2 Answers2

2

I've been using the cfndsl RubyGem for about 9 months now in a couple of projects, and have been very happy with it for the following reasons:

  • Comments
    Invaluable as your scripts get more complex.
  • Related items are colocated in the source
    For example, the Output for a resource can be physically located in the source file next to that resource, rather than in another unrelated part of the file. The JSON generator ensures that all definitions go in their appropriate places.
  • Properties files
    I find that I have minimal need for stack parameters; they just add one more thing that isn't tracked in source control.
  • Ruby string substitutions
    Far easier to use than the Fn:Sub intrinsic function, although I still use the function when I want to include things like the account ID.
  • Configuration-driven stacks
    This is the big one: all of our application configs (instance, auto-scaling group, load balancer, DNS) are pretty similar, and can be represented in a few lines of configuration. Since the stack JSON is being generated by a full-featured programming language, I can loop over the config and use if-then statements to define relevant resources. A simple 100-line app template serves to generate the stacks for a dozen or more apps.

CFNDSL isn't the only option; it just happened to be something I was introduced to at the right time. I've also heard of a stack generator written in Python, and at a different time I might have preferred it.

Bottom line: generating stacks programmatically is far better than maintaining a purely declarative format, especially JSON.

kdgregory
  • 5,220
  • 23
  • 27
0

I might be a little bit late with my answer, but there is a cfpack tool that i have created to split huge CloudFormation templates into smaller ones. The idea is that when you split your main template, you create small templates that are fully compatible and actually are CF templates by itself. Then when you run cfpack bulid command it combines all templates and builds main template file that can be deployed to CloudFormation.

Another good thing is that you can create reusable drop-in templates and share it between different projects.

So, take a look at it, perhaps it can help you too. Here is a sample usage, check templates folder (that project isn't done yet, but demonstrates how to use cfpack).

Eugene Manuilov
  • 109
  • 2