13 Feb Yak shaving – Introducing Krzysztof
Hi, my name is Krzysztof and I am the last addition to the Teal team. Before Teal I was associated with the Telco sector, doing Purchasing, Project Management, Program Management, technology architecture in the areas of IP, IMS (Voice) and LTE (4G) and finally everything to do with network quality, both on data and voice services. Recently a management consultant, still around telco, with focus shifted towards enterprise agility and related software’ization. I spent a lot of time explaining why change done the “old” way is barely useful nowadays, why CI/CD is important, why testing once-half-a-year is a mistake, why clouds and automation are taking over and how to drive them in companies.
I was touching all the interesting topics, but very seldom to the point when I felt this:
The decision to stop circling around and do them came after a long internal struggle and long preparation related to clouds: AWS:SA certification, GCP certification prep, including hours on QuikLabs with Kubernetes and cloud networking (GCP cert pending).
Still, coming to a technology company after years in management consulting is what you might expect… and more. And I do not feel any shame to share that 😉 My first task, to get my feet wet (and other than looking for a project, of course) was to write a blog post about our work on an EASE training. The idea was to simplify Terraform and PowerShell scripts we used and add some description of ideas behind. What sounded so innocent ended up as a constant stream of yak shaving opportunities. BTW if you do not know what “yak shaving” is, I recommend having a look here:
I started off wanting Terraform to work in my mixed Windows/Linux environment, on the Linux side. But alas, WSL 1 had strange issues with file permissions when running Visual Studio Code in WSL mode. After some searching, it turned out they are known issues. The solution was to either use WSL2, which required Windows Insider program entry, or a proper Linux installation. I like my Windows/Linux so after reading a bit about my chances of frying the machine with unstable Windows, I decided to opt in Windows Insider. Well, the computer didn’t fry. But I still had a close encounter with a heart attack, when I noticed my main Veracrypt-encrypted data drive missing from the drive list and marked unusable and unformatted in the drive partition tool. Took me a few hours to figure out that it was only a Volume Header missing and that Veracrypt was able to fix.
With a running Visual Studio Code and Terraform I started playing with code. “Let’s do it properly and do version control” they said. I used to be a _very_ basic git user. When a chance to improve comes my way I never say “no” 😉. A lesson about git: make sure your .gitignore is there BEFORE you do ANY commits. A helpful resource for that is: https://www.gitignore.io/ Otherwise, you will be wondering for a few hours why in the world git is trying to push “/.terraform/…/azure_plugin” (110 MB) to GitHub, even AFTER you added a .gitignore to the directory, explicitly telling it to ignore that directory and all its contents. In case you’re like me from a before writing this text, and still wonder why it is doing that, it is because I did “git add .” before one of the earlier commits when .gitignore was not present neither in the index nor tree. On push they get transferred to the tree on the other side (GitHub) one after another, in the order of commits…
The solution was to git squash those commits together and do “git rm cached” to unwind those earlier commits without .gitignore, to finally recommit again. This is when you learn the hard way that “There are three places where a file can be – the tree, the index and the working copy. When you just add a file to a folder, you are adding it to the working copy. When you do something like git add file you add it to the index. And when you commit it, you add it to the tree as well.” (source: manojlds on stackoverflow)
Even with yak shaving going great, finally I was able to simplify our Terraform script that setup a minimal resource set including a virtual network, subnet, an interface plus a public IP, VM and security rules. The remaining puzzle element was making Terraform run a script inside the VM. Earlier, we chose azure machine extensions to run PowerShell, taking the script from GitHub. My thinking was around ways of simplifying it, so that it does something, while not having too much text and avoid the GitHub pull. I started playing with the script… No, dear yak, I did not forget you. I am right here with my scissors…
Starting simple script:
# "$(Get-Date) This is a test." | Out-File -FilePath "C:\Users\Public\TestFile.txt" -Append Put in a key-value pair for the Azure Extension in JSON: "commandToExecute": "powershell.exe -ExecutionPolicy Unrestricted " $(Get-Date) This is a test. " | Out-File -FilePath "C:\Users\Public\TestFile.txt\" -Append", #
# azurerm_virtual_machine_extension.MYDEPLOY: "settings" contains an invalid JSON: invalid character '$' after object key:value pair #
Hmm… Should I sanitize the “$” character? 20 minutes later I was SURE based on 3 different sources that it is not even on the list of characters requiring escaping. All quotation marks are white, while text in-between is orange. Brackets looks fine. What the…? Newbie mistake… There are multiple quotation mark sets in there: “key: value, text, value, text, value”, which isn’t correct JSON, but since the number of quotes is correct, everything LOOKS fine. After I figured that out, which took me a while, I found an online json format checker, allowing me to escape all \ and “ without typing “terraform plan” every time.
All sanitized. Unfortunately, Terraform first builds the whole stack for 6 minutes, just to get stuck without any feedback from the VM. Great. How can I get this cycle shorter? After reading/listening about Terraform a bit more, I decide that MODULES are the thing I need. I will make one module, building the infra under the extension and later execute the extension as many times as I want. The big awareness jump was when I understood, that this is just NOT the way terraform works. I was mixing up the description of the infra (all “.tf” files, including module descriptions) its state (.tfstate). There is a method of transferring state using “terraform state mv”, but that really is not needed here… Once I got into automating fervor, I forgot that I was using abstractions that translate into virtual (and sometimes physical) resources somewhere. Extensions are available in Azure and testing the script is so much simpler directly…
So, 4 days after the moment in which I thought it would take “an hour or two”, I can say I am done both with the task as well as with the newly short-haired yak. Hope I at least made you chuckle 😉
In our January blog, we started an SAE deep dive series and explained how to use Hyper-V as a secure hypervisor in an (E)SAE scenario. Since by far not all our customers use Hyper-V, but many also use VMWare...15 June, 2020
(E) SAE Deep Dive Series Part 1: Hyper-V Host Guardian Service (HGS) and Shielded VMs in an EASE Environment
After the success of the first ESAE series, we decided to launch a deep dive series in which we go into a little more detail on various measures....16 January, 2020
After Hyper-V HGS and VM protection with VMWare, now the third part of our (E) SAE Deep Dive Series follows. Maybe you follow us on LinkedIn, Xing, Facebook, Instagram or Twitter and ...15 July, 2020