Downloaded 118 times
![Software development is more than just
writing [Go, Java, NodeJS, C#, …]](https://image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-19-2048.jpg)
![Why Platform Engineering
“[T]he reality is that state of the art cloud native technology is still too hard to use if
every product engineering team has to individually solve common problems
around networking, observability, deployment, provisioning, caching, data storage,
etc.”
https://medium.com/@mattklein123/the-human-scalability-of-devops-e36c37d3db6a](https://image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-24-2048.jpg)
![$> curl -s localhost:9200/_cluster/settings?include_defaults=true
| jq '.defaults'
| grep ""[a-z_.-]*": ["[]"
| wc -l
361
Elasticsearch settings](https://image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-56-2048.jpg)
![Software development is more than just
writing [Go, Java, NodeJS, C#, …]](https://crownmelresort.com/image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-19-2048.jpg)
![Why Platform Engineering
“[T]he reality is that state of the art cloud native technology is still too hard to use if
every product engineering team has to individually solve common problems
around networking, observability, deployment, provisioning, caching, data storage,
etc.”
https://medium.com/@mattklein123/the-human-scalability-of-devops-e36c37d3db6a](https://crownmelresort.com/image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-24-2048.jpg)
![$> curl -s localhost:9200/_cluster/settings?include_defaults=true
| jq '.defaults'
| grep ""[a-z_.-]*": ["[]"
| wc -l
361
Elasticsearch settings](https://crownmelresort.com/image.slidesharecdn.com/platformengineering101-190412111542/75/Platform-engineering-101-56-2048.jpg)
More Related Content
Similar to Platform engineering 101
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Platform engineering 101
- 1. Platform Engineering 101 Empoweringdevelopers to deliver - as-a-service Skyworkz - https://skyworkz.nl Sander Knape - https://sanderknape.com
- 2. Hello, I’m SanderKnape Cloud Engineer @ Skyworkz
- 3. Let’s build anapp ● Application that connects to a database ● Works on my machine™ → Let’s get it into the Cloud!
- 4. Cloud provider /runtime environment
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. There’s quite abit going on Cloud provider / runtime environment Infrastructure as Code CI / CD Artifact management Security scanning Configuration management Secret management Monitoring, Logging, Metrics Alerts Database migrations Database anonymization Cost insights Security Application
- 19. Software development ismore than just writing [Go, Java, NodeJS, C#, …]
- 20. You need aplatform
- 21.
- 22. Definition time: “PlatformEngineering” The composition and integration of a set of processes, tools and automation (components) to build a coherent platform with the goal of empowering developers to be able to easily build, maintain and operate their business logic.
- 23.
- 24. Why Platform Engineering “[T]hereality is that state of the art cloud native technology is still too hard to use if every product engineering team has to individually solve common problems around networking, observability, deployment, provisioning, caching, data storage, etc.” https://medium.com/@mattklein123/the-human-scalability-of-devops-e36c37d3db6a
- 25.
- 26. Specific knowledge isrequired
- 27.
- 28.
- 29. Who has theresponsibility? 1 or 2 platform engineers 1 platform team multiple platform teams
- 30. Multiple approaches ● “Justhand me your source code on a USB stick. I’ll put it on the server and make sure and that it keeps working” Or… ● “Create a pull request <here> to automatically get a Git repository, CI/CD, filled with a Hello World application, that deploys all the way to production”
- 31. Challenges we’ll talkabout 1. Protecting the platform for organizational scalability 2. Building an opinionated platform 3. Specifying contracts 4. Collaborating with your users
- 32. Protecting the platformfor organizational scalability
- 33. AWS Lambda: LanguageSupport
- 34. AWS Lambda: CustomRuntimes
- 35.
- 36. The question toask for each new feature How much support will I need to provide after I have delivered this feature? Less (or equal) is the only acceptable answer
- 37. Reducing toil “Toil isthe kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” https://landing.google.com/sre/sre-book/chapters/eliminating-toil/
- 38. Examples of toil ●Manually creating Git repositories and granting the correct people access ● Manually creating CI/CD pipelines based on another project ● Running load tests from a laptop with a security token that you can not share ● Putting secrets into your secret management solution
- 39. Examples of toil:old-school operations ● Manually; ○ Flushing caches ○ Rebooting servers ○ Renewing SSL licenses ○ ...
- 40. Toil leads tobottlenecks
- 41. Toil leads toboredom...
- 42. … or toilleads to stress
- 43.
- 44. Self-service instead oftoil ● Manually creating Git repositories and granting the correct people access ● Manually creating CI/CD pipelines based on another project ● Running load tests from a laptop with a security token that you can not share ● Putting secrets into your secret management solution
- 45. Scalable support Developer: Hello,can you help me with “something” for feature X? Platform: Did you check the docs for feature X? Developer: Yes, but it doesn’t mention anything about “something” Platform: I see, give me as second Developer: Sure Platform: I edited the docs to include “something”, does this answer your question? https://github.com/org/docs/pull/1337 Developer: Totally, thanks!
- 46. How to protectyourselves ● Automate. Think “as a service”. ● Prefer managed solutions above self-hosted. Every minute you spend on maintaining a self-hosted solution is time not spend on improving the platform usability ● Prefer documentation above single answers ● Make the platform as simple as possible
- 47. Building an opinionatedplatform
- 48. AWS Lambda: LanguageSupport
- 49.
- 50. Abstraction levels “Serverless” “IaaS” Levelofabstraction “Bricks” Platform Managed externally Platformfeatures Splunk Cloud AWS CloudWatch AWS Secrets Manager Google Cloud Datastore
- 51. Abstraction levels “Serverless” “IaaS” Levelofabstraction “Bricks” Platform Managed externally Platformfeatures HashiCorp Vault Splunk Self-Hosted Prometheus/Grafana MySQL on EC2 GitLab on EC2
- 52.
- 53.
- 54. Abstraction levels “Serverless” “IaaS” Levelofabstraction “Bricks” Platform Managed externally Platformfeatures Developer responsibility
- 55. Self-service ● Lambda languagesv.s. Lambda runtimes ● Hosted, shared CI/CD solution vs. BYO CI/CD solution ● Default Docker / EC2 images vs. built your own ● Default network with every application vs. create your own VPC/Subnet
- 56. $> curl -slocalhost:9200/_cluster/settings?include_defaults=true | jq '.defaults' | grep ""[a-z_.-]*": ["[]" | wc -l 361 Elasticsearch settings
- 57.
- 58.
- 59. Business value What isthe business value of each development team maintaining their own; ● CI/CD server ● Monitoring solution ● Network ● ...
- 60. Getting buy-in Team Manifesto Webuild everything with developers. We’ll always select a pioneering team that has interest in a new service/feature, and work with them to make sure we pick and configure the right tool and solve the root problem.
- 61. How to sellan opinionated platform ● Simple to use, high-level abstractions; developers shouldn’t want to use anything else ● Make everyone aware of the constant balance seeking ● Build the features together with developers
- 62.
- 63. AWS Lambda: LanguageSupport
- 64. Your contract isyour interface “Serverless” “IaaS” Levelofabstraction “Bricks” Platform Managed externally Platform features Interface
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71. Documentation “Serverless” “IaaS” Levelofabstraction “Bricks” Platform Documented by theexternal party Documented by the platform team Interface
- 72.
- 73. Shared Responsibility ● Gitrepository automation ● CI/CD autoscaling ● Network ACLs ● Kubernetes cluster ● Logging / Metrics agent on EC2, listening on specific port Platform Development ● Creating Git repositories ● CI/CD configuration ● Security groups ● Application in Kubernetes ● Shipping logs/metrics to specific port
- 74. Continuously test yourresponsibility ● Are my agents autoscaling? ● Are my NACLs correct? ● Is my Kubernetes cluster stable? ● Are the logging/metrics agent operable? → Make this transparent!
- 75. Shift left Idea DevelopmentProductionStaging Early feedback Late feedback
- 76. Codify your contract execution: -concurrency: 5 hold-for: 20m ramp-up: 5m scenarios: requests: - method: GET url: https://example.com modules: blazemeter: projectName: Team Name testName: Test Name modules: blazemeter: projectName: .* testName: .* Implementation Contract
- 77. Codify your contract BestReally good
- 78. Codify your contract execution: -concurrency: 5 hold-for: 20m ramp-up: 5m scenarios: requests: - method: GET url: https://example.com modules: blazemeter: projectName: Team Name Must specify testName: https://url/to/docs
- 79. $> git cloneexample/template/skeleton Codify your microservices
- 80. Default microservice ● Unittesting ● Integration testing ● Log example ● Metric example ● Secret example ● CI/CD to production ● ...
- 81. How to defineyour contracts ● Be aware of the contract and make the company aware ● Minimize the gray area ● Codify your contract ○ Shift left ○ Examples / blueprints ● Make sure the company understand the interface to your team
- 82.
- 83. AWS Lambda: LanguageSupport
- 84. Scalable support Developer: Hello,can you help me with “something” for feature X? Platform: Did you check the docs for feature X? Developer: Yes, but it doesn’t mention anything about “something” Platform: I see, give me as second Developer: Sure Platform: I edited the docs to include “something”, does this answer your question? https://github.com/org/docs/pull/1337 Developer: Totally, thanks!
- 85. Getting buy-in Team Manifesto Webuild everything with developers. We always select a pioneering team that has interest in a new service/feature, and work with them to make sure we pick and configure the right tool and solve the root problem.
- 86.
- 87. Imagine… a poweroutage You’re at home, watching TV, and BAM: TV is out, lights are out. What do you do? ● Check the other rooms to see the scope of the outage ● Check with the neighbours if it’s just your home or more ● Check with the power company
- 88. How to collaboratewith your users ● Teach everyone how the platform works, in a scalable way ○ Documentation ○ Workshops ● Enable collaboration between development teams ● Make it clear how you work
- 89. Summarizing 1. You needa platform - and someone should own it 2. Always think about scale - both organizational and technical 3. Opinions are OK 4. Specify contracts, but be careful about what you support 5. Collaborate
- 90. Thank you! Questions? Skyworkz -https://skyworkz.nl Sander Knape - https://sanderknape.com - @SanderKnape
Editor's Notes
- #17 I mention it last but it should really be at top. Is all my data encrypted at rest? Is all traffic encrypted? What about patch management?
- #18 And there is more. Networking, storage, distributed tracing, e.t.c.
- #22 It’s expectation management that causes failure. If you underestimate the work required to get a solid platform up and running, you won’t have one and all developers will need longer to work on their applications.
- #23 Components: see previous examples “Easy” is a vague word, but it should be your goal. Keep it easy. Developers are already doing complex things, they don’t need more. Note the word “Operate”: develops should operate their application in production. It’s 2019.
- #24 First of all, it’s efficient. If every developers needs to solve I mentioned earlier, everyone is doing the same work. In fact, it takes specific knowledge. Because: slide with quote Responsibility / ownership. Everything should be owned.
- #26 VPC’s in AWS are not hard, but also not trivial. Every developer can surely pick this up, but it’s not very efficient if every developers needs to understand it to this level.
- #27 In other words: specific knowledge is required. Development teams own logical groups of services: Platform teams owns platform services Ownership is of course important in general. We’ll come back to who owns what later (contracts). Ownership is important, but no different from “normal” (other) applications. Which is the bridge to the next slide: there is no difference.
- #28 Development teams own logical groups of services: Platform teams owns platform services Ownership is of course important in general. We’ll come back to who owns what later (contracts). Ownership is important, but no different from “normal” (other) applications. Which is the bridge to the next slide: there is no difference.
- #29 “Specific knowledge” is also not rare. When a team builds an event sourcing mechanism… they need specific knowledge required for that.
- #30 But I’ll just say “Platform team”
- #34 Lambda was always opinionated. Introducing a new language takes time. And not only new languages: also version updates for existing languages (e.g. Python 2.7, 3.6, 3.7)
- #42 Your best people will leave if they can not pick up work that excites them.
- #43 Operational teams typically use Kanban instead of Scrum/sprints, because of the operational work that constantly comes in and that is also important. This means a lot of context switching, and constant battles about priority. If you work on “projects”, just like other development teams, this will happen way less.
- #44 Provide self-service solutions for toil. The platform team does not have to be involved anymore.
- #45 Provide self-service solutions for toil. The platform team does not have to be involved anymore.
- #46 Also mention how this is the happy path. Dev may also say “I have no time” or “Where are the docs”. You have also raised awareness about documentation. You need to educate people that you work like this. More on education later.
- #47 Last point is bridge to next section. Because “as simple as possible” typically means “opinionated”. Having to make choices leads to complexity.
- #49 Back to the AWS Lambda example. Lambda has always been very opinionated: only a specific set of languages and versions was supported. That changed when they released custom runtimes. Still opinionated, but on a lower abstraction level.
- #54 Going back to the previous point about organizational scalability: less orange scales better. Higher abstractions lead to more opinionated solutions. It means developers can easily use them, but are limited to what they can pick.
- #55 When you lower the abstraction of your platform features, more responsibility goes to the developer. E.g. with Lambda again; Give us code in language X, version Y. High abstraction. Give us an HTTP endpoint. Lower abstraction. Contracts!
- #56 Provide self-service solutions for toil. The platform team does not have to be involved anymore.
- #58 Of course there is an alternative: running it yourself on EC2
- #60 The team that does this themselves may go faster, because they use the tools they are familiar with. However, how much do you lose company-wide because of the fragmentation?
- #61 The team that does this themselves may go faster, because they use the tools they are familiar with. However, how much do you lose company-wide because of the fragmentation?
- #64 Again back to the AWS Lambda example. When they released custom runtimes - and became a bit less opinionated - they changed their contract. Or actually: appended it Before: give us a function in language X, version Y Now: either this ^^ or: provide us an HTTP endpoint that we’ll sent a specific request to
- #65 Contract can be “double”: either specific use of feature, or lower abstraction.
- #66 So a contract says: “this is where the platform ends, and this is where you come in”.
- #69 Same when one microservice is talking to another. Underlying implementation is hidden away. Makes life easier for everyone.
- #70 The (REST) contract to another microservice is typically very clear. Do the same with your platform interfaces.
- #71 The (REST) contract to another microservice is typically very clear. Do the same with your platform interfaces.
- #72 Blazemeter example: if you want to know how to create the YAML, check the blazemeter documentation.
- #82 The way you work together with the platform team is also a “contract”.
- #84 Again, AWS Lambda. If you would ask AWS to add an additional language, do you think they would do it? If you would ask AWS to increase the maximum timeout time - just for you - would they do it? How is that any different from asking your platform team? How do you expect either AWS or your Ops team to prioritize your request?
- #85 This is “helping”. Not only the question they asked, but also that this is the way we provide support.
- #86 The team that does this themselves may go faster, because they use the tools they are familiar with. However, how much do you lose company-wide because of the fragmentation?
- #87 “Collaboration” is also making sure developers collaborate with each other.
- #88 Of course if there is a big production outage, you may want to involve other teams sooner.