Azure pipelines is a cloud-based continuous integration and deployment (CI/CD) platform to build, test and deploy applications to cloud environments like Azure, AWS, or GCP through Kubernetes Services, Azure Web Apps, and Azure Functions. Azure DevOps pipelines can build container images, push the images to a container registry and deploy these images to Kubernetes environments of our choice.
Setting up a pipeline that leverages a custom container registry or deploys to multi-cloud Kubernetes environments requires cloud-specific expertise and is time-consuming. For instance, building a pipeline from scratch requires:
Containerizing a service – Creating Dockerfile and Kubernetes YAML files
Preparing the agent pool – Agent pools are the environments within which the pipeline scripts are executed, including the container builds. Either Microsoft Hosted Agents or Self-Hosted agents can be used. In either scenario, the agents need to be configured to build and push the Docker Images
Deployment Configuration – Only Azure Kubernetes Service is supported as the target Kubernetes environment.
Introducing gopaddle
gopaddle is a no-code platform that helps to build, deploy and maintain cloud-native applications across hybrid environments. It automatically generates Dockerfiles and Kubernetes YAML files through the intelligent scaffolding process. gopaddle exposes APIs that can be extended and integrated with other pipeline tools like Azure DevOps.
No-Code and Multi-Cloud Capabilities with Azure DevOps
By integrating gopaddle with Azure DevOps pipelines, developers get the no-code automation with the flexibility of deploying to multiple cloud platforms. The following table illustrates the additional capabilities developers get by integrating both solutions:
Azure DevOps (without gopaddle)
Azure DevOps (with gopaddle)
Releases and Distributions
Supports multi-branch triggers with validation.Multi-strategy to build for different OS – supports Linux, MacOS and Windows
gopaddle supports multi-branch, parallel builds across multiple releases and distributions.Customized build strategy for individual release and distribution.
Build Environments
Agent Pools – Microsoft-hosted agents and Self-hosted agents are supported. Agent pools can be Linux, MacOS or Window VMs or Linux/Windows Containers
On-demand: Builds run inside your Kubernetes cluster as isolated build jobs with a specified CPU/Mem capacity.Requires no additional effort. Standardize the environments across build and deployment workflows.Uses a shared Kubernetes environment and can save cost as the environment scales up/down automatically based on build jobs.
Registries
Azure Container Registry, Google Container Registry (via Google Service Accounts), Docker Hub (username/password-based authentication) are supported. For artifacts other than container images, Azure Artifacts or a NuGet repository can be used.
Get additional support for a variety of Docker registries – gopaddle supports Azure Container Registry, AWS Elastic Container Registry, Google Container Registry, Docker Hub private registry, On-premise Container Registry like GitHub Registry.
Get additional support for a variety of source control repositories – gopaddle supports GitHub Cloud & On-prem, GitLab Cloud & On-prem, BitBucket Cloud
Multi-Arch Build
Multi-Arch build requires Linux agents along with a QEMU emulator.
Get support multi-arch builds natively. gopaddle supports – Intel x86_64 – Linux onlyAMD64 – Linux onlyARM64 – Linux onlyMulti-arch – Linux only
Deploy to K8s
Has integrations with Azure Kubernetes Service. Support for other Kubernetes services through service connections (using username/password or token-based authentication) along with scripts to deploy to Kubernetes clusters using cloud-specific APIs or kubectl or kube APIs. In either scenario, it requires scripting to deploy applications to Kubernetes.
Works seamlessly with Azure AKS, Google GKE, AWS EKS, HPE Ezmeral Runtime Enterprise, Huawei Kubernetes Service, MicroK8s, and any CNCF compatible Kubernetes environment.
Workflow
Having explored the advantages of using gopaddle with the Azure DevOps pipeline, let us take a look at the process of integrating these two platforms. We will explore this in the context of a .NET Core application in the GitHub repository. To build a pipeline for this application, we need to initialize the project and create Docker/Kubernetes artifacts like Dockerfile and the Kubernetes YAML files. We can leverage gopaddle’s intelligent scaffolding process to generate these artifacts.
The below workflow explains the step-by-step process of initializing the project.
b. Provision Kubernetes cluster: Follow the steps here to provision a Kubernetes cluster on AWS/Google/Azure. If you are provisioning a GKE cluster, then we need a minimum capacity specified below to build the eShopOnWeb container in this cluster.
Type - N1-Standard-2
Disk - 40GB
Note: If you are using Azure Container Registry and AKS, then you need to register the Azure Cloud Account. Registering an Azure Cloud account requires Single-Sign-On (SSO) to your Azure Account. This requires an OAuth application be to created in your Azure Cloud Account which can authenticate our SSO requests. You can find more information here on creating an OAuth application and registering an Azure Cloud account with gopaddle.
c. Create an Allocation policy in gopaddle UI, with a minimum capacity for building eShopOnWeb container.
CPU Requests - 500 millicore
CPU Limits - 1500 millicore
Memory Requests - 4 G
Memory Limits - 7 G
With the above node size and the allocation capacity, it takes approximately 4-5 mins to generate a Docker image. If you would like to speed up the build process, then you need to increase the node capacity and the allocation policy.
d. Add a container registry to gopaddle by following the steps here. This registry will be used to push/pull the Docker images during the build and the deployment process.
e. Download and install gpctl command-line utility by following the steps here.
Note:During the init process, choose the Kubernetes cluster to build and deploy the application and choose the Docker registry to use for pushing the image.
Once the gpctl init completes, gopaddle creates a .gp file with a list of resource IDs that can be used to build the pipeline.
Creating Azure DevOps Pipeline
Now that we have initialized the project, it is time for us to create a pipeline in Azure DevOps.
From the Azure DevOps subscription, let us perform these steps.
a. Create a new pipeline.
b. Select the forked Github repository and the branch. Once you select the repo and the branch, you will be redirected to the Github sign-in page (or) if have you already logged in to GitHub, you will see the list of repositories in your Github account. You can choose the repository and authenticate the request from Azure DevOps.
c. Create an Azure DevOps Pipeline of type ‘Stater pipeline’.
Once the pipeline is created, a pipeline script file named azure-pipelines.yml gets added to the project root folder in the GitHub repository.
d. Edit this pipeline script and modify the contents based on the template script from here.
e. Create a secure variable to access the gopaddle API token by creating a variable group by name gp-api-key under the Library option in the Azure DevOps portal. Store the gopaddle API token under this group with the variable key as GP_API_TOKEN and the value as the API token from the gpctl init output.
f. Replace the contents of the variables section in the pipeline script with the values from the .gp file generated during the gpctl init process.
A Closer Look at the Pipeline Script
The pipeline has 2 stages – Build the application and Perform a rolling update once the build completes. But this pipeline can be extended to include more steps like running a regression suite or sending an email to the project stakeholders.
As soon as the code is committed to the project, Azure DevOps detects a change and triggers the pipeline. When the pipeline script is executed, it calls the gopaddle API to trigger a container build. The container build process builds the container image in the pre-configured Kubernetes cluster, pushes the Docker image to the container registry. The pipeline script waits until the build process is complete. The second script, picks the build information from the previous step – like the build ID, commit description, etc and initiates a rolling update using the gopaddle API.
Demo
You can find a brief demo of the pipeline integration:
As illustrated, the modernization process takes about 10 minutes (most of this time is spent on building the container image and deploying the application) to generate the Dockerfile and the Kubernetes YAML files and about 10 minutes to create the pipeline.
Through a simple integration, we get the benefit of modernizing the application and creating an end-to-end CI/CD pipeline by integrating gopaddle and Azure DevOps pipelines.
About The Author
Founder & CEO @ gopaddleio. Talks about distributed systems and composite architectures.
Gone are the days when ideation to productization took years. Between 2011 and 2016, the cloud and DevOps showed the path to versatility and automation. However, every cloud offering had a unique interface. Multicloud was a wishful strategy, as automating operations even for a single cloud platform was a daunting task due to lack of standardization.
When Kubernetes came to light, it offered a standardized interface across hybrid environments. Automation tools converged, multicloud became a reality. However, Kubernetes implementation required a steep learning curve across the teams. According to D2IQ, 41% of the developers and architects felt extremely burned out and 51% of them felt that application modernization (aka cloud native apps) added stress to their day-to-day job. Complexity has been an impeding factor.
In the last 1.5 years, the impact of COVID-19 has added a new dimension to this equation. Businesses now have to deliver more with less, adding pressure on the team to upskill faster. While the demands are increasing on one end, there is a shortage of skilled developers on the other. Business and Product owners are aggressively trying to optimize their software delivery process. This has given rise to the next-gen development paradigm called “citizen development” in the cloud native space.
What Is Citizen Development?
Citizen development isn’t a completely new phenomenon. It has been around for a while in other aspects of software development, especially in building a business process modeling (BPM), customer relation management (CRM), or an enterprise resource planning (ERP) application. Citizen development leverages no or low code platforms to build applications through visual representation without the need for deep coding skills. These platforms have proven to drastically increase productivity and accelerate the time to market.
Low-code no-code software development helps developers make cloud-native applications more than 10-times faster (with 70% fewer resources)
Statistics augmenting the rise of no-code platforms aren’t surprising. Humans process visuals much faster than text representation. Thus, visuals are not only easy to comprehend but also create shared knowledge across technical and non-technical teams. This enables the teams to focus on high-value tasks than spending time on resolving technical issues. Extending this paradigm from application development to delivery and automation, can empower the entire organization, right from business/product owners to developers, to own software delivery and cut down any communication barriers.
The Fabric of No Code Platforms
Before we dig deep into no-code platforms for cloud-native deployments, let us analyze some fundamental characteristics of these platforms.
Visual Models instead of code – The fundamental building blocks of these no-code platforms are the visual modeling tools. They use visual models to auto-generate data models and workflows.
Model-Driven Approach – Since the development starts and ends with visual models, no-code platforms inherently use a model-driven approach to building applications.
Guard Rails – If no-code platforms are required to generate code, then it is imperative that they encapsulate validation and security checks to ensure the quality of the code – preventing manual errors and oversights.
Reusability and Extendibility – One of the fundamental characteristics of no-code platforms is to offer or enable reusable modules that can be composed dynamically. These platforms often come with marketplaces or product hubs, each with a rich set of reusable components and third-party tool integrations.
Change management – Since the visual models act as the single source of truth across the business, they need to be version controlled for better traceability and compliance.
Governance – These platforms encourage collaborative development, Role-Based Access Controls, centralized management of resources to avoid shadow IT.
What Is a No-Code Platform for Cloud Native Deployments?
A no-code cloud native platform takes a code-free approach to transform and maintain micro-services as cloud native deployments. During the transformation process, these platforms auto-generate Docker files, Kubernetes artifacts, and the pipeline code for end-to-end automation.
Drawing parallels between no-code platforms in the BPM domain to those in cloud native deployments, most of the characteristics of no-code platforms in the BPM application development are still applicable in the cloud native context. For example, centralized governance across the resources can reduce the service or cluster sprawl and can give a bird’s eye view of the entire DevOps process. However, a few additional factors are paramount to the success of a no-code cloud native platform.
Scaffolding – This provides a seamless experience to onboard a microservice.
Sophisticated Modeling – visual modeling has to go beyond stacking up the LEGO blocks of microservices, it has to cover a wide range of application platforms, Kubernetes resources, and the underlying hybrid infrastructure.
Distribute 12 factors across different stages of the release cycles – This enforces best practices at different stages of building the applications so that the applications are cloud native compliant from day 1.
Simplified hybrid cloud management – a simple interface either through UI wizards or command-line utility to provision and manage Kubernetes environments across cloud, edge, and on-premise environments.
No Lock-in – While the dominant purpose of Kubernetes is standardization and portability, no-code platforms should embrace these core principles and deliver portable artifacts.
Developer tools and Out-of-box experience – Unlike traditional citizen development, since the cloud native no-code platforms deal with a multitude of 3rd party tools, hybrid environments, and loosely coupled microservices, it is essential to provide a developer-friendly environment to build applications. This includes logs, metrics, direct terminal access to an environment, etc
Coding is not the main event anymore. Building software is the main event. Coding is just one small part of it. We think the future of coding is no coding at all. we think autonomous coding is a very real thing.
Citizen development is not just for novice developers. It automates laborious and mundane tasks making room for smarter software development and innovation. In this new age, where the need for speed is an unquestionable differentiator for businesses, no Code platforms are a big boon to cloud native development. Let us enrich the experience of building software on Kubernetes, be it on the cloud or at the edge. Together, let us enter the realm of hyper-automation.
This new year brings a great relief to the Cloud Native Developers. Today we are pleased to announce gopaddle v4.0 release. This is a major release from gopaddle as we have extended the platform to address the developer needs right from the microservice transition to maintenance, in a simple, yet powerful fashion.
Challenges today
Refactoring Applications: Most often the barrier to modernization is refactoring the existing infrastructure oriented and monolithic application to Cloud Native architecture. Every application is unique and the team dynamics in every organizaton is unique. Thus the strategy to refactor the application has to be crafted to suite the application and the organization.
Cultural Shift : Once the strategy is streamlined, developers need to acquire new skills to build & package their modules as Containers & deploy their modules on Kubernetes.
Maintaining YAML files & Separation of Concern : One of the biggest challenges in modernization today is to write and maintain thousands of lines of YAML code.When it comes to maintaining these YAML files, GitOps is fairly a popular methodology to centrally manage these Kubernetes YAML files, but is inadequate for large distributed teams where DevOps responsibilities are shared across the teams.
Dependency Management : When the number services increase, and when the deployments need to be maintained for a longer period of time, dependency management and change tracking becomes even more critical. Making a change to a service can result in a ripple effect and can lead to application downtime.
Service/Cluster Sprawl : Similarly, when the number of services and environments increase, it can lead to service or a cluster sprawl.
How gopaddle helps ?
How gopaddle helps in Cloud Native Transformation ?
gopaddle is a Do-It-Yourself (DIY) platform to build, deploy and maintain Cloud Native Applications. With this new release, we have introduced tools and features that not only reduce the barriers to entry, but also ease the Day-2 operation and maintenance.
Use Kubernetes environments to build Docker Images
Convert Source Code to Docker Images without writing Dockerfiles
Propeller : Kubernetes Deployment Service
No Code/YAML Free Service to build and deploy complex Kubernetes Deployments
Create reusable Application templates
Centrally Manage Applications across different Kubernetes Environments, ideal for Edge deployments
Simplifies Day 2 Operations with the production grade capabilities built in to the system – like Ingress, LoadBalancer, Domain & TLS support, Network Access control , Logging, Monitoring, Alerting etc.
Gear : Multi/Hybrid-Cloud Kubernetes Management Service
Provision managed Kubernetes environments across popular Cloud Platforms like AWS, Google & Azure
Use pre-existing on-premise Kubernetes Clusters for build and deployment
Capabilities & Use cases
gopaddle capabilities
With this new release, you can perform :
Speed up the moderization effort as the Docker and Kubernetes artifacts are automatically generated.
Automate end to end Build and Release automation including CI/CD automation
Multi-Cloud and Edge management from a central dashboard.
Reduce the risk of misconfiguration and/or overlooking YAML changes with the help of granular version control for Kubernetes artifacts.
We are committed to making this platform Developer friendly and to removing any frictions or obstacles in modernizing the applications.
Apart from the product capabilities, we offer Cloud Native Transformation Consulting to analyse your application architecture and suggest a transformation strategy.
Do you want to schedule a call with us to see if gopaddle works for your application stack ? Contact us today !
About The Author
Founder & CEO @ gopaddleio. Talks about distributed systems and composite architectures.
In order to comply with the Payment Card Industry Data Security Standard (PCI DSS), application access, transaction and system logs need to be monitored. Fluentd is an open source, distributed logging and aggregation framework. It offers a unified logging layer to collate & filter logs of different nature like access logs, application logs and system logs. Fluentd also helps to aggregate, analyse and archive these logs. Kubernetes uses the glog library module for logging Fluentd for application monitoring.
For application (microservice) log monitoring, a sidecar container is suggested pattern for deploying Fluentd within the same pod where the application lives. A shared volume between the application container and the Fluentd container holds the log files for the containers. Fluentd uses the shared volume as the data input and the Google Cloud Logging Service as the data output, provided the cluster is on Google Compute and the VM has cloud-logging.write scope.
The Daemon Set feature, which went GA in Kubernetes 1.2, allows you to deploy Fluentd as a pod across all the nodes in your cluster, instead of deploying them specific to an application.
Technical Implementation
Daemon Set is an extended feature available in Kubernetes 1.2 beta. Some of the typical use cases of Daemon Sets are log aggregation, monitoring services that need to be run on a selected set of nodes within a cluster. Daemon Sets are headless services and are very different from replication controllers. Replication controllers are stateless services that can scaled up or down and can be scheduled across different nodes, whereas Daemon Sets help in performing specific tasks within selected nodes.
Benefits of Daemon Sets
Daemon Sets can be deployed the same way as deploying an application on Kubernetes
Daemon Sets can be deployed on a selected set of nodes with specific tags. Whenever a new node is added with a similar tag, Daemons get deployed automatically.
Daemon Sets with different resource requirements like CPU, memory & storage can be grouped and deployed on selected nodes
Running Daemon Sets as separate PODs & containers within a node gives better resource isolation rather than running them as simple daemons within the node
Fluentd Use Case
Let us consider a use case with a 2-tier application deployment using goPaddle. The application design composes of ngnix and tomcat services. Using goPaddle, each of these services can be deployed as scalable micro-services using POD per container architecture shown below. Each service will have its own replication controller and can scale independently.
The services are deployed across a 3 node Kubernetes 1.2 cluster.
Fluentd is available as an add-on component in goPaddle designer. It can be configured to accept log folder(s) and write logs to an external MongoDB service. Fluentd can be created as a single service design and can be deployed the same way as any other service. At the time of deployment, goPaddle provides an option to select the nodes where it needs to be deployed.
In all the selected nodes, Fluentd gets deployed as a Daemon Set and gets configured automatically to collect the logs from the specified folders. goPaddle deploys the services and the daemons using the Sidecar Pods pattern (similar to Sidecar containers pattern) where the pods share the volumes for log read/write.
Best practices
As a best practice, if the application loggers are configured to prepend the host IPs, then the aggregated logs would be prepended with the Pod IPs which can later be used to filter the logs from specific service instance.
Benefits:
By deploying Fluentd as a Daemon Set using goPaddle,
It can be customized to monitor specific or all applications running within the same node
It can collect logs from specific log files across all applications within the same node
It can collect system logs across all the containers within the same node
Since they are run as a separate pod, they run outside the application scaling rules
Use custom log data outputs other than Google Cloud Logging
Since it is run as a Daemon Set, it is ensured that is always in running state
Note: Using goPaddle, it is also possible to deploy a Daemon Set in ambassador pod pattern (similar to ambassador container pattern) where the application Pod can communicate with the Daemon Set pod using localhost reference. A typical use case would be to deploy etcd services within a Pod and the applications can access etcd for key/value store and retrieval.
About The Author
Founder & CEO @ gopaddleio. Talks about distributed systems and composite architectures.
2016 being the year of DevOps, IT organizations are already looking at scale-out and economies of scale, while e-commerce companies have understood the impact of agile delivery and zero downtime of their applications. As Gartner predicted, it is becoming a mainstream strategy employed by 25 per cent of Global 2000 organisations. While the need for DevOps is building up, micro-services based architecture is becoming the key enabler for rapid application delivery. Micro-services are sizable units of execution that are independent, polyglot and reusable. Independent teams / developers can make versions of these micro-services, package them as container such as Docker and deploy them on an infrastructure of their choice. Container Clusters like Kubernetes provide a distributed and portable platform that can manage the lifecycle of these containers and provide consistent environments so that applications can be moved painlessly across datacenter and clouds.
While a spectrum of DevOps ready solutions are emerging in the market, a platform that is focused on continuous build, integration and deployment for micro-services will be a huge cost & time saving solution for businesses.
Customers have been using goPaddle, for continuous build and deployments since last year. With the new release, goPaddle is optimized for Google Cloud, thus making authentication, on-boarding micro-services and cluster provisioning simple and seamless for Google Cloud Developers. goPaddle provides a powerful yet simple designer interface, that developers can use to drag drop components to create micro-service designs and generate container/cluster specifications on the fly. These designs can then be deployed on to Google Container Engine (GKE) or to a cluster of their choice. Developers can leverage Google Container Registry (GCR) or use Docker Hub for storing/retrieving the container images. As these features signify, goPaddle has moved the infrastructure management hassles away from the developers, while the developers can focus on building their applications.
While adding simplicity, goPaddle also gives the traceability and control to audit and monitor the applications. In general, log aggregation platforms like fluentd have greatly been in use by the developers and system administrators to monitor the system, application and access logs. goPaddle v2.0 brings in the support for fluentd leveraging the newest features in Kubernetes v1.2. With goPaddle, fluentd can be deployed on selective cluster infrastructure to aggregate logs from micro-services running within those infrastructure.
gopaddle is available as a hosted version at https://gopaddle.io and as enterprise ready on-premise version with extended support. For more information, get in touch with the Bluemeric team at info@bluemeric.com.
About The Author
Founder & CEO @ gopaddleio. Talks about distributed systems and composite architectures.
Bluemeric Technologies, an India-based start-up with focus on DevOps solution and services, announced the launch of it’s cutting edge SDLC 3.0 platform, goPaddle.
Cloud computing has started making a profound impact on the application development. Mobile and web based application development companies have significantly benefited by developing and rolling out their applications on clouds. But application downtimes and development process can delay the lab to market cycle. 80% of downtime of mission critical applications is caused by miscommunications about the release change. Similarly 45 to 65 percent of the issues are introduced during the requirements and design phase of the software development. goPaddle addresses these two problems through improved collaboration between the teams and helps to identify issues early in the release cycle through container based shift left approach. Due platform neutrality, goPaddle offers Developers and IT teams the flexibility to use different operating systems and different cloud platforms across the software release cycle.
“Container technology adds a new dimension of portability in Software release cycles. Development and IT teams now have a choice of using either Infrastructure or Platform Clouds across the release cycle without worrying about application portability issues. With goPaddle, companies can take advantage of light weight and portable software release management on cloud,” says Vinothini Raju, Founder & CEO at Bluemeric.
While model-driven PaaS and container adoption is on the raise, most of the applications still run on Infrastructure clouds compared to platform clouds. A solution like goPaddle connects these technologies with the existing application deployments. This announcement came following Red Hat’s OpenShift Enterprise 3 announcement last week. goPaddle currently integrates with Open source version of Openshift v3 PaaS, Kubernetes and with other infrastructure clouds like AWS and Google Cloud.
Using goPaddle, cloud based application development companies can now speed up their software development process. They can do forward and reverse engineering of a multi-tier application in a similar fashion. In addition to this agile application delivery, they can also achieve application portability across different cloud platforms.
“We have been closely following the technology trends in the portability space and we have found Docker to be a game changer in the container management space. We wanted to contribute to the Docker adoption and have also released open source tools like DockYard to manage Docker containers. Based on our DevOps consulting experiences, we have seen how Docker can simplify application release management. Our technical expertise in Docker and clustering platforms like Kubernetes has helped us develop a solution like goPaddle”, says Giragadurai Vallirajan, CTO at Bluemeric.
goPaddle is available as a closed source SaaS application and is currently in public beta. It is available to customers as a free subscription with limited licenses for the next 3 months.
Based in India, Bluemeric has establishment in US as well to have more closer interaction with the Developer community in US, bridging the container technology space between India and US.
About The Author
Founder & CEO @ gopaddleio. Talks about distributed systems and composite architectures.
