Hello people, Let's talk about a trending topic in the market - Azure DevOps.

Azure is a cloud solution that provides lots of services and features to help your organization in almost all ways during the development of any product by providing multiple tools and techniques.

Azure DevOps is a feature or service of Azure that is used to plan smarter, collaborate better, and ship faster. We can use all of them or choose from them as per your need and role. Azure DevOps is developed and managed by Microsoft which makes it a secure, fast, and reliable product for DevOps.

Let's have a glimpse of service provided by Azure DevOps :

Azure DevOps gives you a visual environment to collaborate with your team to create an application. The board helps you with the agile piece of work. With boards, it becomes easy to track tasks, features, and bugs associated with the project. It has built-in scrum boards and planning tools to help your team run sprints, standups, and planning meetings.

A place that contains all the source control tools that are used to save your work and coordinate code changes across the team. 

All the tasks including the automation process, CICD will be available in the pipeline. Though Azure DevOps is a Microsoft product, you can configure pipelines for applications that are developed using any language and platform. Also, Azure pipeline lets you deploy your application into Azure app servers thereby making integration with azure services seamless.

As testing is one of the main features of the development, automation testing, manual testing, and all other tests are taken care of under the test plans.

Artifacts are basically used to store packages you get from the private or the public repositories that are required for the deployment process.

Lots of companies do not want to use the cloud because they do not want to store their data on some other server. They want to store data on their own server. 

Let's understand what is azure DevOps !!..

Shifting your mindset towards Cloud - Azure and Azure DevOps

Docker Client: The Docker client (docker) is the primary way that many Docker users interact with Docker. When you use commands such as docker run, the client sends these commands to dockerd, which carries them out. The docker command uses the Docker API. The Docker client can communicate with more than one daemon.

Docker Daemon :  The Docker daemon (dockerd) listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. A daemon can also communicate with other daemons to manage Docker services 

Docker Registry   :A Docker registry stores Docker images. Docker Hub and Docker Cloud are public registries that anyone can use, and Docker is configured to look for images on Docker Hub by default. You can even run your own private registry.

1) End users writing docker commands in docker client

2)Daemon will take the request from client for processing 

3a)Daemon will verify if specified images is available in host machine 

3b)If images is not available then daemon will downloaded Jenkins image from Hub ( internally run will use pull )

4) Daemon will run specified images inside the container 

5) Daemon will send out to client for  container status 

for more information please refer  https://www.youtube.com/watch?v=u8RN55KiiSU&t=238s 

Understanding about Docker components and Architecture

Docker Components and Architecture

 Operational Costs :  Key components of operational costs are unit price of infrastructure, the ability to match supply and demand, finding a pathway to optionality, employing an elastic cost base, and transparency  

 Productivity  :  getting up and ready in seconds and various service availability. 

 Cost Avoidance  : eliminating the need for hardware refresh programs and constant maintenance programs 

 Operational Resilience :   increases resilience and thereby reducing organization’s risk profile

 Agility  :  react to market conditions more quickly 

 PROJECT :   It execute projects to get familiar and experience benefits from the cloud. 

 FOUNDATION :  This includes creating a landing zone (a pre-configured, secure, multi-account AWS environment), Cloud Center of Excellence (CCoE), operations model, as well as assuring security and compliance readiness 

 MIGRATION   Migrate existing applications including mission-critical applications or entire data centers to the cloud as you scale your adoption across a growing portion of the IT portfolio .

 REINVENTION :  Focus on reinvention by taking advantage of the flexibility and capabilities of AWS to transform business by speeding time to market and increasing the attention on innovation. 

Migration Preparation and Business Planning

 Starts with some foundational experience and developing a preliminary business case for a migration, which requires taking objectives into account, along with the age and architecture of the existing applications, and their constraints.

  Understand the IT portfolio, the dependencies between applications, and begin to consider what types of migration strategies needed to meet the business case objectives.

 Designing, Migrating, and Validating Application

 AWS provides migration services that help for moving applications and data from on-premises to AWS – AWS Server Migration Service (SMS), AWS Database Migration Service (DMS)

 Once applications are migrated, iterate on the new foundation, turn off old systems, and constantly iterate toward a modern operating model.

 Migration strategies depend upon what is in your environment and the what is suitable for the portfolio, taking into account the business and technical requirements.

 provides better opportunity for re-architect the applications once they are already running in cloud, with the organization having already developed cloud skills and the application with its data is migrated and handling traffic.

 Moving your application to the Cloud with optimizations, without any major changes .  Replatform helps achieve some tangible benefit without changing the core architecture of the application. For e.g., using RDS for database or Elastic Beanstalk for applications.

 Move to a different product, which likely means the organization is willing to change the existing used licensing model

 Driven by a strong business need to add features, scale, or performance with agility and improvement in business continuity that would otherwise be difficult to achieve in the application’s existing environment 

 Identifying IT assets that are no longer useful and can be turned off will help boost your business case and direct your attention towards maintaining the resources that are widely used.

 Keep the applications as is in the current environment. Retain portions of the IT portfolio, which have tight dependencies, difficult, not in priority or ready for migration

Cloud Migration process from On-premises to Cloud.  

AWS Cloud Migration

 We started to design and build a system that imports and administrates business data in the main business region, then ships the data as a suite of microservice API endpoints and batch file feeds to the global markets.

The database choice was PostgreSQL and the whole infrastructure is on AWS. Since Aurora PostgreSQL didn’t come into being until right before the project started, the database solution was designed to use RDS PostgreSQL with the same- and cross-region replicas.

In the main market’s AWS region, the master instance serves to write traffic coming from importing upstream data and admin applications, one replica serves in-region API read traffic, while another replica serves all regions’ batch read traffic. In each overseas market, one replica was created to serve that market’s API read traffic.

An architectural level change to address the database replica limit is unavoidable. With the fourth overseas market’s rollout date approaching, the fifth, sixth and the seventh overseas market also had their rollout plan laid out.

However, we must tread very carefully. The system has been running on production for nearly 2.5 years, now serving eight client systems from seven countries. The daily traffic could amount to 20 million accesses, any downtime has impact on real users, any read downtime could result in business loss. 

For the first requirement, AWS Aurora Global Database is a low-hanging fruit solution. It supports cross-region read-only replicas in up to five AWS regions (secondary regions as named in AWS docs, as opposed to the primary region where the master database instance resides,) with less than one second cross-region replication latency. As for the number of replicas, it will allow us 15 replicas in the primary region and 16 in each secondary region . Plus the scalability and availability improvement (think about cross-region failover), migrating to Aurora Global Database is a no-brainer.

Geographically it is easy to group the seven (and more to come in the future) overseas markets into less than five AWS regions. Once we have an Aurora database cluster, adding the first secondary AWS region will automatically turn it into an Aurora Global Database. 

The only problem is, our PostgreSQL database was still on version 9.6.11. Although the lowest Aurora compatible PostgreSQL is 9.6.8. Aurora Global Database is only compatible with PostgreSQL 10.11, 10.12, and 11.7 up.

 For some more buffer time before the next major version upgrade, we chose the newest Aurora compatible PostgreSQL version in our target AWS regions, 11.8, as the upgrade target version. 

 With AWS CLI, the following command checks available engine versions of Aurora PostgreSQL in a particular AWS region:

 $aws rds describe-db-engine-versions --engine aurora-postgresql --query '*[].[EngineVersion]' --output text --region <your-aws-region> 

 Also with AWS CLI, the following commands check available upgrade target versions from a given PostgreSQL version in a given AWS region :

 First, export the AWS region and that region’s RDS endpoint as the terminal tool’s environment variables: 

$export ENDPOINT=https://rds.<your-aws-region>.amazonaws.com

Then, to check available upgrade target versions for RDS PostgreSQL from a given version, use this:

 $aws rds describe-db-engine-versions \
--engine postgres \
--region $REGION \
--endpoint $ENDPOINT \
--output text \
--query '*[].ValidUpgradeTarget[?IsMajorVersionUpgrade==`true`].{EngineVersion:EngineVersion}' \
--engine-version <your-start-version>

 To check available upgrade target versions for Aurora PostgreSQL from a given version, use this: 

--query 'DBEngineVersions[].ValidUpgradeTarget[?IsMajorVersionUpgrade == `true`]'

This is how we found out that it is impossible to upgrade 9.6.11 directly to 11.8, no matter in RDS or Aurora. We have to first upgrade to 10.x, then 11.8.This is how we found out that it is impossible to upgrade 9.6.11 directly to 11.8, no matter in RDS or Aurora. We have to first upgrade to 10.x, then 11.8.

To achieve the least user impact, especially to avoid potential business loss, we set the goal of zero read downtime, and minimizing write downtime as much as possible.

According to AWS doc, there are two ways to migrate from RDS to Aurora:

Migrate from an RDS PostgreSQL DB snapshot, or

Create an Aurora read replica from an RDS instance, then promote it to be an independent Aurora cluster

Both options can achieve zero read downtime, since the existing RDS DB can keep serving the read traffic until the Aurora cluster is ready, then we just switch the applications’ DB connections to the new Aurora cluster.

As for the write downtime, at first glance, the second option looks more appealing.

Option one requires us to stop all DB write operations (importers, admin applications) as soon as we start to create the RDS snapshot, otherwise we would lose all the changes after this point in the new Aurora cluster.

In contrast, option two allows us to keep accepting writes even after the migration starts. The Aurora read replica will replicate all changes from the RDS master after it is created. We only need to stop write operations at this point, wait for it to catch up with the master, and promote it to be an independent Aurora cluster. Then we can switch all read and write applications to the new Aurora DB, then resume the write operations.

However, the AWS doc failed to mention one key fact, which we had to find out only in our first staging rehearsal. In our use case, this rendered option two essentially impossible:

If the RDS master instance has cross-region replicas, no Aurora read replica can be created.

In the end, we had to go with option one, despite the longer write downtime.

After migrating on the staging environment, we noticed severe write performance degradation in the new Aurora DB. Compared with using RDS, importing the same amount of data from upstream now takes ten times longer.

Run ANALYZE to refresh the pg_statistic table (which was recommended in AWS doc for PostgreSQL major version upgrade, both RDS and Aurora)

Tune the Aurora parameter group according to our data’s characteristics (mainly some Autovacuum related parameters needed to be adjusted)

Our migration procedure turned out like this:

Stop all DB writing applications/batch jobs

Create Aurora cluster from the RDS snapshot

Upgrade the Aurora cluster from 9.6.11 to 10.11

Upgrade the Aurora cluster from 10.11 to 11.8

Connect test applications to the new DB clusters

In this article, we know how to migrate PostgreSQl on AWS from RDS to Aurora..

Migrate PostgreSQL on AWS from RDS to Aurora

I am always faced with the term "cloud platform" and don't understand how a cloud platform differs from "a normal website" as a cloud platform still is a content delivered by the internet to the client computer typically through the web browser!

Microsoft 365, Slack are some examples of a cloud platform, but still i can use slack like any other website, so why i can't simply tell it is " a website"

1 - Can i call a cloud platform also a "website"?

2 - How a cloud platform differ from a normal website?

What is the real difference between a cloud platform and a normal website?

Consider two or more applications "talking" to each other and deployed to the cloud (cloud foundry). What are the best practices for a team to follow to work (develop/test/debug) on the same instance of the applications but in his/her "own" space without creating another instance of the application in the cloud? Or should every developer need to have a local copy of those application and run it in a docker/kubernetes, for example? 

How a team can work on the same cloud application

i am a newbie to cloud computing and studying for Comptia CLOUD+ Exam. I am confused between Vendor and Provider. it says in the book that vendor sells solutions to customer.I am not able to figure out like what solutions.. how is it different from providers?  

cloud

Trending Blogposts on cloud

Shifting your mindset towards Cloud - Azure and Azure DevOps

Docker Components and Architecture

AWS Cloud Migration

Migrate PostgreSQL on AWS from RDS to Aurora

Popular Discussions on cloud

openWhat is the real difference between a cloud platform and a normal website?

1

0

0

openHow a team can work on the same cloud application

2

0

0

openVendor vs Provider confusion

1

0

0

PRODUCTS

QUICK LINKS

LEGAL

cloud

Trending Blogposts on cloud

Shifting your mindset towards Cloud - Azure and Azure DevOps

Docker Components and Architecture

AWS Cloud Migration

Migrate PostgreSQL on AWS from RDS to Aurora

Popular Discussions on cloud

openWhat is the real difference between a cloud platform and a normal website?

1

0

0

openHow a team can work on the same cloud application

2

0

0

openVendor vs Provider confusion

1

0

0