DevOps is a software development methodology that combines software development (Dev) with information technology operations (Ops) participating together in the entire service lifecycle, from design through the development process to production support.
- Fast Development Methodologies
- Fast Quality Assurance Methodologies
- Fast Deployment Methodologies
- Faster time to market
- Iteration & Continuous Feedback (strong and continuous communication between stakeholders – the end users and customers, product owners, development, quality assurance, and production engineers)
-
Environment Stabilization
Enforces consistency, increase up-time
-
Shorter Development Cycle
Manage requirements and code-repository
-
Increased Release Velocity
Continuous build, push-button deployments
-
Reduced Defects
Regiment processes, automated testing
-
Process Metrics
Track both time at each stage, and the errors and exceptions
- Plan: task management, schedules
- Code: code development and code review, source code management tools, code merging
- Build: continuous integration tools, version control tools, build status
- Test: continuous testing tools that provide feedback on business risks, determine performance
- Package: artifact repository, application pre-deployment staging
- Release: change management, release approvals, release automation
- Operate: infrastructure installation, infrastructure changes (scalability), infrastructure configuration and management, infrastructure as code tools, capacity planning, capacity & resource management, security check, service deployment, high availability (HA), data recovery, log/backup management, database management
- Monitor: service performance monitoring, log monitoring, end user experience, incident management
-
Step 1: Learn a language
- Begin with HTML & CSS
- Learn basics of the JavaScript
- Understand TypeScript
-
Step 2: Learn about source code management (recommend using Git)
-
Step 3: Practice what you have learnt
-
Step 4: Learn Package Manager
-
Step 5: Learn about Unit Testing
-
Step 6: Learn about databases (RDBMS & NoSQL)
-
Step 7: Learn frameworks
-
Step 8: Learn how to implement caching
- Redis
- Memcached
-
Step 9: Creating RESTful APIs
-
Step 10: Learn about different Authentication/Authorization methods
-
Step 11: Understand Message Brokers
- NSQ
- Kafka
- RabbitMQ
- ZeroMQ
-
Step 12: Learn how to use Docker
-
Step 13: Knowledge of Web Servers
-
Step 14: Learn how to test service
- Test API
- Test Performance
- Test Security
- Test Automation
-
Step 15: Learn different protocols
- TCP/UDP
- Web Socket
- AMQP
- MQTT
- Protocol Buffers
- gRPC
-
Step 16: Learn Microservice, Event Driven, Lambda architectures
-
Step 17: Learn Big Data technologies/tools
-
Step 18: Learn algorithms
Reference: https://github.com/TheAlgorithms/Python
-
Step 19: Learn AI technologies
-
Step 20: Learn configuration/deployment services
-
Step 21: Learn how to monitor services
-
Step 22: Use open source tools
-
Mock REST APIs: https://github.com/typicode/json-server
-
UML Designer: https://www.modelio.org
-
Planning: https://wiki.gnome.org/Apps/Planner
-
MindMap: MindMaple Lite
-
UI Wireframe: Pencil (https://pencil.evolus.vn), Balsamiq
-
SSH clients for Windows: SuperPuTTY
-
-
Step 23: Keep exploring
A collection of LARGE Datasets, so it can NOT be Processed by traditional methods…
[Source]: https://topics.amcham.com.tw/wp-content/uploads/2016/03/BigData_2267x1146_white.png
- Recommendation System
- Clickstream Analysis
- Real-time Analytics
- Sentiment Analysis
- Clustering Analysis
- Search
- Customer Segmentation
- Fraud Detection/Prevention
- Internet of Things
- Image Classification
- Anomaly Detection
- Data usually flows from one application to another. It is produced by one application and used by one or more other applications.
- Generally, the application generating or sending data is referred to as a producer, and the one receiving data is called a consumer.
- A simple way to send data from one application to another is to connect them to each other directly. However, tight coupling between producers and consumers requires them to be run at the same time or to implement a complex buffering mechanism. Therefore, direct connections between producers and consumers does not scale.
[Source]: Apache Kafka
A flexible and scalable solution is to use a message broker or messaging system. Instead of applications connecting directly to each other, they connect to a message broker or a messaging system. This architecture makes it easy to add producers or consumers to a data pipeline.
-
Batch processing is processing a large volume of data at once.
Example processing all the transactions that have been performed by a major ecommerce firm in a week. This data can tune into millions of records for a day which can be stored as a file/records etc. This particular file will undergo processing at the end of the day for various analysis the company wants to do and naturally it will take large amount of time for this to be processed. That would be what Batch processing is. And what Hadoop does is it provides Map Reduce as a processing tool for these large files which can be years of data stored monthly of data etc.
-
Stream processing changes the approach for processing. What it does is it applies the business logic to every transaction that is being recorded for the above ecommerce portal. It applies business logic to every event that is being captured rather than store all the events and then process them, which turns out now that whatever analysis is being done reflects the real state of the data at this instance. So, it becomes real time. And therefore stream processing is also associated with Real time.
| Batch Processing | Stream Processing | |
|---|---|---|
| Data Scope | Queries or processing over all or most of the data in the dataset | Queries or processing over data within a rolling time window, or on just the most recent data record |
| Data Size | Large batches of data | Individual records or micro batches consisting of a few records |
| Performance | Latencies in minutes to hours | Requires latency in the order of seconds or milliseconds |
| Analyses | Complex analytics | Simple response functions, aggregates, and rolling metrics |
-
Machine learning is creating and using models that are learned from data.
-
Machine learning referred to as predictive modeling or data mining.
Examples:
- Spam prediction
- Fraudulent credit card transaction prediction
- A product or advertisement recommendation engine
-
Best Seller in Computer Neural Networks on Amazon 2018: Hands-On Machine Learning with Scikit-Learn and TensorFlow https://github.com/raycad/devops-roadmap/blob/master/books/ai/ml/hands_on_machine_learning_with_scikit_learn_and_tensorflow.pdf
-
System Design Primer: https://github.com/raycad/devops-roadmap/blob/master/books/software-architecture/system_design_primer.pdf
-
Programming Collective Intelligence: https://github.com/raycad/devops-roadmap/blob/master/books/ai/ml/oreilly_programming_collective_intelligence_aug_2007.pdf
-
Azure Cloud Architecture: https://github.com/raycad/devops-roadmap/blob/master/books/software-architecture/azure_cloud_architecture.pdf
https://codeburst.io/the-2018-web-developer-roadmap-826b1b806e8d