Wednesday, May 22, 2019

Terraform

$ terraform
Usage: terraform [--version] [--help] <command> [args]

The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.

Common commands:
    apply              Builds or changes infrastructure
    console            Interactive console for Terraform interpolations
    destroy            Destroy Terraform-managed infrastructure
    fmt                Rewrites config files to canonical format
    get                Download and install modules for the configuration
    graph              Create a visual graph of Terraform resources
    import             Import existing infrastructure into Terraform
    init               Initialize a new or existing Terraform configuration
    output             Read an output from a state file
    plan               Generate and show an execution plan
    providers          Prints a tree of the providers used in the configuration
    push               Upload this Terraform module to Terraform Enterprise to run
    refresh            Update local state file against real resources
    show               Inspect Terraform state or plan
    taint              Manually mark a resource for recreation
    untaint            Manually unmark a resource as tainted
    validate           Validates the Terraform files
    version            Prints the Terraform version
    workspace          Workspace management

All other commands:
    debug              Debug output management (experimental)
    force-unlock       Manually unlock the terraform state
    state              Advanced state management

AWS X-Ray SDK

Installing

The AWS X-Ray SDK for Node.js is compatible with Node.js version 4 and later. The AWS X-Ray SDK for Node.js has been tested with versions 4.x through 12.x of Node.js. There may be issues when running on versions of Node.js newer than 12.x.
The SDK is available from NPM. For local development, install the SDK in your project directory with npm.
npm install aws-xray-sdk
Use the --save option to save the SDK as a dependency in your application's package.json.
npm install aws-xray-sdk --save

Thursday, May 9, 2019

DynamoDB Partitions links


1) Partitions and Data Distribution: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html

2) DynamoDB streams: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

3) DynamoDB streams Boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodbstreams.html

4) Queries with DynamoDB: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html 

Tuesday, May 7, 2019

Kafka vs AWS Kinesis

Apache KafkaAmazon Kinesis
Developed/Hosted ByLinkedInAmazon
SoftwareOpen-SourceProprietary
SDK SupportAWS SDK supports Android, Java, Go, .NETKafka SDK supports Java
Configuration & FeaturesMore control on configuration and better performance.Number of days/shards can only be configured
Data Stored InKafka PartitionKinesis Shard
ReliabilityReplication factor can be configuredKinesis writes synchronously to 3 different machines/data-centers
PerformanceKafka winsKinesis writes each message synchronously to 3 different machines
Configuration StoreApache ZookeeperAmazon DynamoDB
SetupWeeksCouple Of hours
Data RetentionConfigurable7 days at max
Log CompactionSupportedOnly can store logs for 7 days
Processing EventsMore than 1000s of events/secAtmost 1000s of events/sec
CheckpointingOffsets stored in special topicDynamoDB
OrderingPartion levelShard level
Human CostsRequire human support for installing and managing their clusters, and also accounting for requirements such as high availability, durability, and recoveryKinesis is just about pay and use
Producer ThroughputKafka WinsKinesis is bit slower than Kafka
Incident Risk/MaintainenceMore In KafkaAmazon takes care
Ordered sequence of immutable data recordsKafka TopicKinesis Stream
Each record has a unique number calledOffset numberSequence number
ConceptsKafka StreamsKinesis Analytics

Saturday, January 19, 2019

Misc



Think globally and act locally

Categorize and Conquer

Customer centric, business driven arch.

Career + Impact

EAI vs SOA vs ESB

EAI:  Concept of integrating 2 or more business apps

SOA: Style of integrating 2 or more business apps

ESB: Implementation of those integration using tools like IBM mq series, Mulesoft, Apache Camel..

EAI focusses on integration between Enterprise Business Applications, that SOA is a particular way of organizing your IT. And that ESB can be an implementation of both.

Friday, January 11, 2019

EA mistakes


1. Focusing on one domain

The scope of Enterprise Architecture(EA) includes business, application, data and technical architecture. Many architects focus on one domain more than the others. Some Architects become business-oriented and lack insight into technical details. Others become embroiled in low level technical architecture and lack business awareness.

2. Long and complex

Have you ever see a architecture diagram that looks like the schematic for a nuclear submarine ― it is not pretty. The enterprise architecture needs to be widely understood and adopted ― this is impossible if it is overly complex. EA should be short and simple.

3. Working in a bubble

Enterprise architecture is all about breaking down technology silos in favour of common solutions. However, all too often enterprise architects work in a silo themselves. The primary role of the EA Architect is to lead the EA process. The business needs to play an active role in defining the architecture.

4. Governance what?

Governance is probably the most challenging aspect of Enterprise Architecture. It is also the area that is most often neglected. It is critical for EA to be involved in major projects and initiatives. Compliance assessments should be required activities for all projects.

5. The never ending process

Enterprise architecture is a long term planning activity ― but it needs to be relevant in the here and now. An enterprise architect should never lose touch with day-to-day business pressures and challenges.

6. Dogma and silver bullets

Some enterprise architects have preferred solutions they tend to apply to every situation. This often takes the shape of a trendy concept such as SOA.

7. Jargon

Most enterprise architects are familiar with standardized vocabularies for IT such as that provided by ITIL. These standardized terms are great ― if you are talking to other architects. An enterprise architect needs to be a great communicator and part of this is using the words that people understand. If the organization refers to Service Level Management as "Quality Service Management" then that is the terminology the EA team should adopt.

Distributed Computing: A Guide to Comparing Data Between Hive Tables Using Spark

In big data, efficient data comparison is essential for ensuring data integrity and validating data migrations. Apache Spark, with its in-me...