Version: DRAFT

Overview

At the current time Gestalt is in beta. For access to the beta repositories please contact sales@galacticfog.com. The private repos will be made public near the end of june. Our target date is June 29th, 2016.

Gestalt Framework

This document provides and overview of Galactic Fog’s Gestalt Framework including dependencies, deployment patterns, and current adapter road map.

Components

The Gestalt Framework is comprised of three major components:

• Infrastructure Layer – A generalized hosting layer that runs both the gestalt services and any client applications in a scalable and resilient major across bare meta, virtualized, and cloud environments.

• Integration Layer – The Gestalt Integration Layer is a set of micro services that abstraction integration with most enterprise systems in a consistent and governable way.

• Meta Layer – Acts as the central repo of configuration and policy for the infrastructure and Integration layers.

Infrastructure Layer

The Gestalt Framework’s infrastructure layer is used to host the gestalt integration services as well as client applications. The infrastructure layer is designed to make everything that runs inside it scalable and resilient.

The way the infrastructure layer achieves the goals of resiliency is by using Apache Mesos to ensure the applications have the required number of application services running at all times. An application service can be a simple process, or a docker container. If an application service dies it is restarted automatically. If the server that the application service is running on dies, the service is restarted on a different server. A distributed file system is used to ensure that when the service is restarted on another node it has access to all the state information required to resume functioning.

Scalability is achieved by allowing easy scale up and load balancing of all application services. If additional instances of an application service are required, then the scale parameter is increased and additional nodes are started on the compute cluster. As those nodes come up they are put behind a load balancer automatically.

Note: Gestalt will not make unscalable technologies magically scalable.

Gestalt Infrastructure Node Requirements

The infrastructure layer can be installed onto RHEL/CENTOS 6.X & 7.X. The 7.X series is recommended for both performance reasons and future support reasons.

The minimum recommended instance sizing is:

Cloud Size: For Amazon, an m4.xlarge with an additional data device meets these requirements.

Note: These are the recommended minimums. More CPU/MEMORY allows for higher density and efficiency.

Infrastructure Node Components

The following software is installed on the infrastructure nodes:

• Apache ZooKeeper

• Apache Mesos

• HAProxy

• Docker

• Nagios

• Java 8

• Scala 2.11

• SBT

• Marathon

• Distributed File System.

Infrastructure Deployment Patterns

While all the infrastructure nodes are identical in terms of the software installation there are still specific topologies required to achieve specific behavior. High Availability is achieved with a cluster of at least 3 nodes in the same data center. The infrastructure layer supports cross data center deployment but requires a separate cluster in each data center. You can not run them as a single cluster across multiple datacenters as there can be unforeseen performance penalties. For systems that will have some number of nodes with direct internet access we recommend at least 5 nodes, with two of the nodes being in the DMZ. This is for isolation of the management infrastructure.

Deployment Patterns

Appliance VM (DEVELOPMENT USE ONLY)

The appliance vm is a single node that works with no HA. It does not use mesos, zookeeper, etc. This vm is useful when you wish to test applications with the gestalt framework. It consists of a virtualbox VM with all the gestalt micro-services loaded into it as docker containers. You can load your own application in and start it using standard docker semantics. The vm consumes about 6-8gb ram, before client applications are loaded onto it, so resources can be tight on a standard laptop with only 16gb ram.

HA Cluster

The most common use case is the HA cluster. An HA cluster is comprised of a minimum of 3 nodes.

If the application is going to be internet facing the recommendation is to use at least 5 nodes, where two of the nodes sit in the DMZ. This isolates the management interfaces. This is just best practice, not a hard requirement, as it is possible to lock down and secure the management interfaces, it just takes additional work.

Capacity Requirements: The infrastructure + gestalt services consume about 25 GB ram and 4 cores in a minimum configuration of 3 nodes leaving about 20gb ram free and 8 cores free for client processes/containers. Additional instances added to the cluster lose only about 20% to overhead.

Distributed File System

Each node runs a server and client process for the distributed file system. The distributed file system is used so that when an application container dies it can be restarted on any infrastructure node without losing its state. The distributed file system is comprised of a server and a client. Either of which can be run independently of the other. The server is used for creating and adding storage to the pool. The client is used to mount a fuse based file system that connects to the server instances. A minimum of 3 nodes is required for the DFS to tolerate failure.

The primary requirement for running the server is that a separate data device must exist on the server that will be contributing storage. That device must be at least 15gb in size. The device can be local storage, cloud storage (eg, Amazon EBS), San, or iscsi. It is possible to use the ephemeral storage on ec2, but this is not recommended as complete failure in an availability zone is possible.

HA Cluster Networking Configuration

The table below lists the ports that need to be open between all nodes to facilitate communications. Unless otherwise stated these ports should only be open to other nodes of the cluster or to the management network

DR Cluster

A common requirement is for a DR cluster to be enabled to support failing applications across datacenters in the event of a failure.

The gestalt deployment model for this requirement is consists of two HA Clusters, each in a different data center.

The only networking requirements between them is that the Launcher needs access to both marathon services, the Gestalt Meta DB Service (port 5432) must be open between both clusters, and the Distributed File System need to be able to communicate between the datacenters.

Gestalt Integration Services

Overview

The Gestalt Integration Services are designed to make integration with common services simple and high quality. This is done by providing a well designed micro-services for each function that does the following:

• Has an adapter layer that allows easy swap out of service implementations.

• Are pre-integrated with pluggable authentication, configuration management, change management, notification, auditing, and enterprise service bus.

- Controllable by policy

Services List

Service Architecture and Descriptions

The diagram above shows the integration services breakdown. The services on the bottom, such as auth/task/event, are process services, which are services that all other gestalt services consume in order to function.

The process services can be thought of as best practices. In today’s world services must support being asynchronous, have variable auth-n/auth-z strategies, have integration with change management, config management, etc. This is accomplished in gestalt using these process services. User applications can consume and use these process as well.

Meta Service – The meta service is the “brain” of the gestalt framework, and doesn’t classify as a micro-service. It has a section all to itself, refer to the “Meta Layer” section of this document.

Security Service - This service acts as the authentication and authorization service for all aspects of the gestalt framework and can be used by user applications as well.

Adapters:

• OAuth

• SAML

• LDAP

• AD

• Play Framework – Custom Plugin for Play Based Apps.

• Custom Plugins

Repo:

Task Service – The task service is responsible for tracking the status of any tasks. It’s primary use is to make all other service asynchronous.

Adapters:

• None

Repo:

Event Service – The event service is used as a message bus. Services and Applications can throw and listen for events. This service is used heavily for gestalt’s change management, policy and lambda services. The only adapter currently supported is kafka, and the service provides a rest interface and standardized message semantics. It includes a security filter for Kafka.

Adapters:

• Kafka

• (Q1 2016) – Storm

  • The storm/spout model will be used to rethrow events to other message buses, in order to support integration with Rabbit, MQ, and other traditional enterprise message queues.

    Change Management Service – The change management services listen for specific events on the event bus and ensures that they are approved before propagating them to the next state. It optionally can allow the events to be changed before moving to the next state.

    Adapters:

    • Java/Scala – Currently only supported plugin in java/scala.
    • (1H-2016) Servicenow
    • (1H-2016) BPML
    • (2H-2016) Remedy

    Config Management Service – The config management service has been tightly integrated into meta. Please refer to the section entitled “Meta Layer’ for details on how configuration works.

    Scheduling Service – The scheduling service is a thin service wrapper around an embedded chronos implementation.

    Adapters: - Chronos

    Notification – The notification service is used to handle bi-directional messaging between applications and their users via alternative communication channels such as SMS, email, voice.

    Adapters: - Twilio (SMS / Voice) - Email (Gmail / SMTP) - (1h 2016) Plivio

    Provisioning / Launcher – The provisioning service, referred to as “Launcher”, is used to start, stop, scale, and migrate applications.

    At current the provisioning model is entirely around processes or containers. Gestalt leverage mesos inside a vm to do this on top of vmware / openstack and cloud providers today. In 2016 we will begin to support traditional VMs with no container requirement.

    Adapters: - Mesos / Marathon - (Q4 2016) – Mesos / Aurora - (Mid 2016) – EC2 Container Service - (1H 2016) – EC2 & Cloud Formation (no migrate functionality) - (1h 2016) – Tosca / Heat (no migrate functionality)

    DNS – The DNS layer is use to update DNS dynamically during deployment, migration, and failover.

    Adapters: - Route 53 - Mesos-DNS - Bind9

    Load Balancing – The load balancing service is based on HA proxy. It is currently not usable as a standalone service and must be leveraged using the launcher service.

    Adapters: - HA Proxy - (2h 2016 – NGINX) - (2h 2016 – F5)

    Firewall – The firewalling service is currently only available via the launcher process. It uses a combination of iptables and haproxy routing rules to dynamically open/remove access to applications during deployment, scale, and kill events

    Adapters: - HA Proxy - IPTables - (1h 2016 – EC2 Security Groups)

    SSL – The SSL service is used to obtain SSL certificates for applications and services as they are being deployed. Currently this service is accessible only to applications launched via the provisioning service. It requires the use of wildcard SSL certificates.

    Adapters: - SSLMate

    Logging / Auditing – The logging/auditing service leverages the event service. It can be used by applications to log events, or to harvest log files for querying from a central location.

    Adapters: - Elastic Search - Log4j - Logback - Kafka.

    Backup – The backup service leverage the distributed file system for regular snapshots and backup of applications and their data. It is currently only accessible via the provisioning service.

    Adapters: - Quobyte - S3

    Accounting / Transactions – The accounting / transaction services are provide a General Ledger capability to applications so they can track capital flows into and out of a system. The transaction functionality is used to execute payment transactions across crypto-currencies and traditional currencies.

    Adapters: - Stripe - Tether - (Q1 2016) Paypal - (Q2 2016) Quickbooks

    Lambda – The lambda service is used to execute code in response to events.

    Adapters: - Vert.x - (1h 2016 – Amazon Lambda)

    Event Transformer – The event transformer service is used to transform events from a variety of channels into something suitable for a Lambda event. You can think of it as an optional bridge between the notification service and the Lambda service that makes it easy to build notification driven request/response systems.

Meta Layer

Overview

The meta layer acts as the central point for configuration and management of for all Gestalt service, the integration layer, and for applications managed by the Gestalt Framework.

The name “Meta” is derived from the service being used to create meta-models of a variety of domains and then apply policy and configuration to them.

Description

The Meta layer acts as both a graph and a hierarchal data structure at the same time. Every resource in meta can contain links to other resources.

The high level meta objects are:

• Organizations

• People

• Applications

• Infrastructure

• Services

• Assets

• Events

• Workspace

  • Environments
    
    • Clusters
      
      • Nodes
        

Typically, resources in meta are modeled in a hierarchy under Organizations. This is pragmatic as it’s how most companies and organizations function in life. Organizations can contain sub organizations nested as deep as necessary.

Configuration and policy can be specified at the root organization and will be inherited to all sub organizations. There is a type of container called a “workspace” which must be owned by an org. Workspaces are basically containers for applications. The reasoning is that many times several applications must be combined to form a working platform, so for that reason we use the term workspace instead of application. The “application” namespace is reserved for the description of an application and its design time information.

The workspace container holds “environments’, which are a subtype of a workspace used to specify constraints. Environments contain application blueprints as well as running applications and their infrastructure.

The following diagram shows an example of a 3 node application cluster. You will note that at any level configuration can be specified and it will be inherited download. Services are exposed by clusters and nodes.

Environments also contain other meta data or links to external gestalt services. For example, an application using the gestalt security or transaction services would have links to them at the environment level. This provides the end user with the ability to browse their applications and see all the users/transactions/infrastructure/services those applications consume or expose in a particular environment.

Obtaining Gestalt

The gestalt framework is spread across three locations:

NOTE: Our public repos WILL NOT be published until June 29, 2016.

Public Github Repo - https://github.com/gestalt-framework

Private Github Repo - https://github.com/GalacticFog

• Artifactory - https://galacticfog.artifactoryonline.com

Artifactory contains the compiled binaries and Docker images. The public repo contains the source for the apache 2 licensed components. The private github repo is contains the source for the commercial components as well as being the repository used for all work on the framework.