Mesos by zigi

1KrDAG – ZIGI zigispace.netKrDAG – ZIGI zigispace.net

Introduce

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications with API’s for resource management and scheduling across entire datacenter and cloud environments.

소개

메소스

Mesos는 Machine(혹은 Node)의 자원(CPU, Memory, Storage 등) 을 공유해서 사용.

일종의 데이터센터 자원의 커널

Cloud Infrastructure 및 Computing Engine들의 자원을 통합적 으로 관리 할 수 있도록만든 자원관리 프로젝트.

• UC Berkeley에서 Nexus 라는 이름으로 개발이 진행되던 프로젝트.

• Mesos라는 이름으로 Apache 재단에 오픈소스로 발표.

• Mesos를 활용 기업

MesosHistory &Reference

• Scalability to 10,000s of nodes

• Fault-tolerant replicated master and slaves using ZooKeeper

• Support for Docker containers

• Native isolation between tasks with Linux Containers

• Multi-resource scheduling (memory, CPU, disk, and ports)

• Java, Python and C++ APIs for developing new parallel applications

• Web UI for viewing cluster state

MesosFeatures

• 인프라에 대한 Utility를 향상 시켜, 비용 감소

• 개발자 관점에서 리소스에 대한 View를 단순화 함.

• 다양한 워크로드를 수행할 수 있는 프레임워크를 지원

• 특정 OS 혹은 벤더에 Lock-In되지 않기 때문에 H/W나 OS 선택의 폭이 넓음.

MesosWhy?

Architecture : Master

◦ Slave 리소스와 Framework 간의 중재 역할

◦ 안정성을 위해서 다수의 Master를 운영 가능

- Active 상태의 Master는 1대만 존재

- Master Active 선출/관리는 ZooKeeper 사용

- Active 이외의 Master는 Standby 상태 유지

◦ Master는 Framework가 Slave Resource를

사용할 수 있도록 제공.

Architecture : Slave

◦ Mesos 클러스터의 실제 작업(Task)을 수행

◦ 개별 노드에 대한 자원 관리 및 비즈니스 우선

순위를 반영하기 위한 자원 정책 구성.

Architecture : Frameworks

◦ 각 프레임워크는 스케줄러와 Executor로 구성.

◦ scheduler

- Master에 등록된 리소스를 사용하도록 등록.

◦ executor

- 슬레이브 노드에서 프레임워크 Task를

실행하기 위한 프로세스

Architecture

• Aurora is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of Mesos' scalability, fault-tolerance, and resource isolation.

• Marathon is a private PaaS built on Mesos. It automatically handles hardware or software failures and ensures that an app is “always on”.

• Singularity is a scheduler (HTTP API and web interface) for running Mesos tasks: long running processes, one-off tasks, and scheduled jobs.

• SSSP is a simple web application that provides a white-label “Megaupload” for storing and sharing files in S3.

MesosFramework : Long Running Services

• Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.

• Dpark is a Python clone of Spark, a MapReduce-like framework written in Python, running on Mesos.

• Exelixi is a distributed framework for running genetic algorithms at scale.

• Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.

• Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations e.g., matrix, graph and network algorithms.

• MPI is a message-passing system designed to function on a wide variety of parallel computers.

• Spark is a fast and general-purpose cluster computing system which makes parallel jobs easy to write.

• Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing.

MesosFramework : Big Data Processing

• Chronos is a distributed job scheduler that supports complex job topologies. It can be used as a more fault-tolerant replacement for Cron.

• Jenkins is a continuous integration server. The mesos-jenkins plugin allows it to dynamically launch workers on a Mesos cluster depending on the workload.

• JobServer is a distributed job scheduler and processor which allows developers to build custom batch processing Tasklets using point and click web UI.

MesosFramework : Batch Scheduling

• Cassandra is a performant and highly available distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.

• ElasticSearch is a distributed search engine. Mesos makes it easy to run and scale.

• Hypertable is a high performance, scalable, distributed storage and processing system for structured and unstructured data.

MesosFramework : Data Storage

High availability

Mesos is configured for high availability, the leading master will abort itself when it discovers it has been partitioned away from the Zookeeper quorum. This is a safety precaution to ensure the previous leader doesn’t continue communicating in an unsafe state.

• Master Quorum이 2N-1 일 때, N개 이상의 Master가 동작 중이어야 함.

• Master Quorum의 수량은 3 혹은 5개를 권고

MesosMaster Quorum

Master Quorum Size Failure Tolerance

… … …

2N-1 N N-1

Mesos 동작방식

Step 1. Slave 자신의 자원을 Master 서버에게 알림.

Slave Master

Resource register

Step 2. Master 서버는 Slave 자원 정보를 Framework에 알림.

Master Framework

Resource Inform

Step 3. Framework에 Job 1을 수행토록 지시.

Admin Framework

Task (Job)

Step 4. Master 서버는 Job 1을 Slave 서버에 전송.

MasterFramework Slave

Step 5. Slave 서버는 해당 작업을 수행

Running

Example of resource offer

Datacenter OS

Datacenter operating system - DCOS

The DCOS runs in any modern Linux environment: public and private cloud, virtual machines and bare metal.

Built in to the DCOS, Mesos pools your infrastructure, automatically allocating resources and scheduling tasks based on demands and policy.

Control your entire DCOS-powered datacenter from a single command line and a single web user interface.

Run all your apps and workloads, from PaaS-based microservices to big data to databases — and everything in between.

DATACENTER OPERATING SYSTEM - DCOS

Mesos 미리보기

Dashboard

Dashboard : Frameworks

Dashboard : Slaves

Dashboard : Application

Marathon

Dashboard : Application Detail

Marathon

Pushing control to the frameworks has two benefits. First, it allows frameworks to implement diverse approaches to various problems in the cluster (e.g., achieving data locality, dealing with faults), and to evolve these solutions independently. Second, it keeps Mesos simple and minimizes the rate of change required of the system, which makes it easier to keep Mesos scalable and robust.

Mesos by zigi

Technology

Transcript of Mesos by zigi

Mesos de l'any alicia

12 Mesos 12 Flors

· (Edados comprondidas mtro O mesos O día. y 102 mesos 15 diar) NiXor soloccionados ..... 139 Concurrencias soloccionadas ..739 Patronos ..... dr crocimiontoi CLAPti.PED CLAPH.TED

Mesos intro

Els mercats perden l’impuls dels últims mesos · CONJUNTURA • Els mercats perden l’impuls dels últims mesos Els mercats internacionals posen entre parèntesis l’episodi

Convocatòria Erasmus Pràctiques 2019-2020...Conjunta amb Erasmus Estudis Durada mínima: 2 mesos Durada màxima : 12 mesos Període finançat : màxim 4 mesos Entre l’1 de setembre

Gestión de recursos con Apache Mesos - dit

Mesos HTTP API

Service discovery in mesos

Mesos de l'any

Mesos framework API v1

Mesos originals

Psicologia Clínica...Atenció Comunitària, ambulatòria i suport de l’atenció primària 8 mesos Addiccions 4 mesos Tercer any de residència Hospital de Dia d’Adults 2 mesos

Dites dels mesos de l'any

Programa activitats mesos d'abril, maig i juny 2010

Sis mesos de cirurgia ambulatória

Jenkins в docker in mesos in ...

Introduce Apache Mesos

Els Mesos De Lany

Peritoneo, Omentos y Mesos