Introduction
DC/OS is an open-source operating system and distributed system for data center built on Apache Mesos distributed system kernel. As a distributed system, it is a cluster of master nodes and private/public nodes, where each node also has host operating system which manages the underlying machine.
It enables the management of multiple machines as if they were a single computer. It automates resource management, schedules process placement, facilitates inter-process communication, and simplifies the installation and management of distributed services. Its included web interface and available command-line interface (CLI) facilitate remote management and monitoring of the cluster and its services.
- Distributed System : DC/OS is distributed system with group of private and public nodes which are coordinated by master nodes.
- Cluster Manager : DC/OS is responsible for running tasks on agent nodes and providing required resources to them. DC/OS uses Apache Mesos to provide cluster management functionality.
- Container Platform : All DC/OS tasks are containerized. DC/OS uses two different container runtimes, i.e. docker and mesos. So that containers can be started from docker images or they can be native executables (binaries or scripts) which are containerized at runtime by mesos.
- Operating System : As name specifies, DC/OS is an operating system which abstracts cluster h/w and s/w resources and provide common services to applications.
Unlike Linux, DC/OS is not a host operating system. DC/OS spans multiple machines, but relies on each machine to have its own host operating system and host kernel.
The high level architecture of DC/OS can be seen below :

For the detailed architecture and components of DC/OS, please click here.
Adoption and usage of Mesosphere DC/OS:
Mesosphere customers include :
- 30% of the Fortune 50 U.S. Companies
- 5 of the top 10 North American Banks
- 7 of the top 12 Worldwide Telcos
- 5 of the top 10 Highest Valued Startups
Some companies using DC/OS are :
- Cisco
- Yelp
- Tommy Hilfiger
- Uber
- Netflix
- Verizon
- Cerner
- NIO
Installing and using DC/OS
A guide to installing DC/OS can be found here. After installing DC/OS on any platform, install dcos cli by following documentation found here.
Using dcos cli, we can manager cluster nodes, manage marathon tasks and services, install/remove packages from universe and it provides great support for automation process as each cli command can be output to json.
NOTE: The tasks below are executed with and tested on below tools:
- DC/OS 1.11 Open Source
- DC/OS cli 0.6.0
- jq:1.5-1-a5b5cbe
DC/OS commands and scripts
Setup DC/OS cli with DC/OS cluster
CODE: https://gist.github.com/velotiotech/295350482d9b64d9cb161e30fa613f49.js
Example :
CODE: https://gist.github.com/velotiotech/e06c26a1946dba6cdfb97a6fc5d0a000.js
The above command will give you the link for oauth authentication and prompt for auth token. You can authenticate yourself with any of Google, Github or Microsoft account. Paste the token generated after authentication to cli prompt. (Provided oauth is enabled).
DC/OS authentication token
CODE: https://gist.github.com/velotiotech/0eecf11aa9e07ead414f388b93211858.js
DC/OS cluster url
CODE: https://gist.github.com/velotiotech/4b39163ac7449b0ad85d025460bef133.js
DC/OS cluster name
CODE: https://gist.github.com/velotiotech/837fa685375036e4865eec7f23994c62.js
Access Mesos UI
CODE: https://gist.github.com/velotiotech/1d0d18f6a477e591bbb8ea7f4aa234a6.js
Example:
CODE: https://gist.github.com/velotiotech/fdbb58c19480eb86f28c39afde605dd4.js
Access Marathon UI
CODE: https://gist.github.com/velotiotech/d7b9e80b6ff24632984db076cb6aaaa2.js
Example:
CODE: https://gist.github.com/velotiotech/5c52cfefd5907ef8fe84940ced9872f7.js
Access any DC/OS service, like Marathon, Kafka, Elastic, Spark etc.[DC/OS Services]
CODE: https://gist.github.com/velotiotech/5bc2ec7130f67cbe00d64d3bdf4e4075.js
Example:
CODE: https://gist.github.com/velotiotech/d1142103bcbc24f8218d94c9da158285.js
Access DC/OS slaves info in json using Mesos API [Mesos Endpoints]
CODE: https://gist.github.com/velotiotech/8119b9997e1660d69ce273a0d08091f7.js
Access DC/OS slaves info in json using DC/OS cli
CODE: https://gist.github.com/velotiotech/37b72f27283692bff66974c7bd03d72c.js
Note : DC/OS cli ‘dcos node --json’ is equivalent to running mesos slaves endpoint (/mesos/slaves)
Access DC/OS private slaves info using DC/OS cli
CODE: https://gist.github.com/velotiotech/eed89c20bbed0e2f19301e408bf96961.js
Access DC/OS public slaves info using DC/OS cli
CODE: https://gist.github.com/velotiotech/8469484366f669f39bfb0a6bcd9735ac.js
Access DC/OS private and public slaves info using DC/OS cli
CODE: https://gist.github.com/velotiotech/8aa35b273bf816542e626f75a5a876c6.js
Get public IP of all public agents
CODE: https://gist.github.com/velotiotech/d3c6b316789d2f30c68e6a525599431b.js
Note: As ‘dcos node ssh’ requires private key to be added to ssh. Make sure you add your private key as ssh identity using :
CODE: https://gist.github.com/velotiotech/c7c18fd96e60a1a393546ff10a6c8e11.js
Get public IP of master leader
CODE: https://gist.github.com/velotiotech/197a5a0f399c2db088d3f43427a82124.js
Get all master nodes and their private ip
CODE: https://gist.github.com/velotiotech/c05ff1417387b090778f1791555752d3.js
Get list of all users who have access to DC/OS cluster
CODE: https://gist.github.com/velotiotech/7df391e854d15c11f4bb452b0613bf6f.js
Add users to cluster using Mesosphere script (Run this on master)
Users to add are given in list.txt, each user on new line
CODE: https://gist.github.com/velotiotech/0e36501fcd8bad66716441a47f727551.js
Add users to cluster using DC/OS API
CODE: https://gist.github.com/velotiotech/cd979e577dc87e6f04a2f380d48f2204.js
Delete users from DC/OS cluster organization
CODE: https://gist.github.com/velotiotech/8c9e8cbb891943d46804ef5ab8812abb.js
Offers/resources from individual DC/OS agent
In recent versions of the many dcos services, a scheduler endpoint at
CODE: https://gist.github.com/velotiotech/d7974f63a3eb0bef6c3990a5b3d446da.js
will display an HTML table containing a summary of recently-evaluated offers. This table’s contents are currently very similar to what can be found in logs, but in a slightly more accessible format. Alternately, we can look at the scheduler’s logs in stdout. An offer is a set of resources all from one individual DC/OS agent.
CODE: https://gist.github.com/velotiotech/acc4953580a35a73383581da346e4b78.js
Example:
CODE: https://gist.github.com/velotiotech/91ac308fa9facea1adfbfc241f602fab.js
Save JSON configs of all running Marathon apps
CODE: https://gist.github.com/velotiotech/ed799dc49557cf11e75ca166839abfd9.js
Get report of Marathon apps with details like container type, Docker image, tag or service version used by Marathon app.
CODE: https://gist.github.com/velotiotech/6e872cd5e990bd7544c4f332e43b6833.js
Get DC/OS nodes with more information like node type, node ip, attributes, number of running tasks, free memory, free cpu etc.
CODE: https://gist.github.com/velotiotech/4b660121b5e50ad6f48c5cf1e7265c9f.js
Framework Cleaner
Uninstall framework and clean reserved resources if any after framework is deleted/uninstalled. (applicable if running DC/OS 1.9 or older, if higher than 1.10, then only uninstall cli is sufficient)
CODE: https://gist.github.com/velotiotech/253b1a5e34236ec8b1ddff2980281bc2.js
Get DC/OS apps and their placement constraints
CODE: https://gist.github.com/velotiotech/c0e6b9687ce50c05e4431e8526460b32.js
Run shell command on all slaves
CODE: https://gist.github.com/velotiotech/2e9053d37a37face5b245b56c13a45c4.js
Run shell command on master leader
CODE: https://gist.github.com/velotiotech/c08fb02aa8a150f9d2a1c92ede167e9f.js
Run shell command on all master nodes
CODE: https://gist.github.com/velotiotech/9f6e881c4e39dffb4280d3c42b265e4f.js
Add node attributes to dcos nodes and run apps on nodes with required attributes using placement constraints
CODE: https://gist.github.com/velotiotech/fbaa43aace1f9e3a0feee8ac367180c7.js
Install DC/OS Datadog metrics plugin on all DC/OS nodes
CODE: https://gist.github.com/velotiotech/6df2f95d5566ec70343f0e5b0e283512.js
Get app / node metrics fetched by dcos-metrics component using metrics API
- Get DC/OS node id [dcos node]
- Get Node metrics (CPU, memory, local filesystems, networks, etc) : <dc os_cluster_url="">/system/v1/agent/<agent_id>/metrics/v0/node</agent_id></dc>
- Get id of all containers running on that agent : <dc os_cluster_url="">/system/v1/agent/<agent_id>/metrics/v0/containers</agent_id></dc>
- Get Resource allocation and usage for the given container ID. : <dc os_cluster_url="">/system/v1/agent/<agent_id>/metrics/v0/containers/<container_id></container_id></agent_id></dc>
- Get Application-level metrics from the container (shipped in StatsD format using the listener available at STATSD_UDP_HOST and STATSD_UDP_PORT) : <dc os_cluster_url="">/system/v1/agent/<agent_id>/metrics/v0/containers/<container_id>/app </container_id></agent_id></dc>
Get app / node metrics fetched by dcos-metrics component using dcos cli
- Summary of container metrics for a specific task
CODE: https://gist.github.com/velotiotech/5329771b824e3bccb94fe8646606ce43.js
- All metrics in details for a specific task
CODE: https://gist.github.com/velotiotech/a93572fc2c935c8a2e74ffbf28a032c4.js
- Summary of Node metrics for a specific node
CODE: https://gist.github.com/velotiotech/01bd594403d7dc437bf7ab74e709134b.js
- All Node metrics in details for a specific node
CODE: https://gist.github.com/velotiotech/af628af75dada728de9fed7223c67cc5.js
NOTE - All above commands have ‘--json’ flag to use them programmatically.
Launch / run command inside container for a task
DC/OS task exec cli only supports Mesos containers, this script supports both Mesos and Docker containers.
CODE: https://gist.github.com/velotiotech/54fc03b343bf3dff88c3f97122eae2ac.js
Get DC/OS tasks by node
CODE: https://gist.github.com/velotiotech/07dbeafa0a74bd2f34b4cd48b07d5144.js
Get cluster metadata - cluster Public IP and cluster ID
CODE: https://gist.github.com/velotiotech/03a1ddc04a73cf3dcaaa9be457fbf699.js
Sample Output:
CODE: https://gist.github.com/velotiotech/b6791ab15cf41899df3a7fe9bbdb61f7.js
Get DC/OS metadata - DC/OS version
CODE: https://gist.github.com/velotiotech/0b69e92d5b7d2e5609c975d0fed458b7.js
Sample Output:
CODE: https://gist.github.com/velotiotech/b9c7bfe4d81f1a2ded7fe10c488950bd.js
Get Mesos version
CODE: https://gist.github.com/velotiotech/0429ef0928c771c3c5471b1bc59f3640.js
Sample Output:
CODE: https://gist.github.com/velotiotech/fdc9543771fad7366e8be69e6db8ba84.js
Access DC/OS cluster exhibitor UI (Exhibitor supervises ZooKeeper and provides a management web interface)
CODE: https://gist.github.com/velotiotech/cfce6fcee74f1dcd9528417595b48c6f.js
Access DC/OS cluster data from cluster zookeeper using Zookeeper Python client - Run inside any node / container
CODE: https://gist.github.com/velotiotech/d01f0ddf9b30bdf800cd85b9d3afd659.js
Access dcos cluster data from cluster zookeeper using exhibitor rest API
CODE: https://gist.github.com/velotiotech/7b4e6e6510990324c1b3b4155e637a01.js
Sample Output:
CODE: https://gist.github.com/velotiotech/53c2eb51b871a1707b66655f84ae17b2.js
Get cluster name using Mesos API
CODE: https://gist.github.com/velotiotech/3c67246ca3e23276489aa2236b00139a.js
Mark Mesos node as decommissioned
Some times instances which are running as DC/OS node gets terminated and can not come back online, like AWS EC2 instances, once terminated due to any reason, can not start back. When Mesos detects that a node has stopped, it puts the node in the UNREACHABLE state because Mesos does not know if the node is temporarily stopped and will come back online, or if it is permanently stopped. In such case, we can explicitly tell Mesos to put a node in the GONE state if we know a node will not come back.
CODE: https://gist.github.com/velotiotech/e8c66afa28da4e0359363f136608e103.js
Conclusion
We learned about Mesosphere DC/OS, its functionality and roles. We also learned how to setup and use DC/OS cli and use http authentication to access DC/OS APIs as well as using DC/OS cli for automating tasks.
We went through different API endpoints like Mesos, Marathon, DC/OS metrics, exhibitor, DC/OS cluster organization etc. Finally, we looked at different tricks and scripts to automate DC/OS, like DC/OS node details, task exec, Docker report, DC/OS API http authentication etc.