Data Aggregation Sources

How to setup data sources for Daggy

Supported formats

Daggy supports yaml and json formats. In yaml repeated nodes can be grouped by yaml standard.

Data sources template

YAML
JSON
YAML
sources:
host1:
#....
host2:
#....
#....
hostN:
#....
JSON
{
"sources": {
"host1" : {
},
"host2" : {
},
"hostN" : {
}
}
}

Each config conatins map of hosts. Host (data source) parameters is next:

Parameter

Type

Description

Is Requiered

type

string

Type of connection to host. Daggy supportes local and ssh host connection types

Yes

commands

array

Array of commands for simultaneous launch

Yes

parameters

map

Connection parameters

Required for ssh2 type

host

string

Host address

No. For ssh2 is 127.0.0.1 by default

reconnect

boolean

true, if need reconnect connection

No

Data Sources Types

Daggy supportes local and ssh2 (remote) host connection types.

Local type

YAML
JSON
YAML
sources:
localhost:
type: local
commands:
pingYa:
exec: ping ya.ru
extension: log
JSON
{
"sources": {
"localhost": {
"commands": {
"pingYa": {
"extension": "log",
"exec": "ping ya.ru"
}
},
"type": "local"
}
}
}

SSH2 type

YAML
JSON
YAML
remotehost:
type: ssh2
host: 192.168.1.9
restart: false
connection:
user: muxa
key: /home/muxa/.ssh/id_rsa
commands:
pingYa:
command: ping ya.ru
extension: log
JSON

{
"localhost": {
"type": "ssh",
"host": "192.169.1.9",
"restart": false,
"connection": {
"login": "muxa",
"key": "/home/muxa/.ssh/id_rsa"
},
"commands": {
"pingYa": {
"command": "ping ya.ru",
"extension": "log"
}
}
}
}

SSH type additional parameters

  • host - remote host ip address or url

  • connection - map of ssh connection parameters. Includes next parameters

Connection Parameter

Type

Description

Default value

user

string

ssh2 connection login

Current user

passphrase

string

ssh2 connection password. If field is not empty, then will be used password authentication for ssh2 connection. In other case, will be used public key authentication

key

string

path to private key for ssh2 connection

~/.ssh/id_rsa

keyphrase

string

passphrase for private key file

port

integer

ssh2 connection port

22

timeout

integer

limit to establish ssh2 connection, in milliseconds

1000

Commands

YAML
JSON
YAML
commands:
pingYa:
exec: ping ya.ru
extension: log
restart: false
JSON
"commands": {
"pingYa": {
"command": "ping ya.ru",
"extension": "log"
}
}

Each command must contain:

  • name - unique within host command identifier, using in command output file name template.

  • exec - shell script

  • extension - extension for command output file

  • restart - restart command if it finished

Using environment and other variables in Data Aggregation Sources

Mustache syntax

YAML/JSON Data Aggregation Sources are support mustache syntax:

  • env_* - template for environment variable, where * is environment variable name.

  • output_folder - output folder path

Example of using mustache syntax in Data Aggregation Sources

YAML
JSON
YAML
aliases:
- &my_commands
pingYa:
exec: ping ya.ru
extension: log
pingGoo:
exec: ping goo.gl
extension: log
- &ssh_auth
user: {{env_USER}}
passphrase: {{env_PASSWORD}}
sources:
localhost:
type: local
commands: *my_commands
remotehost:
host: 192.168.1.9
type: ssh2
parameters: *ssh_auth
commands: *my_commands
remotehost2:
host: 192.168.1.9
type: ssh2
parameters: *ssh_auth
commands: *my_commands
remotehost3:
host: 192.168.1.9
type: ssh2
parameters: *ssh_auth
commands: *my_commands
JSON
{
"sources": {
"remotehost3": {
"host": "192.168.1.9",
"type": "ssh2",
"commands": {
"pingYa": {
"extension": "log",
"exec": "ping ya.ru"
},
"pingGoo": {
"extension": "log",
"exec": "ping goo.gl"
}
},
"parameters": {
"user": "{{env_USER}}",
"passphrase": "{{env_PASSWORD}}"
}
},
"remotehost2": {
"host": "192.168.1.9",
"type": "ssh2",
"commands": {
"pingYa": {
"extension": "log",
"exec": "ping ya.ru"
},
"pingGoo": {
"extension": "log",
"exec": "ping goo.gl"
}
},
"parameters": {
"user": "{{env_USER}}",
"passphrase": "{{env_PASSWORD}}"
}
},
"remotehost": {
"host": "192.168.1.9",
"type": "ssh2",
"commands": {
"pingYa": {
"extension": "log",
"exec": "ping ya.ru"
},
"pingGoo": {
"extension": "log",
"exec": "ping goo.gl"
}
},
"parameters": {
"user": "{{env_USER}}",
"passphrase": "{{env_PASSWORD}}"
}
},
"localhost": {
"commands": {
"pingYa": {
"extension": "log",
"exec": "ping ya.ru"
},
"pingGoo": {
"extension": "log",
"exec": "ping goo.gl"
}
},
"type": "local"
}
}
}