Data Aggregation Sources

How to setup data sources for Daggy

Supported formats
Daggy supports yaml and json formats. In yaml repeated nodes can be grouped by yaml standard.
Data sources template
YAML
JSON
YAML
sources:
    host1:
        #....
    host2:
        #....
    #....
    hostN:
        #....
    
JSON
{
    "sources": {
        "host1" : {
        },
        "host2" : {
        },
        "hostN" : {
        }
    }
}
Each config conatins map of hosts. Host (data source) parameters is next:
Parameter
Type
Description
Is Requiered
type
string
​
Type of connection to host. Daggy supportes local and ssh host connection types
Yes
commands
array
Array of commands for simultaneous launch
Yes
parameters
map
Connection parameters
Required for ssh2 type
host
string
Host address
No. For ssh2 is 127.0.0.1 by default
reconnect
boolean
true, if need reconnect connection
No 
Data Sources Types
Daggy supportes local and ssh2 (remote) host connection types.
Local type
YAML
JSON
YAML
sources:
    localhost:
        type: local
        commands:
            pingYa:
                exec: ping ya.ru
                extension: log
JSON
 {
    "sources": {
        "localhost": {
            "commands": {
                "pingYa": {
                    "extension": "log",
                    "exec": "ping ya.ru"
                }
            },
            "type": "local"
        }
    }
}
SSH2 type
YAML
JSON
YAML
remotehost:
    type: ssh2
    host: 192.168.1.9
    restart: false
    parameters:
      user: muxa
      key: /home/muxa/.ssh/id_rsa
    commands:
      pingYa:
        command: ping ya.ru
        extension: log
JSON
​
{
    "localhost": {
        "type": "ssh",
        "host": "192.169.1.9",
        "restart": false,
        "parameters": {
            "user": "muxa",
            "key": "/home/muxa/.ssh/id_rsa"
        },
        "commands": {
            "pingYa": {
                "command": "ping ya.ru",
                "extension": "log"
            }
        }
    }
}
SSH type additional parameters
host - remote host ip address or url
connection - map of ssh connection parameters. Includes next parameters
Connection Parameter
Type
Description
Default value
user
string
ssh2 connection login
Current user
passphrase
string
ssh2 connection password. If field is not empty, then will be used password authentication for ssh2 connection. In other case, will be used public key authentication 
​
key
string
path to private key for ssh2 connection
~/.ssh/id_rsa
keyphrase
string
passphrase for private key file
​
port
integer
ssh2 connection port
22
timeout
integer
limit to establish ssh2 connection, in milliseconds
1000
Commands
YAML
JSON
YAML
commands:
      pingYa:
        exec: ping ya.ru
        extension: log
        restart: false
JSON
"commands": {
            "pingYa": {
                "command": "ping ya.ru",
                "extension": "log"
            }
        }
Each command must contain:
name - unique within host command identifier, using in command output file name template.
exec - shell script
extension - extension for command output file
restart - restart command if it finished
Using environment and other variables in Data Aggregation Sources
Mustache syntax
YAML/JSON Data Aggregation Sources are support mustache syntax:
env_* - template for environment variable, where * is environment variable name.
output_folder - output folder path
Example of using mustache syntax in Data Aggregation Sources
YAML
JSON
YAML
aliases:  
    - &my_commands
        pingYa:
            exec: ping ya.ru
            extension: log
        pingGoo:
            exec: ping goo.gl
            extension: log
        
    - &ssh_auth
        user: {{env_USER}}
        passphrase: {{env_PASSWORD}}
            
sources:
    localhost:
        type: local
        commands: *my_commands
    remotehost:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost2:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost3:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
JSON
{
  "sources": {
    "remotehost3": {
      "host": "192.168.1.9", 
      "type": "ssh2", 
      "commands": {
        "pingYa": {
          "extension": "log", 
          "exec": "ping ya.ru"
        }, 
        "pingGoo": {
          "extension": "log", 
          "exec": "ping goo.gl"
        }
      }, 
      "parameters": {
        "user": "{{env_USER}}", 
        "passphrase": "{{env_PASSWORD}}"
      }
    }, 
    "remotehost2": {
      "host": "192.168.1.9", 
      "type": "ssh2", 
      "commands": {
        "pingYa": {
          "extension": "log", 
          "exec": "ping ya.ru"
        }, 
        "pingGoo": {
          "extension": "log", 
          "exec": "ping goo.gl"
        }
      }, 
      "parameters": {
        "user": "{{env_USER}}", 
        "passphrase": "{{env_PASSWORD}}"
      }
    }, 
    "remotehost": {
      "host": "192.168.1.9", 
      "type": "ssh2", 
      "commands": {
        "pingYa": {
          "extension": "log", 
          "exec": "ping ya.ru"
        }, 
        "pingGoo": {
          "extension": "log", 
          "exec": "ping goo.gl"
        }
      }, 
      "parameters": {
        "user": "{{env_USER}}", 
        "passphrase": "{{env_PASSWORD}}"
      }
    }, 
    "localhost": {
      "commands": {
        "pingYa": {
          "extension": "log", 
          "exec": "ping ya.ru"
        }, 
        "pingGoo": {
          "extension": "log", 
          "exec": "ping goo.gl"
        }
      }, 
      "type": "local"
    }
  }
}
  

Getting Started

Data Aggregation Snippets

Last updated 1 year ago

Parameter	Type	Description	Is Requiered
type	string	Type of connection to host. Daggy supportes `local` and `ssh` host connection types	Yes
commands	array	Array of commands for simultaneous launch	Yes
parameters	map	Connection parameters	Required for ssh2 type
host	string	Host address	No. For ssh2 is 127.0.0.1 by default
reconnect	boolean	true, if need reconnect connection	No

Connection Parameter	Type	Description	Default value
user	string	ssh2 connection login	Current user
passphrase	string	ssh2 connection password. If field is not empty, then will be used password authentication for ssh2 connection. In other case, will be used public key authentication
key	string	path to private key for ssh2 connection	`~/.ssh/id_rsa`
keyphrase	string	passphrase for private key file
port	integer	ssh2 connection port	22
timeout	integer	limit to establish ssh2 connection, in milliseconds	1000