Getting Started

Getting started data aggregation and streaming with daggy
Getting Daggy
Windows
Download installation package (exe) from last release page and install it
MacOS
brew install https://raw.githubusercontent.com/synacker/daggy_homebrew/master/daggy.rb
Fedora 30/31/32
sudo dnf install daggy daggy-devel
Linux
Download rpm/deb package from last release page and install it
All daggy dependencies (Qt5, libssh2, libyaml-cpp) are included into separate dir and does not effect system applications.
Build from source
Environment requirenments
​Conan, cmake, git and C++ compiler.
Build steps
git clone https://github.com/synacker/daggy.git
mkdir build
cd build
conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
conan install ../daggy --build=missing
conan build ../daggy
cmake install
Check installation
daggy --help
Usage: daggy [options] *.yaml|*.yml|*.json
​
Options:
  -o, --output <folder>       Set output folder
  -f, --format <json|yaml>    Source format
  -i, --stdin                 Read data aggregation sources from stdin
  -t, --timeout <time in ms>  Auto complete timeout
  -h, --help                  Displays help on commandline options.
  --help-all                  Displays help including Qt specific options.
  -v, --version               Displays version information.
​
Arguments:
  file                        data aggregation sources file
Getting Started with data aggregation and streaming
Simple Data Aggregation Source
Create yaml or json file
YAML
JSON
YAML
simple.yaml
sources:
    localhost:
        type: local
        commands:
            pingYa:
                exec: ping ya.ru
                extension: log
JSON
simple.json
{
    "sources": {
        "localhost": {
            "commands": {
                "pingYa": {
                    "extension": "log",
                    "exec": "ping ya.ru"
                }
            },
            "type": "local"
        }
    }
}
Run daggy
YAML
JSON
YAML
daggy simple.yaml
JSON
daggy simple.json
Check console output
23:07:23:977 | AppStat  | Start aggregation in 01-04-20_23-07-23-977_simple
23:07:23:977 | ProvStat | localhost       | New state: Started
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Starting
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Started
There are all commands from simple.yaml/simple.json are streams in 01-04-20_23-07-23-977_simple with output files
Tailing streams from Simple Data Source
tail -f 01-04-20_23-07-23-977_simple/*
64 bytes from ya.ru (87.250.250.242): icmp_seq=99 ttl=249 time=21.2 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=100 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=101 ttl=249 time=23.5 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=102 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=103 ttl=249 time=18.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=104 ttl=249 time=17.4 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=105 ttl=249 time=17.4 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=106 ttl=249 time=20.1 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=107 ttl=249 time=25.8 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=108 ttl=249 time=35.1 ms
64 bytes from ya.ru (87.250.250.242): icmp_seq=109 ttl=249 time=21.1 ms
Stop data aggregation and streaming
Type CTRL+C for stopping data aggregation and streaming. Type CTRL+C twice for hard stop application, without waiting cancelation of child local and remote processes. 
23:07:23:977 | AppStat  | Start aggregation in 01-04-20_23-07-23-977_simple
23:07:23:977 | ProvStat | localhost       | New state: Started
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Starting
23:07:23:977 | CommStat | localhost       | pingYa          | New state: Started
^C23:17:56:667 | ProvStat | localhost       | New state: Finishing
23:17:56:668 | CommStat | localhost       | pingYa          | New state: Finished. Exit code: 0
23:17:56:668 | ProvStat | localhost       | New state: Finished
23:17:56:668 | AppStat  | Stop aggregation in 01-04-20_23-07-23-977_simple
Investigate aggregated data
ls -l 01-04-20_23-07-23-977_simple/
-rw-r--r-- 1 muxa muxa 45574 апр  1 23:17 localhost-pingYa.log
Example of Data Aggregation Sources with multiple commands and remote data aggregation and streaming
YAML
JSON
YAML
multiple_and_remote.yaml
aliases:  
    - &my_commands
        pingYa:
            exec: ping ya.ru
            extension: log
        pingGoo:
            exec: ping goo.gl
            extension: log
        
    - &ssh_auth
        user: muxa
        key: /home/muxa/.ssh/id_rsa
            
sources:
    localhost:
        type: local
        commands: *my_commands
    remotehost:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost2:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
    remotehost3:
        host: 192.168.1.9
        type: ssh2
        parameters: *ssh_auth
        commands: *my_commands
JSON
multiple_and_remote.json
{
    "sources": {
      "remotehost3": {
        "parameters": {
          "user": "muxa", 
          "key": "/home/muxa/.ssh/id_rsa"
        }, 
        "commands": {
          "pingYa": {
            "extension": "log", 
            "exec": "ping ya.ru"
          }, 
          "pingGoo": {
            "extension": "log", 
            "exec": "ping goo.gl"
          }
        }, 
        "type": "ssh2", 
        "host": "192.168.1.9"
      }, 
      "remotehost2": {
        "parameters": {
          "user": "muxa", 
          "key": "/home/muxa/.ssh/id_rsa"
        }, 
        "commands": {
          "pingYa": {
            "extension": "log", 
            "exec": "ping ya.ru"
          }, 
          "pingGoo": {
            "extension": "log", 
            "exec": "ping goo.gl"
          }
        }, 
        "type": "ssh2", 
        "host": "192.168.1.9"
      }, 
      "remotehost": {
        "parameters": {
          "user": "muxa", 
          "key": "/home/muxa/.ssh/id_rsa"
        }, 
        "commands": {
          "pingYa": {
            "extension": "log", 
            "exec": "ping ya.ru"
          }, 
          "pingGoo": {
            "extension": "log", 
            "exec": "ping goo.gl"
          }
        }, 
        "type": "ssh2", 
        "host": "192.168.1.9"
      }, 
      "localhost": {
        "commands": {
          "pingYa": {
            "extension": "log", 
            "exec": "ping ya.ru"
          }, 
          "pingGoo": {
            "extension": "log", 
            "exec": "ping goo.gl"
          }
        }, 
        "type": "local"
      }
    }
  }
About Daggy
Data Aggregation Sources
Last updated 1 month ago