Daggy - Data Aggregation Utility
Daggy - Data Aggregation Utility
Install
GitHub
About Daggy
Getting Started
How it works
Data Aggregation Config
Data Aggregation Snippets
Troubleshooting
Powered by GitBook

Getting Started

Getting Started aggregate data via command line

Download and Install Daggy

Fedora 29/30/31

$ sudo dnf install daggy

Debian/Ubuntu

sudo add-apt-repository ppa:synacker/daggy
sudo apt-get update
sudo apt-get install daggy

Arch Linux

yay daggy

MacOS

brew install https://raw.githubusercontent.com/synacker/homebrew-core/daggy/Formula/daggy.rb

Windows 7/8/10

Download and install from releases​

Build from sources

Dependency

  • Qt Core and Qt Network 5.5 or newer

  • libbotan-devel 2.7 or newer

  • yaml-cpp 0.6.0 or newer

Build instractions

git clone https://github.com/synacker/daggy.git
cd daggy
qmake && make
cd Release
daggy -h

Configuration

Simple Data Aggregation Config

simple.yaml
sources:
  localhost:
    type: local
    commands:
      - name: pingYa
        command: ping ya.ru
        extension: log
daggy simple.yaml
23:15:41:005 | AppStatus  | Application     | Start receiver
23:15:41:005 | ConnStatus | localhost       | Server connected
23:15:41:006 | CommStatus | localhost       | pingYa          | Command started

In your working directory daggy were created directory with next template:

{date}-{source_name}

ls
12-06-19_23-15-41-simple  simple.yaml

Output folder contain file with pingYa command output with next template:

{server_name}_{command_name}.{command_extension}

ls 12-06-19_23-15-41-simple/
localhost_pingYa.log

The output file filling at runtime, you can tail content of the file:

tail -f 12-06-19_23-15-41-simple/localhost_pingYa.log

Type CTRL+C for stop commands execution

23:16:25:685 | CommStatus | localhost       | pingYa          | Command was crashed
23:16:25:686 | ConnStatus | localhost       | Server disconnected
23:16:25:686 | AppStatus  | Application     | Stop receiver

Add parallel command

Add new command to our simple config - pingGoo

simple_add_command.yaml
sources:
  localhost:
    type: local
    commands:
      - name: pingYa
        command: ping ya.ru
        extension: log
      - name: pingGoo
        command: ping goo.gl
        extension: log
daggy simple_add_command.yaml 
21:38:02:154 | AppStatus  | Application     | Start receiver
21:38:02:154 | ConnStatus | localhost       | Server connected
21:38:02:154 | CommStatus | localhost       | pingGoo         | Command started
21:38:02:155 | CommStatus | localhost       | pingYa          | Command started
​

pingYa and pingGoo work in parallel, and both commands filling output files localhost_pingGoo.log localhost_pingYa.log at runtime.

You can tail all output files while daggy is working:

tail 14-06-19_21-38-02-simple_add_command/*.log

Add remote server

Now, add remote server 192.168.1.9 with ssh connection with the same commands:

simple_add_remote_server.yaml
sources:
  localhost:
    type: local
    commands:
      - name: pingYa
        command: ping ya.ru
        extension: log
      - name: pingGoo
        command: ping goo.gl
        extension: log
  remotehost:
    type: ssh
    host: 192.168.1.9
    connection:
      login: muxa
      key: /home/muxa/.ssh/id_rsa
    commands:
      - name: pingYa
        command: ping ya.ru
        extension: log
      - name: pingGoo
        command: ping goo.gl
        extension: log
simple_add_remote_server.yaml
daggy simple_add_remote_server.yaml 
22:08:26:093 | AppStatus  | Application     | Start receiver
22:08:26:093 | ConnStatus | localhost       | Server connected
22:08:26:093 | CommStatus | localhost       | pingGoo         | Command started
22:08:26:094 | CommStatus | localhost       | pingYa          | Command started
22:08:26:131 | ConnStatus | remotehost      | Server connected
22:08:26:131 | CommStatus | remotehost      | pingGoo         | Command started
22:08:26:131 | CommStatus | remotehost      | pingYa          | Command started
​

pingYa and pingGoo work in parallel on both servers - localhost and remotehost.

You can tail all output files while daggy is working:

tail -f 14-06-19_22-11-44-simple_add_remote_server/*.log

Grouping commands by name

pingYa and pingGoo are duplicated in last config. Next step, group this commands by name:

simple_group_commands.yaml
aliases:
  - &pingYa
    name: pingYa
    command: ping ya.ru
    extension: log
  - &pingGoo
    name: pingGoo
    command: ping goo.gl
    extension: log
​
sources:
  localhost:
    type: local
    commands:
      - *pingYa
      - *pingGoo
  remotehost:
    type: ssh
    host: 192.168.1.9
    connection:
      login: muxa
      key: /home/muxa/.ssh/id_rsa
    commands:
      - *pingYa
      - *pingGoo
daggy simple_group_commands.yaml 
22:19:18:690 | AppStatus  | Application     | Start receiver
22:19:18:691 | ConnStatus | localhost       | Server connected
22:19:18:691 | CommStatus | localhost       | pingGoo         | Command started
22:19:18:692 | CommStatus | localhost       | pingYa          | Command started
22:19:18:723 | ConnStatus | remotehost      | Server connected
22:19:18:723 | CommStatus | remotehost      | pingGoo         | Command started
22:19:18:723 | CommStatus | remotehost      | pingYa          | Command started
​

Add multiple servers

simple_multiple_servers.yaml
aliases:
  - &pingYa
    name: pingYa
    command: ping ya.ru
    extension: log
​
  - &pingGoo
    name: pingGoo
    command: ping goo.gl
    extension: log
​
  - &my_commands
    - *pingYa
    - *pingGoo
​
  - &ssh_auth
    login: muxa
    key: /home/muxa/.ssh/id_rsa
    
sources:
  localhost:
    type: local
    commands:
      *my_commands
  remotehost:
    host: 192.168.1.9
    type: ssh
    connection:
      *ssh_auth
    commands:
      *my_commands
  remotehost2:
    host: 192.168.1.10
    type: ssh
    connection:
      *ssh_auth
    commands:
      *my_commands
  remotehost3:
    host: 192.168.1.11
    type: ssh
    connection:
      *ssh_auth
    commands:
      *my_commands
daggy simple_multiple_servers.yaml 
23:24:25:118 | AppStatus  | Application     | Start receiver
23:24:25:119 | ConnStatus | localhost       | Server connected
23:24:25:119 | CommStatus | localhost       | pingGoo         | Command started
23:24:25:120 | CommStatus | localhost       | pingYa          | Command started
23:24:25:156 | ConnStatus | remotehost2     | Server connected
23:24:25:156 | CommStatus | remotehost2     | pingGoo         | Command started
23:24:25:156 | CommStatus | remotehost2     | pingYa          | Command started
23:24:25:156 | ConnStatus | remotehost3     | Server connected
23:24:25:156 | CommStatus | remotehost3     | pingGoo         | Command started
23:24:25:156 | CommStatus | remotehost3     | pingYa          | Command started
23:24:25:156 | ConnStatus | remotehost      | Server connected
23:24:25:156 | CommStatus | remotehost      | pingGoo         | Command started
23:24:25:157 | CommStatus | remotehost      | pingYa          | Command started
​
Previous
About Daggy
Next
How it works
Last updated 6 months ago
Contents
Download and Install Daggy
Fedora 29/30/31
Debian/Ubuntu
Arch Linux
MacOS
Windows 7/8/10
Build from sources
Configuration
Simple Data Aggregation Config
Add parallel command
Add remote server
Grouping commands by name
Add multiple servers