See www.zabbix.com for the official Zabbix site.

Docs/specs/ZBX-3531

From Zabbix.org
Jump to: navigation, search

XML export/import in 2.0

ZBX-3531

Status: Draft

Description

This specification deals with host, template, network map and screen XML import and export.

Summary

  • XML export format should be updated to support host interfaces, host discoveries etc.
  • Full validation of all tags and values should be performed. (postponed)
  • XML format and it's data does not change in any way during the 2.0 release cycle.

Specification TODO:

  • inventory missing --Richlv 18:45, 14 December 2011 (SAST)
added basic possible layout --Richlv 09:04, 27 January 2012 (SAST)
  • template screens missing --Richlv 18:45, 14 December 2011 (SAST)
  • for all entities, all tags must be listed (for example, all possible tags for items etc) --Richlv 21:36, 11 January 2012 (EET)
  • we should also decide on version number format --Vedmak 10:51, 12 January 2012 (SAST)
  • zabbix_import or zabbix_export ? 1.8 uses 'export', current suggestion - 'import' --Richlv 09:11, 27 January 2012 (SAST)
  • graph & trigger item reference tags should use 'item', not 'itemid' --Richlv 15:01, 27 January 2012 (SAST)
  • discoveries - should mention lld somehow --Richlv 15:01, 27 January 2012 (EET)

Specification

New format should be easy extensible (easy to add new elements in future).

One file is always used for export and supports all entities. Entities are placed in subtags, containing subentities, that are unique to them (for example, "hosts" is a separate subtag which has "items" subtag etc). Entities that may be linked to multiple other entities are placed on the lowest possible level that avoids duplication (for example, applications are on host or template level, triggers are on the upper level).

New format proposal:

<zabbix_import>
  <version>2.0</version>
  <timestamp>2013-03-13T13:13:13Z</timestamp>
  <templates>
    <template>
      <name>tpl1</name>
      <host>tpl1</host>
      <applications>
        <application>
          <name>app1</name>
        </application>
        <application>
          <name>app2</name>
        </application>
      </applications>
      <groups>
        <group>group1</group>
        <group>group2</group>
      </groups>
      <items>
        <item>
          <name>item1</name>
          <key>key_1</key>
          <interface_ref>if1</interface_ref>
        </item>
        <item>
          <name>item2</name>
          <key>key_2</key>
          <interface_ref>if2</interface_ref>
        </item>
      </items>
      <discoveries>
        <discovery>
          <name>disc1</name>
          <key>dkey_1</key>
          <item_prototypes>
            <item_prototype>
              <name>iprot1</name>
              <key>ipkey1</key>
            </item_prototype>
            <item_prototype>
              <name>iprot2</name>
              <key>ipkey2</key>
            </item_prototype>
          </item_prototypes>
        </discovery>
        <discovery>
          <name>disc2</name>
          <key>dkey_2</key>
          <item_prototypes>
            <item_prototype>
              <name>iprot3</name>
              <key>ipkey3</key>
            </item_prototype>
            <item_prototype>
              <name>iprot4</name>
              <key>ipkey4</key>
            </item_prototype>
          </item_prototypes>
        </discovery>
      </discoveries>
      <linked_templates>
        <linked_template>ltpl1</linked_template>
        <linked_template>ltpl2</linked_template>
      </linked_templates>
    </template>
  </templates>
  <hosts>
    <host>
      <name>tpl1</name>
      <host>tpl1</host>
      <interfaces>
        <interface>
          <interface_ref>if1</interface_ref>
          <ip>127.0.0.1</ip>
          <dns/>
          <main>1</main>
          <type>1</type>
        </interface>
        <interface>
          <interface_ref>if2</interface_ref>
          <ip>127.0.0.2</ip>
          <dns/>
          <main>0</main>
          <type>1</type>
        </interface>
      </interfaces>
      <applications>
        <aplication>
          <name>app1</name>
        </aplication>
        <aplication>
          <name>app2</name>
        </aplication>
      </applications>
      <groups>
        <group>group1</group>
        <group>group2</group>
      </groups>
      <items>
        <item>
          <name>item1</name>
          <key>key_1</key>
        </item>
        <item>
          <name>item2</name>
          <key>key_2</key>
        </item>
      </items>
      <discoveries>
        <discovery>
          <name>disc1</name>
          <key>dkey_1</key>
          <item_prototypes>
            <item_prototype>
              <name>iprot1</name>
              <key>ipkey1</key>
            </item_prototype>
            <item_prototype>
              <name>iprot2</name>
              <key>ipkey2</key>
            </item_prototype>
          </item_prototypes>
        </discovery>
        <discovery>
          <name>disc2</name>
          <key>dkey_2</key>
          <item_prototypes>
            <item_prototype>
              <name>iprot3</name>
              <key>ipkey3</key>
            </item_prototype>
            <item_prototype>
              <name>iprot4</name>
              <key>ipkey4</key>
            </item_prototype>
          </item_prototypes>
        </discovery>
      </discoveries>
      <inventory>
        <mode>1</mode>
        <some_field>value comes here</some_field>
      </inventory>
      <linked_templates>
        <linked_template>ltpl1</linked_template>
        <linked_template>ltpl2</linked_template>
      </linked_templates>
    </host>
  </hosts>
  <triggers>
    <name>trigger1</name>
    <expression>{host1:key_1.last(0)}&gt;0</expression>
    <dependencies>
      <dependency>
        <expression>{anyhost1:anykey1.last(0)}&gt;0</expression>
        <name>anytrigger1</name>
      </dependency>
      <dependency>
        <expression>{anyhost2:anykey2.last(0)}&gt;0</expression>
        <name>anytrigger2</name>
      </dependency>
    </dependencies>
    <description>trigger description</description>
  </triggers>
  <graphs>
    <name>graph1</name>
    <type>1</type>
    <yaxis_min_itemid_1>
      <host>host1</host>
      <key>key_1</key>
    </yaxis_min_itemid_1>
    <graph_items>
      <graph_item>
        <itemid>
          <host>host1</host>
          <key>key_1</key>
        </itemid>
        <yaxisside>1</yaxisside>
      </graph_item>
    </graph_items>
  </graphs>
  <trigger_prototypes>
    <trigger_prototype>
      <expression>{host1:key_1.last(0)}&gt;0</expression>
      <name>trigger1</name>
    </trigger_prototype>
    <trigger_prototype>
      <expression>{host1:key_1.last(0)}&gt;0</expression>
      <name>trigger1</name>
    </trigger_prototype>
  </trigger_prototypes>
  <graph_prototypes>
    <name>grpah2</name>
    <type>1</type>
    <yaxis_min_itemid_1>
      <host>host2</host>
      <key>key_2</key>
    </yaxis_min_itemid_1>
    <graph_items>
      <graph_item>
        <itemid>
          <host>host2</host>
          <key>ipkey_3</key>
        </itemid>
        <yaxisside>1</yaxisside>
      </graph_item>
    </graph_items>
  </graph_prototypes>
  <screens>
    <screen>
      <screenitems>
        <screenitem>
          <resourcetype>2</resourcetype>
          <resourceid>
            <name>Local network</name>
          </resourceid>
          <width>0</width>
          <height>0</height>
          <x>0</x>
          <y>0</y>
          <colspan>2</colspan>
          <rowspan>0</rowspan>
          <elements>0</elements>
          <valign>0</valign>
          <halign>0</halign>
          <style>0</style>
          <sort_triggers>0</sort_triggers>
        </screenitem>
        <screenitem>
          <resourcetype>0</resourcetype>
          <resourceid>
            <host>Zabbix server</host>
            <name>CPU Loads</name>
          </resourceid>
          <width>400</width>
          <height>100</height>
          <x>0</x>
          <y>1</y>
          <colspan>0</colspan>
          <rowspan>0</rowspan>
          <elements>0</elements>
          <valign>0</valign>
          <halign>0</halign>
          <style>0</style>
          <sort_triggers>0</sort_triggers>
        </screenitem>
      </screenitems>
      <name>Zabbix server</name>
      <hsize>2</hsize>
      <vsize>3</vsize>
    </screen>
  </screens>
</zabbix_import>
  • Templated items (and other entities) are not exported. This differs from 1.8 and must be listed in the upgrade notes
  • Upon import, if any entity can not be updated (for example, user doesn't have write permissions for it or non-changeable field for a templated entity is attempted to be changed), whole import should fail
  • One XML format supports all exportable entities. Export from individual pages will only export relevant entities, but for import one unified form is used
  • For template import, additional checkbox is added. If new templates are added, they are linked, but existing ones are left in place. If templates are updated, existing templates are unlinkd & cleared, new ones linked in
  • for XML format 2.0, any problem in the XML should make the whole import fail (that includes missing element, excess element, incorrect value etc). The 1.8 format import should be as lenient as possible, but detailed specification is out of scope for this task
  • Element attributes are not used when exporting and are not accepted upon import (for example, <item name="a"> )

Details

  • Date and time format used is YYYY-MM-DDThh:mm:ssZ according to ISO 8601
  • Date and time is the frontend local time and is never changed by the locale, translation etc
  • groups tag is used, as these may be both in host and template tag.
  • groups tag is on the upper level (same as hosts and templates), then groups are referenced from host and template blocks. This allows to specify group only once, and also allows to export host groups separately.
  • applications tag is inside host or template tags, not inside item tags - that should fix inability to export applications that have no items linked to them.
  • Multiple entities of the same kind are grouped - for example, all hosts are grouped under hosts tag, inside individual host all items are grouped under items tag etc. This eases parsing and validation.
  • triggers and custom graphs are on the upper level (same as hosts and templates) as they may draw items from multiple hosts
  • Default values are not exported (including empty values)
Should be discussed againg, now all are exported Vedmak 10:57, 13 March 2012 (SAST)
  • Unused values should not be exported - for example, trapper item should not have SNMP fields exported and text item should not have trends field exported (postponed)
  • For host inventory, if an item is linked to a field, this field is not exported. Upon import, incoming values for such fields are silently ignored
  • Value mapping for items is exported by name, not by ID. Upon import, if value mapping is missing, import should fail
  • Current XML has _1 notation for y axis scale values - this allows to add support for the second axis
  • Related entities for hosts that are on the top level are also exported. That includes triggers, custom graphs and user groups. If those entities are also specified separately, the resulting list is a merge. For example, if two host groups G1 and G2 are explicitly specified for export, and the exported host is member of groups G2 and G3, resulting XML will have 3 groups - G1, G2 and G3. It works the same way with network maps and images.
  • Import only adds or updates elements and linkages, it does not delete anything. For example, if we have a host with 3 groups and then import same host with only one group, nothing will be removed, host will have 3 groups. Same with other elements.
  • When import images and update existing images field "imagetype" is ignored, i.e. it must be impossible to change image type via XML.
  • Empty tags for items, triggers, graphs, host/template applications, discoveryRules, itemPrototypes, triggerPrototypes, graphPrototypes are meaningless i.e. it's same as it is missing. Other tags, for example item applications is meaningfull i.e. empty tag means no applications for item, missing tag means don't update applications.

Whitespace

  • Two spaces are used for indentation in export
  • XML export has an option to strip all non-mandatory whitespace (newlines and indentation)
  • Export never adds any extra whitespace in values or tags
    • <host >name</host>
    • <host> name </host>
  • Import preserves value whitespace as-is (both leading and trailing)

Sorting

Sorting is strictly defined for export. Maximal flexibility should be available for what is considered a valid import and that should be documented.

  • Exported entity order (import accepts any order):
    • Host groups (sorted alphabetically)
    • Templates (sorted alphabetically by technical name)
      • Template definition
      • Items (sorted alphabetically by name)
      • Applications (sorted alphabetically)
    • Hosts (sorted alphabetically by technical name)
      • Host definition
      • Interfaces (sorted by IP first, by DNS if IP is missing and by port afterwards; host being monitored by IP or DNS does not affect the sort order)
      • Items (sorted alphabetically by name)
      • Applications (sorted alphabetically)
    • Triggers (sorted alphabetically by name)
    • Trigger prototypes (sorted alphabetically by name)
    • Custom graphs (sorted alphabetically by name)
    • Custom graph prototypes (sorted alphabetically by name)
    • Network maps (sorted alphabetically by name)
      • Map definition
      • Map elements (sorted by ?)
      • Map links (sorted by ?)
    • Screens (sorted alphabetically by name)
      • Screen definition
      • Screen elements (sorted by type ID ?)

Field names should be sorted alphabetically.

References / Linkages

References and linkages must contain separate tag with unique fields for referenced element, even if unique field is only one. For example reference to host group from host can be defined that way:

 <host>
   <groups>
     <group>
       <name>Group1</name>
     </group>
   </groups>
 </host>

This approach is used for all entities, including groups (host and template), applications, template linkage etc

  • In hosts and templates, groups are referenced by name
  • Applications in item definitions are referenced by name
  • Host interfaces in XML get virtual IDs. Items reference these virtual IDs.
    • Interface ID tag is **interface_ref** both in interfaces and items
    • Interface references use format **ref<n>** as in **ref1**, **ref2** etc
  • All numbered references start at 1 for each host (they are not related to the database IDs)

Import form changes

  • One form will be same for all imports as xml format now supports all elements.
  • "Icon" and "background" rules are merged to one new rule "Images". "Images" is still unchecked by default and when "update existing" is checked confirmation alert is shown.
  • Add new rule for groups with only one option "add missing".
  • Add new rule for lld rules with both "add missing" and "update existing" options.
  • Rules for groups and images are shown only for super admins.
  • Some checkboxes will be checked and some unchecked depending on page we came from. If we click "Import" button in configuration->hosts, elements related to hosts will be checked and other unchecked, etc.
  • Add rules for template screens import control.

LLD export

All resources that are created by LLD or depend on LLD created elements won't be exported, i. e. all graphs and triggers that use LLD created items won't be exported.

Other formatting

  • Encoding of entities to HTML representation...
    • Greater than (>) is encoded as &gt;
    • Less than (<) is encoded as &lt;
  • API JSON only requires escaping doublequote, backslash and control characters, as per JSON specification

Compatibility

  • For Zabbix 2.0, export file format is changed to 2.0.

format must be as forward & backward compatible as possible. if future versions add support for some tags, that must still import in 2.0.0 - how to best achieve that while still having strict validation ? extension namespace seems messy.

First number is used to indicate that this version is fully compatible forwards & backwards. Second number indicates that some small changes might be there, but it still is compatible.

API changes

  • A new API method is added and XML export does no direct DB queries from the frontend.
  • Entities that should be available for export individually:
    • Hosts
    • Templates
    • Network maps
    • Screens
    • Host groups
    • Images
I suggest somehow change "host group" to "group" or something like that. It's strange that host groups can contain templates. --Vedmak 14:42, 7 February 2012 (SAST)
right - definitely. as the "details" above explain, that's exactly the reasoning for using just "group" in the xml itself. here "host group" was used only for clarity. one the other hand, just "group" might be confusing as well, as it's not always clear what type of group it might be (there are also user groups, for example) --Richlv 06:07, 8 February 2012 (SAST)

Database changes