Naplózás
(v0.03) |
a (update 2012) |
||
(2 szerkesztő 11 közbeeső változata nincs mutatva) | |||
15. sor: | 15. sor: | ||
#** "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában a 2.3.4.5-ről szokott belépni" |
#** "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában a 2.3.4.5-ről szokott belépni" |
||
#** "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában csak munkaidőben lép be (99%), a 2.3.4.5-ről (95%)" |
#** "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában csak munkaidőben lép be (99%), a 2.3.4.5-ről (95%)" |
||
+ | #*** A két utóbbi példa csak nagy jóindulattal nevezhető "eseménynek"; a naplóüzenetekben az "elemi eseményhez" egy összetett utófeldolgozás eredményét is csatoltuk. |
||
#** stb. |
#** stb. |
||
# Mi a célunk a naplózással? (Ez befolyásolja, mi lesz az "esemény".) |
# Mi a célunk a naplózással? (Ez befolyásolja, mi lesz az "esemény".) |
||
30. sor: | 31. sor: | ||
#** Pl.: ppp log, cron log, maillog, udev log, általában a "debug" típusú logok, ... |
#** Pl.: ppp log, cron log, maillog, udev log, általában a "debug" típusú logok, ... |
||
#* Szolgáltatások használati profiljának feltérképezése |
#* Szolgáltatások használati profiljának feltérképezése |
||
− | #** Pl.: dnscache query log |
+ | #** Pl.: dnscache query log, webszerver access-logja |
#* Accounting |
#* Accounting |
||
#** Pl.: CUPS page log |
#** Pl.: CUPS page log |
||
36. sor: | 37. sor: | ||
#** irclog, IM log |
#** irclog, IM log |
||
#* Csak úgy, majd csak jó lesz valamire |
#* Csak úgy, majd csak jó lesz valamire |
||
− | #** Pl.: mp3lejátszó által lejátszott számok naplója |
+ | #** Pl.: mp3-lejátszó által lejátszott számok naplója |
# Mihez fogunk kezdeni a naplóval? |
# Mihez fogunk kezdeni a naplóval? |
||
#* Csak legyen, egyszer majd talán belenézünk |
#* Csak legyen, egyszer majd talán belenézünk |
||
49. sor: | 50. sor: | ||
#** Stb. |
#** Stb. |
||
#* Policy-konformanciát ellenőrzünk (ha nem tudjuk vagy akarjuk eleve kikényszeríteni) |
#* Policy-konformanciát ellenőrzünk (ha nem tudjuk vagy akarjuk eleve kikényszeríteni) |
||
− | #** "Tényleg nem játszott éjfél után a gyerek?" :) |
+ | #** "Tényleg nem játszott éjfél után a gyerek?" |
− | #** "Igaz-e, hogy csak a munkával összefüggő weboldalakat nézegetnek az alkalmazottak?" |
+ | #** "Igaz-e, hogy főleg/csak a munkával összefüggő weboldalakat nézegetnek az alkalmazottak?" |
#** "Igaz-e, hogy Gipsz Jakab csak az ebédszünetben foglalkozott magánlevelezéssel?" |
#** "Igaz-e, hogy Gipsz Jakab csak az ebédszünetben foglalkozott magánlevelezéssel?" |
||
#** "Igaz-e, hogy Gáz Géza autóbuszvezető csak a kijelölt útvonalon közlekedik a gondjaira bízott Ikarus-szal, vagy taxizgat is vele?" |
#** "Igaz-e, hogy Gáz Géza autóbuszvezető csak a kijelölt útvonalon közlekedik a gondjaira bízott Ikarus-szal, vagy taxizgat is vele?" |
||
#** "Tényleg nem futtatott senki p2p filecserélőt a Schönherzben?" |
#** "Tényleg nem futtatott senki p2p filecserélőt a Schönherzben?" |
||
#* Gyanús mintákat keresünk benne |
#* Gyanús mintákat keresünk benne |
||
+ | #** [[logcheck]], [[multilogcheck]] |
||
#* Tároljuk, mert muszáj (törvényi előírás) |
#* Tároljuk, mert muszáj (törvényi előírás) |
||
#* Átadjuk az ügyfélnek |
#* Átadjuk az ügyfélnek |
||
75. sor: | 77. sor: | ||
* socklog+svlogd |
* socklog+svlogd |
||
* (klogd) |
* (klogd) |
||
− | ** /proc/kmsg |
+ | * új default a Debianban: rsyslog (még nem volt időm megnézni; potenciális házi feladat) |
+ | |||
+ | De persze minden második alkalmazás saját naplófájlt ír, és persze mindegyik más és más formátumban. Még a dátumformátum sem egyezik. |
||
+ | |||
+ | Néhány példa: |
||
+ | |||
+ | : '''samba''': |
||
+ | |||
+ | <pre> |
||
+ | [2007/10/20 22:56:13, 0] nmbd/nmbd_become_lmb.c:become_local_master_stage2(396) |
||
+ | ***** |
||
+ | |||
+ | Samba name server FOOBAR is now a local master browser for workgroup WORKGROUP on subnet 1.2.3.4 |
||
+ | |||
+ | ***** |
||
+ | [2007/10/20 22:58:26, 1] lib/util_sock.c:open_socket_out(896) |
||
+ | timeout connecting to 2.3.4.5:139 |
||
+ | </pre> |
||
+ | |||
+ | * Egy esemény több sor, még csak nem is mindig ugyanannyi. |
||
+ | ** Hogyan fésülünk össze két ilyen logot? |
||
+ | * A dátumnál a lexikografikus sorrend megegyezik az időbelivel, ez jó. |
||
+ | * Fejlesztőknek szóló napló (megmondja, a forráskód melyik fájljának melyik sorában generálják az adott üzenetet). |
||
+ | ** Ez az üzemeltetőnek nem feltétlenül hasznos. |
||
+ | |||
+ | Hogyan lehetett volna jobban? |
||
+ | |||
+ | <pre> |
||
+ | 2007-10-20 22:56:13 foobar nmbd: info: becomemaster: stage2: FOOBAR now local master for WORKGROUP on 1.2.3.4 |
||
+ | 2007-10-20 22:58:26 foobar nmbd: warn: remoteannounce: timeout connecting to 2.3.4.5:139 |
||
+ | </pre> |
||
+ | |||
+ | : '''squid''' access.log: |
||
+ | |||
+ | <pre> |
||
+ | 1193373264.480 43 192.168.0.4 TCP_MISS/200 1334 GET cache_object://192.168.0.4/server_list - NONE/- text/plain |
||
+ | 1193373264.536 1 192.168.0.4 TCP_MISS/200 1334 GET cache_object://192.168.0.4/server_list - NONE/- text/plain |
||
+ | </pre> |
||
+ | |||
+ | * Dátum helyett UNIX epoch seconds, pont, ms. |
||
+ | ** A milliszekundum-mezőben szerencsére megvannak a kezdő nullák a 100-nál kisebb értékek előtt (ez sajnos nem magától értetődő). |
||
+ | ** Rendezhető és összefésülhető, de nem túl olvasható. |
||
+ | * Mezőkre osztott sor, de a mezők száma nem állandó (vagy nem fix számú szóköz választja el őket egymástól). |
||
+ | ** Az awk-nál egyszerűbb eszközökkel (pl. <tt>cut</tt>) való feldolgozás nehézkes. |
||
+ | |||
+ | : '''squid''' cache.log: |
||
+ | |||
+ | <pre> |
||
+ | 2007/10/26 06:26:03| helperOpenServers: Starting 10 'adzapper.wrapper' processes |
||
+ | 2007/10/26 06:27:49| CACHEMGR: <unknown>@192.168.0.4 requesting 'counters' |
||
+ | </pre> |
||
+ | |||
+ | * Még egy alkalmazáson belül sem egységes a dátumformátum... |
||
+ | ** És minek oda az a pipe karakter? |
||
+ | * De legalább nagyjából használható ''tag'' ("címke") van a sor elején. |
||
+ | |||
+ | : '''rinetd.log''': |
||
+ | |||
+ | <pre> |
||
+ | 21/Sep/2007:19:54:43 127.0.0.1 127.0.0.1 111 192.168.0.4 111 44 72 done-remote-closed |
||
+ | </pre> |
||
+ | |||
+ | * Újabb kiválóan rendezhető dátumformátum. |
||
+ | * Viszont TAB van a mezők között, úgyhogy a <tt>cut</tt> is boldogul vele. |
||
+ | |||
+ | : '''apache''' "combined" accesslog: |
||
+ | |||
+ | <pre> |
||
+ | lj511257.crawl.yahoo.net - - [02/Oct/2007:10:53:05 +0200] "GET /~korn/nws2002/djbdns-DNS-BIND-nelkul.png/Slide05.html HTTP/1.0" 200 1215 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" |
||
+ | </pre> |
||
+ | |||
+ | * Újabb dátumszörny, bár ebben legalább benne van, melyik időzónában kell értelmezni. |
||
+ | ** Rendezni szeretnénk? Mi sem egyszerűbb! <tt>sort -t ' ' -k 4.9,4.12n -k 4.5,4.7M -k 4.2,4.3n -k 4.14,4.15n -k 4.17,4.18n -k 4.20,4.21n</tt> (bár többféle időzónából származó logokra ez sem elég). |
||
+ | * Némelyik mező belsejében lehet szóköz, bár szerencsére főleg az utolsóban (User-Agent) jellemző. |
||
+ | |||
+ | : '''apache''' errorlog: |
||
+ | |||
+ | <pre> |
||
+ | [Tue Oct 2 12:53:54 2007] [error] [client 65.92.126.129] File does not exist: /var/www/links |
||
+ | </pre> |
||
+ | |||
+ | * Miért is lenne ugyanolyan formátumú a dátum? |
||
+ | |||
+ | : '''syslogd''': |
||
+ | |||
+ | <pre> |
||
+ | Oct 25 23:04:33 kernel: [810505.320000] tap2: received packet with own address as source address |
||
+ | </pre> |
||
+ | |||
+ | * Időzóna? |
||
+ | ** Sőt: évszám?! |
||
+ | |||
+ | Na de nézzük meg részletesebben a syslogd-t: |
||
+ | |||
+ | === syslogd === |
||
+ | |||
+ | * Történelmi okokból ez a legelterjedtebb. |
||
+ | ** bár Debianon mára kiszorította az rsyslog, ami viszont 100%-ig kompatibilis akar lenni vele. |
||
+ | * "facility" és "severity" alapján tudja célobjektumokba (l. alant) válogatni az üzeneteket |
||
+ | * tud figyelni: |
||
+ | ** unix domain socketen (/dev/log) |
||
+ | *** egyszerre csak egyen, chroot esetén gond lehet (a linuxos és az openbsd-s syslogd tud több socketet is kezelni, a régebbiek általában nem) |
||
+ | **** <tt>mount --bind</tt> a socketet tartalmazó könyvtárra jó workaround lehet; a /dev/log ebben az esetben symlink kell, hogy legyen. |
||
+ | **** Hátrány: a naplóban nem fog látszani, melyik chrootból jött egy-egy üzenet. |
||
+ | ** udp socketen (514-es port) |
||
+ | * célobjektumok: |
||
+ | ** file (választhatóan pufferelt vagy sync) |
||
+ | *** nem feltétlenül hozza létre! |
||
+ | ** udp |
||
+ | ** named pipe (pl. <tt>/dev/xconsole</tt> |
||
+ | ** bejelentkezett felhasználó terminálja |
||
Példa /etc/syslog.conf-ra: |
Példa /etc/syslog.conf-ra: |
||
162. sor: | 164. sor: | ||
</pre> |
</pre> |
||
− | to be continued... |
+ | Problémák: |
+ | |||
+ | * Több implementáció, közös gyökerekkel, de eltérő fejlődéssel => a felszín alatt megbúvó inkompatibilitások. |
||
+ | ** Pl. van olyan verzió, amelyiknek nem mindegy, hogy a configfile-ban szóközök vagy TABok vannak. |
||
+ | * Nem minden implementáció hozza létre a logfile-okat, ha nem léteznek - némelyik egyszerűen nem logolja az adott fájlba írandó üzeneteket, ha nem létezik a fájl. |
||
+ | * Szűkös konfigurációs lehetőségek: |
||
+ | ** Csak facility és severity/priority alapján lehet szétválogatni a logokat. |
||
+ | ** Pedig rengeteg daemon.* üzenet van, jó lenne ezeket külön logolni. |
||
+ | * A hagyományos dátumformátum rossz: |
||
+ | ** "Oct 23 12:03:42" |
||
+ | ** Nem rendezhető lexikografikusan => logok összefésülése körülményes |
||
+ | ** Nincs benne évszám |
||
+ | ** Nincs benne az időzóna |
||
+ | ** Még a dátumban sem állandó a mezők száma ("Jan 1" vs. "Jan 10": "Jan__1" - három mező, "Jan_10" - két mező) |
||
+ | * Az udp-s logolás megbízhatatlan, üzenetek veszhetnek el. |
||
+ | * Nincs hitelesítés, hálózati naplózás esetén könnyű hamis naplóüzeneteket beszúrni. |
||
+ | * A sorok struktúrája változó, nehézkes az egységes kezelésük. |
||
+ | * Csak egy IP-nek van hely a logolt sor formátumában, így proxy esetén ott mindig a proxy IP-je lesz, |
||
+ | ** hacsak a proxy bele nem írja az üzenet szövegébe a kliens IP-jét; |
||
+ | ** így viszont változó számú IP-prefix kerülhet a sorok elejére. |
||
+ | * Rotáció problémás (nem beépített) |
||
+ | ** PID-lottó... |
||
+ | * Egy esemény egy sor, ráadásul általában max. 1024 karakter (ennél egy java stack trace kapásból több). |
||
+ | * "Tömörítés": ha ugyanaz az üzenet sokszor ismétlődik, a syslogd nem logolja, csak egy idő után azt, hogy hány ismétlődés volt. |
||
+ | ** Így pl. nem nagyon tudjuk, az egyes ismétlődések mikor voltak. |
||
+ | * Általában rootként futtatják. |
||
+ | ** pl. mert egyébként HUP signal (logok újramegnyitása) után esetleg már nem tudna írni a logokba, mert csak a root számára írhatók. |
||
+ | * Implementációfüggő, hogy melyik mezők szerepelnek az udp-n átküldött üzenetekben: |
||
+ | ** pl. Solaris: <13>Oct 23 12:34:56 last message repeated 15 times. |
||
+ | ** Linux: <13>last message repeated 4 times |
||
+ | ** Cisco: 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 |
||
+ | *** Ez nem is hasonlít egy syslogd-üzenetre, így a syslogd elécsap egy saját fejlécet, és így logolja: Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16 |
||
+ | |||
+ | === syslog-ng === |
||
+ | |||
+ | * Hasonlít a syslogra, annak a hibáit próbálja javítani. |
||
+ | * Több generációváltáson van túl; az egyes generációk részben inkompatibilisek egymással. |
||
+ | * Tud SOCK_STREAM és SOCK_DGRAM típusú Unix domain socketet is (legalábbis Linuxon) |
||
+ | * Tud TCP-n is figyelni |
||
+ | ** Nemcsak megbízható átvitel, hanem |
||
+ | ** SSL-esíthető is |
||
+ | * Testreszabható a naplózott sorok formátuma (elsősorban a mezők sorrendje: dátum, hoszt, üzenet) |
||
+ | * Viszonylag egyszerűen tudjuk sok különböző gép logjait fogadni (nem kell mindegyikhez külön konfiguráció: a naplófájl neve függhet a kliens IP-jétől vagy hosztnevétől) |
||
+ | * Tud több Unix domain socketen figyelni, így: |
||
+ | <pre> |
||
+ | source local { unix-stream("/dev/log"); internal(); }; |
||
+ | source jail1 { unix-stream("/jail/dns/dev/log"); }; |
||
+ | source jail2 { unix-stream("/jail/www/dev/log"); }; |
||
+ | </pre> |
||
+ | * és így: |
||
+ | <pre> |
||
+ | source local { |
||
+ | unix-stream("/dev/log"); |
||
+ | internal(); |
||
+ | unix-stream("/jail/dns/dev/log"); |
||
+ | unix-stream("/jail/www/dev/log"); |
||
+ | }; |
||
+ | </pre> |
||
+ | |||
+ | Hátrányok: |
||
+ | |||
+ | * Bonyolult. |
||
+ | * Statikusan linkelik a <tt>GLib</tt>hez, ami egy viszonylag nagy library - dinamikusan viszont nem lehet, mert a /usr alá szokás telepíteni a GLibet, így a syslog-ng indulásakor esetleg nem áll rendelkezésre. |
||
+ | * Aránylag összetett konfigurációs szintaxis, ráadásul szószátyár is és nem is túl átlátható. |
||
+ | * DNS-feloldáskor blokkolódik (ez 2007. körül volt így; lehet, hogy már javították). |
||
+ | * A reguláris kifejezések illesztése lassú, az alternatíva viszont bonyolult: |
||
+ | |||
+ | Reguláris kifejezéssel: |
||
+ | <pre> |
||
+ | filter f_xntp_filter_no_regexp { |
||
+ | # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); |
||
+ | program("xntpd") and |
||
+ | match("time error .* is way too large .* set clock manually"); |
||
+ | }; |
||
+ | </pre> |
||
+ | Ehelyett így kéne: |
||
+ | <pre> |
||
+ | filter f_xntp_filter_no_regexp { |
||
+ | # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); |
||
+ | program("xntpd") and |
||
+ | match("time error") and match("is way too large") and match("set clock manually"); |
||
+ | |||
+ | }; |
||
+ | </pre> |
||
+ | Ez egyrészt nem ugyanazt jelenti (illeszkedik arra is, hogy "'set clock manually' is way too large for a time error", bár ez itt nem túl nagy gond), másrészt nem kifejezetten karbantartásbarát. |
||
+ | |||
+ | Példakonfiguráció (''ha valakinek van kedve magyarul felkommentezni, ne habozzon''): |
||
+ | |||
+ | <pre> |
||
+ | #---------------------------------------------------------------------- |
||
+ | # Program: syslog-ng.conf |
||
+ | # Notes: Embedded most of the manual notes within the configuration |
||
+ | # file. The original manual can be found at: |
||
+ | # |
||
+ | # http://www.balabit.com/products/syslog_ng/reference/book1.html |
||
+ | # http://www.campin.net/syslog-ng/faq.html |
||
+ | # |
||
+ | # Many people may find placing all of this information in a |
||
+ | # configuration file a bit redundant, but I have found that |
||
+ | # with a little bit of extra comments and reference, |
||
+ | # maintaining these beasties is much easier. |
||
+ | # |
||
+ | # This particular log file was taken from the examples that |
||
+ | # are given at the different web sites, and made to emulate |
||
+ | # the logs of a Mandrake Linux system as much as possible. |
||
+ | # Of course, Unix is Unix, is Linux. It should be generic |
||
+ | # enough for any Unix system. |
||
+ | #---------------------------------------------------------------------- |
||
+ | # 16-Mar-03 - REP - Added some extra definitions to the file. |
||
+ | # 15-Mar-03 - REP - Added back the comments on filtering. |
||
+ | # 27-Feb-03 - REP - Further modified for local environment. |
||
+ | # 27-Feb-03 - REP - Updated for new configuration and version 1.6.0 |
||
+ | # 12-Dec-02 - REP - Continued updates for writing to databases. |
||
+ | # 30-Nov-02 - REP - Initial creation for testing. |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Options |
||
+ | #---------------------------------------------------------------------- |
||
+ | # |
||
+ | # Name Values Description |
||
+ | # ------------------------- ------- ------------------------------------ |
||
+ | # bad_hostname reg exp A regexp which matches hostnames |
||
+ | # which should not be taken as such. |
||
+ | # chain_hostnames y/n Enable or disable the chained |
||
+ | # hostname format. |
||
+ | # create_dirs y/n Enable or disable directory creation |
||
+ | # for destination files. |
||
+ | # dir_group groupid |
||
+ | # dir_owner userid |
||
+ | # dir_perm perm |
||
+ | # dns_cache y/n Enable or disable DNS cache usage. |
||
+ | # dns_cache_expire num Number of seconds while a successful |
||
+ | # lookup is cached. |
||
+ | # dns_cache_expire_failed num Number of seconds while a failed |
||
+ | # lookup is cached. |
||
+ | # dns_cache_size num Number of hostnames in the DNS cache. |
||
+ | # gc_busy_threshold num Sets the threshold value for the |
||
+ | # garbage collector, when syslog-ng is |
||
+ | # busy. GC phase starts when the number |
||
+ | # of allocated objects reach this |
||
+ | # number. Default: 3000. |
||
+ | # gc_idle_threshold num Sets the threshold value for the |
||
+ | # garbage collector, when syslog-ng is |
||
+ | # idle. GC phase starts when the number |
||
+ | # of allocated objects reach this |
||
+ | # number. Default: 100. |
||
+ | # group groupid |
||
+ | # keep_hostname y/n Enable or disable hostname rewriting. |
||
+ | # This means that if the log entry had |
||
+ | # been passed through at least one other |
||
+ | # logging system, the ORIGINAL hostname |
||
+ | # will be kept attached to the log. |
||
+ | # Otherwise the last logger will be |
||
+ | # considered the log entry owner and |
||
+ | # the log entry will appear to have |
||
+ | # come from that host. |
||
+ | # log_fifo_size num The number of lines fitting to the |
||
+ | # output queue |
||
+ | # log_msg_size num Maximum length of message in bytes. |
||
+ | # long_hostnames on/off This options appears to only really |
||
+ | # have an affect on the local system. |
||
+ | # which removes the source of the log. |
||
+ | # As an example, normally the local |
||
+ | # logs will state src@hostname, but |
||
+ | # with this feature off, the source |
||
+ | # is not reported. |
||
+ | # mark num The number of seconds between two |
||
+ | # MARK lines. NOTE: not implemented |
||
+ | # yet. |
||
+ | # owner userid |
||
+ | # perm perm |
||
+ | # stats num The number of seconds between two |
||
+ | # STATS. |
||
+ | # sync num The number of lines buffered before |
||
+ | # written to file |
||
+ | # time_reap num The time to wait before an idle |
||
+ | # destination file is closed. |
||
+ | # time_reopen num The time to wait before a died |
||
+ | # connection is reestablished |
||
+ | # use_dns y/n Enable or disable DNS usage. |
||
+ | # syslog-ng blocks on DNS queries, |
||
+ | # so enabling DNS may lead to a |
||
+ | # Denial of Service attack. To |
||
+ | # prevent DoS, protect your |
||
+ | # syslog-ng network endpoint with |
||
+ | # firewall rules, and make sure that |
||
+ | # all hosts, which may get to |
||
+ | # syslog-ng is resolvable. |
||
+ | # use_fqdn y/n Add Fully Qualified Domain Name |
||
+ | # instead of short hostname. |
||
+ | # use_time_recvd y/n Use the time a message is |
||
+ | # received instead of the one |
||
+ | # specified in the message. |
||
+ | #---------------------------------------------------------------------- |
||
+ | # 15-Mar-03 - REP - Since some of the clocks are not quite right, we |
||
+ | # are going to go ahead and just use the local time |
||
+ | # as the master time. |
||
+ | # 12-Mar-03 - REP - We have taken a few configuration options from the |
||
+ | # newer Solaris configuration because some of the |
||
+ | # reasons are valid for us as well. We have increased |
||
+ | # the log_msg_size and log_fifo_size to increase the |
||
+ | # amount of buffering that we do. While for most |
||
+ | # systems this may not have a noticeable affect, it |
||
+ | # will for systems that are at the end of a lot of |
||
+ | # logging systems. |
||
+ | # 20-Dec-02 - REP - Changed the stat() time from the default of 10 |
||
+ | # minutes to once an hour. |
||
+ | #---------------------------------------------------------------------- |
||
+ | options |
||
+ | { |
||
+ | chain_hostnames(no); |
||
+ | create_dirs (no); |
||
+ | dir_perm(0755); |
||
+ | dns_cache(yes); |
||
+ | keep_hostname(yes); |
||
+ | log_fifo_size(2048); |
||
+ | log_msg_size(8192); |
||
+ | long_hostnames(on); |
||
+ | perm(0644); |
||
+ | stats(3600); |
||
+ | sync(0); |
||
+ | time_reopen (10); |
||
+ | use_dns(yes); |
||
+ | use_fqdn(yes); |
||
+ | }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Sources |
||
+ | #---------------------------------------------------------------------- |
||
+ | # |
||
+ | # fifo/pipe - The pipe driver opens a named pipe with the |
||
+ | # specified name, and listens for messages. It's |
||
+ | # used as the native message getting protocol on |
||
+ | # HP-UX. |
||
+ | # file - Usually the kernel presents its messages in a |
||
+ | # special file (/dev/kmsg on BSDs, /proc/kmsg on |
||
+ | # Linux), so to read such special files, you'll need |
||
+ | # the file() driver. Please note that you can't use |
||
+ | # this driver to follow a file like tail -f does. |
||
+ | # internal - All internally generated messages "come" from this |
||
+ | # special source. If you want warnings, errors and |
||
+ | # notices from syslog-ng itself, you have to include |
||
+ | # this source in one of your source statements. |
||
+ | # sun-streams - Solaris uses its STREAMS API to send messages to |
||
+ | # the syslogd process. You'll have to compile |
||
+ | # syslog-ng with this driver compiled in (see |
||
+ | # ./configure --help). |
||
+ | # |
||
+ | # Newer versions of Solaris (2.5.1 and above), uses a |
||
+ | # new IPC in addition to STREAMS, called door to |
||
+ | # confirm delivery of a message. Syslog-ng supports |
||
+ | # this new IPC mechanism with the door() option. |
||
+ | # |
||
+ | # The sun-streams() driver has a single required |
||
+ | # argument, specifying the STREAMS device to open and |
||
+ | # a single option. |
||
+ | # tcp/udp - These drivers let you receive messages from the |
||
+ | # network, and as the name of the drivers show, you |
||
+ | # can use both UDP and TCP as transport. |
||
+ | # |
||
+ | # UDP is a simple datagram oriented protocol, which |
||
+ | # provides "best effort service" to transfer |
||
+ | # messages between hosts. It may lose messages, and |
||
+ | # no attempt is made to retransmit such lost |
||
+ | # messages at the protocol level. |
||
+ | # |
||
+ | # TCP provides connection-oriented service, which |
||
+ | # basically means a flow-controlled message pipeline. |
||
+ | # In this pipeline, each message is acknowledged, and |
||
+ | # retransmission is done for lost packets. Generally |
||
+ | # it's safer to use TCP, because lost connections can |
||
+ | # be detected, and no messages get lost, but |
||
+ | # traditionally the syslog protocol uses UDP. |
||
+ | # |
||
+ | # None of tcp() and udp() drivers require positional |
||
+ | # parameters. By default they bind to 0.0.0.0:514, |
||
+ | # which means that syslog-ng will listen on all |
||
+ | # available interfaces, port 514. To limit accepted |
||
+ | # connections to one interface only, use the |
||
+ | # localip() parameter as described below. |
||
+ | # |
||
+ | # Options: |
||
+ | # |
||
+ | # Name Type Description Default |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # ip or local ip string The IP address to bind to. Note 0.0.0.0 |
||
+ | # that this is not the address |
||
+ | # where messages are accepted |
||
+ | # from. |
||
+ | # keep-alive y/n Available for tcp() only, and yes |
||
+ | # specifies whether to close |
||
+ | # connections upon the receival |
||
+ | # of a SIGHUP signal. |
||
+ | # max-connections number Specifies the maximum number of 10 |
||
+ | # simultaneous connections. |
||
+ | # port or local port number The port number to bind 514 |
||
+ | # to. |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # |
||
+ | # unix-stream - unix-dgram - These two drivers behave similarly: |
||
+ | # they open the given AF_UNIX socket, and start |
||
+ | # listening on them for messages. unix-stream() is |
||
+ | # primarily used on Linux, and uses SOCK_STREAM |
||
+ | # semantics (connection oriented, no messages are |
||
+ | # lost), unix-dgram() is used on BSDs, and uses |
||
+ | # SOCK_DGRAM semantics, this may result in lost |
||
+ | # local messages, if the system is overloaded. |
||
+ | # |
||
+ | # To avoid denial of service attacks when using |
||
+ | # connection-oriented protocols, the number of |
||
+ | # simultaneously accepted connections should be |
||
+ | # limited. This can be achieved using the |
||
+ | # max-connections() parameter. The default value of |
||
+ | # this parameter is quite strict, you might have to |
||
+ | # increase it on a busy system. |
||
+ | # |
||
+ | # Both unix-stream and unix-dgram has a single |
||
+ | # required positional argument, specifying the |
||
+ | # filename of the socket to create, and several |
||
+ | # optional parameters. |
||
+ | # |
||
+ | # Options: |
||
+ | # |
||
+ | # Name Type Description Default |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # group string Set the gid of the socket. root |
||
+ | # keep-alive y/n Selects whether to keep yes |
||
+ | # connections opened when |
||
+ | # syslog-ng is restarted, can be |
||
+ | # used only with unix-stream(). |
||
+ | # max-connections numb Limits the number of 10 |
||
+ | # simultaneously opened |
||
+ | # connections. Can be used only |
||
+ | # with unix-stream(). |
||
+ | # owner string Set the uid of the socket. root |
||
+ | # perm num Set the permission mask. For 0666 |
||
+ | # octal numbers prefix the number |
||
+ | # with '0', e.g. use 0755 for |
||
+ | # rwxr-xr-x. |
||
+ | #---------------------------------------------------------------------- |
||
+ | # Notes: For Linux systems (and especially RedHat derivatives), |
||
+ | # they have a second logging process for kernel messages. |
||
+ | # This source is /proc/kmsg. If you are running this on a |
||
+ | # system that is not Linux, then the source entry for this |
||
+ | # should be removed. |
||
+ | # |
||
+ | # It seems that there is some performance questions related |
||
+ | # to what type of source stream should be used for Linux |
||
+ | # boxes. The documentation states the /dev/log should use |
||
+ | # unix-stream, but from the mailing list it has been |
||
+ | # strongly suggested that unix-dgram be used. |
||
+ | # |
||
+ | # WARNING: TCP wrappers has been enabled for this system, and unless |
||
+ | # you also place entries in /etc/hosts.allow for each of the |
||
+ | # devices that will be delivering logs via TCP, you will |
||
+ | # NOT receive the logs. |
||
+ | # |
||
+ | # Also note that if there is any form of a local firewall, |
||
+ | # this will also need to be altered such that the incoming |
||
+ | # and possibly outgoing packets are allowed by the firewall |
||
+ | # rules. |
||
+ | #---------------------------------------------------------------------- |
||
+ | # There has been a lot of debate on whether everything should be put |
||
+ | # to a single source, or breakdown all the sources into individual |
||
+ | # streams. The greatest flexibility would be in many, but the most |
||
+ | # simple is the single. Since we wrote this file, we have chosen the |
||
+ | # route of maximum flexibility. |
||
+ | # |
||
+ | # For those of you that like simplicity, this could have also been |
||
+ | # done as the follows: |
||
+ | # |
||
+ | # source src |
||
+ | # { |
||
+ | # internal(); |
||
+ | # pipe("/proc/kmsg" log_prefix("kernel: ")); |
||
+ | # tcp(ip(127.0.0.1) port(4800) keep-alive(yes)); |
||
+ | # udp(); |
||
+ | # unix-stream("/dev/log"); |
||
+ | # }; |
||
+ | # |
||
+ | # You would also have to change all the log statements to only |
||
+ | # reference the now single source stream. |
||
+ | #---------------------------------------------------------------------- |
||
+ | # 16-Mar-03 - REP - The default number of allowed TCP connects is set |
||
+ | # very low for a logserver. This value should only |
||
+ | # be set greater than the default for servers that |
||
+ | # will actually be serving that many systems. |
||
+ | #---------------------------------------------------------------------- |
||
+ | source s_dgram |
||
+ | { unix-dgram("/dev/log"); }; |
||
+ | |||
+ | source s_internal |
||
+ | { internal(); }; |
||
+ | |||
+ | source s_kernel |
||
+ | { pipe("/proc/kmsg" log_prefix("kernel: ")); }; |
||
+ | |||
+ | source s_tcp |
||
+ | { tcp(port(4800) keep-alive(yes) max_connections(100)); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Destinations |
||
+ | #---------------------------------------------------------------------- |
||
+ | # |
||
+ | # fifo/pipe - This driver sends messages to a named pipe like |
||
+ | # /dev/xconsole |
||
+ | # |
||
+ | # The pipe driver has a single required parameter, |
||
+ | # specifying the filename of the pipe to open, and |
||
+ | # no options. |
||
+ | # file - The file driver is one of the most important |
||
+ | # destination drivers in syslog-ng. It allows you to |
||
+ | # output messages to the named file, or as you'll see |
||
+ | # to a set of files. |
||
+ | # |
||
+ | # The destination filename may include macros which |
||
+ | # gets expanded when the message is written, thus a |
||
+ | # simple file() driver may result in several files |
||
+ | # to be created. Macros can be included by prefixing |
||
+ | # the macro name with a '$' sign (without the quotes), |
||
+ | # just like in Perl/PHP. |
||
+ | # |
||
+ | # If the expanded filename refers to a directory |
||
+ | # which doesn't exist, it will be created depending |
||
+ | # on the create_dirs() setting (both global and a per |
||
+ | # destination option) |
||
+ | # |
||
+ | # WARNING: since the state of each created file must |
||
+ | # be tracked by syslog-ng, it consumes some memory |
||
+ | # for each file. If no new messages are written to a |
||
+ | # file within 60 seconds (controlled by the time_reap |
||
+ | # global option), it's closed, and its state is freed. |
||
+ | # |
||
+ | # Exploiting this, a DoS attack can be mounted against |
||
+ | # your system. If the number of possible destination |
||
+ | # files and its needed memory is more than the amount |
||
+ | # your logserver has. |
||
+ | # |
||
+ | # The most suspicious macro is $PROGRAM, where the |
||
+ | # possible variations is quite high, so in untrusted |
||
+ | # environments $PROGRAM usage should be avoided. |
||
+ | # |
||
+ | # Macros: |
||
+ | # |
||
+ | # Name Description |
||
+ | # ----------------- ----------------------------------------------- |
||
+ | # DATE Date of the transaction. |
||
+ | # DAY The day of month the message was sent. |
||
+ | # FACILITY The name of the facility, the message is tagged |
||
+ | # as coming from. |
||
+ | # FULLDATE Long form of the date of the transaction. |
||
+ | # FULLHOST Full hostname of the system that sent the log. |
||
+ | # HOST The name of the source host where the message |
||
+ | # is originated from. If the message traverses |
||
+ | # several hosts, and chain_hostnames() is on, |
||
+ | # the first one is used. |
||
+ | # HOUR The hour of day the message was sent. |
||
+ | # ISODATE Date in ISO format. |
||
+ | # MIN The minute the message was sent. |
||
+ | # MONTH The month the message was sent. |
||
+ | # MSG or MESSAGE Message contents. |
||
+ | # PRIORITY or LEVEL The priority of the message. |
||
+ | # PROGRAM The name of the program the message was sent by. |
||
+ | # SEC The second the message was sent. |
||
+ | # TAG The priority and facility encoded as a 2 digit |
||
+ | # hexadecimal number. |
||
+ | # TZ The time zone or name or abbreviation. e.g. 'PDT' |
||
+ | # TZOFFSET The time-zone as hour offset from GMT. e.g. |
||
+ | # '-0700' |
||
+ | # WEEKDAY The 3-letter name of the day of week the |
||
+ | # message was sent, e.g. 'Thu'. |
||
+ | # YEAR The year the message was sent. Time expansion |
||
+ | # macros can either use the time specified in |
||
+ | # the log message, e.g. the time the log message |
||
+ | # is sent, or the time the message was received |
||
+ | # by the log server. This is controlled by the |
||
+ | # use_time_recvd() option. |
||
+ | # ----------------- ----------------------------------------------- |
||
+ | # |
||
+ | # Options: |
||
+ | # |
||
+ | # Name Type Description Default |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # compress y/n Compress the resulting logfile global |
||
+ | # using zlib. NOTE: this is not setting |
||
+ | # implemented as of 1.3.14. |
||
+ | # reate_dirs y/n Enable creating non-existing no |
||
+ | # directories. |
||
+ | # dir_perm num The permission mask of 0600 |
||
+ | # directories created by |
||
+ | # syslog-ng. Log directories are |
||
+ | # only created if a file after |
||
+ | # macro expansion refers to a |
||
+ | # non-existing directory, and dir |
||
+ | # creation is enabled using |
||
+ | # create_dirs(). |
||
+ | # encrypt y/n Encrypt the resulting file. global |
||
+ | # NOTE: this is not implemented as setting |
||
+ | # of 1.3.14. |
||
+ | # fsync y/n Forces an fsync() call on the |
||
+ | # destination fd after each write. |
||
+ | # Note: this may degrade |
||
+ | # performance seriously |
||
+ | # group string Set the group of the created root |
||
+ | # filename to the one specified. |
||
+ | # log_fifo_size num The number of entries in the global |
||
+ | # output fifo. setting |
||
+ | # owner string Set the owner of the created root |
||
+ | # filename to the one specified. |
||
+ | # perm num The permission mask of the file 0600 |
||
+ | # if it is created by syslog-ng. |
||
+ | # remove_if_older num If set to a value higher than 0, 0 |
||
+ | # before writing to a file, |
||
+ | # syslog-ng checks whether this |
||
+ | # file is older than the specified |
||
+ | # amount of time (specified in |
||
+ | # seconds). If so, it removes the |
||
+ | # existing file and the line to |
||
+ | # be written is the first line in |
||
+ | # a new file with the same name. |
||
+ | # In combination with e.g. the |
||
+ | # $WEEKDAY macro, this is can be |
||
+ | # used for simple log rotation, |
||
+ | # in case not all history need to |
||
+ | # be kept. |
||
+ | # sync_freq num The logfile is synced when this global |
||
+ | # number of messages has been setting |
||
+ | # written to it. |
||
+ | # template string Specifies a template which |
||
+ | # specifies the logformat to be |
||
+ | # used in this file. The possible |
||
+ | # macros are the same as in |
||
+ | # destination filenames. |
||
+ | # template_escape y/n Turns on escaping ' and " in yes |
||
+ | # templated output files. It is |
||
+ | # useful for generating SQL |
||
+ | # statements and quoting string |
||
+ | # contents so that parts of your |
||
+ | # log message don't get |
||
+ | # interpreted as commands to the |
||
+ | # SQL server. |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # |
||
+ | # program - This driver fork()'s executes the given program with |
||
+ | # the given arguments and sends messages down to the |
||
+ | # stdin of the child. |
||
+ | # |
||
+ | # The program driver has a single required parameter, |
||
+ | # specifying a program name to start and no options. |
||
+ | # The program is executed with the help of the current |
||
+ | # shell, so the command may include both file patterns |
||
+ | # and I/O redirection, they will be processed. |
||
+ | # |
||
+ | # NOTE: the program is executed once at startup, and |
||
+ | # kept running until SIGHUP or exit. The reason is to |
||
+ | # prevent starting up a large number of programs for |
||
+ | # messages, which would imply an easy DoS. |
||
+ | # tcp/udp - This driver sends messages to another host on the |
||
+ | # local intranet or internet using either UDP or TCP |
||
+ | # protocol. |
||
+ | # |
||
+ | # Both drivers have a single required argument |
||
+ | # specifying the destination host address, where |
||
+ | # messages should be sent, and several optional |
||
+ | # parameters. Note that this differs from source |
||
+ | # drivers, where local bind address is implied, and |
||
+ | # none of the parameters are required. |
||
+ | # |
||
+ | # Options: |
||
+ | # |
||
+ | # Name Type Description Default |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # localip string The IP address to bind to before 0.0.0.0 |
||
+ | # connecting to target. |
||
+ | # localport num The port number to bind to. 0 |
||
+ | # port/destport num The port number to connect to. 514 |
||
+ | # -------------- ------ -------------------------------- -------- |
||
+ | # usertty - This driver writes messages to the terminal of a |
||
+ | # logged-in user. |
||
+ | # |
||
+ | # The usertty driver has a single required argument, |
||
+ | # specifying a username who should receive a copy of |
||
+ | # matching messages, and no optional arguments. |
||
+ | # unix-dgram - unix-stream - This driver sends messages to a unix |
||
+ | # socket in either SOCK_STREAM or SOCK_DGRAM mode. |
||
+ | # |
||
+ | # Both drivers have a single required argument |
||
+ | # specifying the name of the socket to connect to, and |
||
+ | # no optional arguments. |
||
+ | #---------------------------------------------------------------------- |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Standard Log file locations |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination authlog { file("/var/log/auth.log"); }; |
||
+ | destination bootlog { file("/var/log/boot.log"); }; |
||
+ | destination debug { file("/var/log/debug"); }; |
||
+ | destination explan { file("/var/log/explanations"); }; |
||
+ | destination messages { file("/var/log/messages"); }; |
||
+ | destination routers { file("/var/log/routers.log"); }; |
||
+ | destination secure { file("/var/log/secure"); }; |
||
+ | destination spooler { file("/var/log/spooler"); }; |
||
+ | destination syslog { file("/var/log/syslog"); }; |
||
+ | destination user { file("/var/log/user.log"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Special catch all destination sorting by host |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination hosts { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_$DAY" |
||
+ | owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Forward to a loghost server |
||
+ | #---------------------------------------------------------------------- |
||
+ | #destination loghost { udp("10.1.1.254" port(514)); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Mail subsystem logs |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination mail { file("/var/log/mail.log"); }; |
||
+ | destination mailerr { file("/var/log/mail/errors"); }; |
||
+ | destination mailinfo { file("/var/log/mail/info"); }; |
||
+ | destination mailwarn { file("/var/log/mail/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # INN news subsystem |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination newscrit { file("/var/log/news/critical"); }; |
||
+ | destination newserr { file("/var/log/news/errors"); }; |
||
+ | destination newsnotice { file("/var/log/news/notice"); }; |
||
+ | destination newswarn { file("/var/log/news/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Cron subsystem |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination cron { file("/var/log/cron.log"); }; |
||
+ | destination crondebug { file("/var/log/cron/debug"); }; |
||
+ | destination cronerr { file("/var/log/cron/errors"); }; |
||
+ | destination croninfo { file("/var/log/cron/info"); }; |
||
+ | destination cronwarn { file("/var/log/cron/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # LPR subsystem |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination lpr { file("/var/log/lpr.log"); }; |
||
+ | destination lprerr { file("/var/log/lpr/errors"); }; |
||
+ | destination lprinfo { file("/var/log/lpr/info"); }; |
||
+ | destination lprwarn { file("/var/log/lpr/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Kernel messages |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination kern { file("/var/log/kern.log"); }; |
||
+ | destination kernerr { file("/var/log/kernel/errors"); }; |
||
+ | destination kerninfo { file("/var/log/kernel/info"); }; |
||
+ | destination kernwarn { file("/var/log/kernel/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Daemon messages |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination daemon { file("/var/log/daemon.log"); }; |
||
+ | destination daemonerr { file("/var/log/daemons/errors"); }; |
||
+ | destination daemoninfo { file("/var/log/daemons/info"); }; |
||
+ | destination daemonwarn { file("/var/log/daemons/warnings"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Console warnings |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination console { file("/dev/tty12"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # All users |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination users { usertty("*"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Examples of programs that accept syslog messages and do something |
||
+ | # programatically with them. |
||
+ | #---------------------------------------------------------------------- |
||
+ | #destination mail-alert { program("/usr/local/bin/syslog-mail"); }; |
||
+ | #destination mail-perl { program("/usr/local/bin/syslog-mail-perl"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Piping to Swatch |
||
+ | #---------------------------------------------------------------------- |
||
+ | #destination swatch { program("/usr/bin/swatch --read-pipe=\"cat /dev/fd/0\""); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Database notes: |
||
+ | # |
||
+ | # Overall there seems to be three primary methods of putting data from |
||
+ | # syslog-ng into a database. Each of these has certain pros and cons. |
||
+ | # |
||
+ | # FIFO file: Simply piping the template data into a First In, First |
||
+ | # Out file. This will create a stream of data that will |
||
+ | # not require any sort of marker or identifier of how |
||
+ | # much data has been read. This is the most elegant of |
||
+ | # the solutions and probably the most unstable. |
||
+ | # |
||
+ | # Pros: Very fast data writes and reads. Data being |
||
+ | # inserted into a database will be near real |
||
+ | # time. |
||
+ | # |
||
+ | # Cons: Least stable of all the possible solutions, |
||
+ | # and could require a lot of custom work to |
||
+ | # make function on any particular Unix system. |
||
+ | # |
||
+ | # Loss of the pipe file will cause complete |
||
+ | # data loss, and all following data that would |
||
+ | # have been written to the FIFO file. |
||
+ | # |
||
+ | # Buffer file: While very similar to a FIFO file this is would be a |
||
+ | # text file which would buffer all the template |
||
+ | # output information. Another program from cron or |
||
+ | # similar service would then run and source the buffer |
||
+ | # files and process the data into the database. |
||
+ | # |
||
+ | # Pros: Little chance of losing data since everything |
||
+ | # will be written to a physical file much like |
||
+ | # the regular logging process. |
||
+ | # |
||
+ | # This method gives a tremendous amount of |
||
+ | # flexibility since there would be yet another |
||
+ | # opportunity to filter logs prior to inserting |
||
+ | # any data into the database. |
||
+ | # |
||
+ | # Cons: Because there must be some interval between |
||
+ | # the processing of the buffer files, there will |
||
+ | # be a lag before the data is inserted in to the |
||
+ | # database. |
||
+ | # |
||
+ | # There is also a slight chance of data corruption |
||
+ | # (ie bad insert command) if the system crashes |
||
+ | # during a write, although this scenero is very |
||
+ | # unlikely. |
||
+ | # |
||
+ | # Another possible issue is that because multiple |
||
+ | # buffer files be written, the previously run |
||
+ | # sourcing file could get behind the data |
||
+ | # insertion if there is a very large quantity of |
||
+ | # logs being written. This will totally depend |
||
+ | # on the system that this is running on. |
||
+ | # |
||
+ | # Program: The least elegant of the solutions. This method is to |
||
+ | # send the stream of data through some further interrupter |
||
+ | # program such as something in Perl or C. That program |
||
+ | # will then take some action based off the data which |
||
+ | # could include writing to a database similarly to the |
||
+ | # program "sqlsyslogd". |
||
+ | # |
||
+ | # Pros: Allows complete control of the data, and as much |
||
+ | # post processing as required. |
||
+ | # |
||
+ | # Cons: Slowest of all the forms. Since the data will |
||
+ | # have to go through some post processing it will |
||
+ | # cause data being written to the database to |
||
+ | # remain behind actual log records. This could |
||
+ | # cause a race condition in that logging is lost |
||
+ | # either due to system crash, or high load on |
||
+ | # the logging system. |
||
+ | # |
||
+ | #---------------------------------------------------------------------- |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Writing to a MySQL database: |
||
+ | # |
||
+ | # Assumes a table/database structure of: |
||
+ | # |
||
+ | # CREATE DATABASE syslog; |
||
+ | # USE syslog; |
||
+ | # |
||
+ | # CREATE TABLE logs ( host varchar(32) default NULL, |
||
+ | # facility varchar(10) default NULL, |
||
+ | # priority varchar(10) default NULL, |
||
+ | # level varchar(10) default NULL, |
||
+ | # tag varchar(10) default NULL, |
||
+ | # date date default NULL, |
||
+ | # time time default NULL, |
||
+ | # program varchar(15) default NULL, |
||
+ | # msg text, seq int(10) unsigned NOT NULL auto_increment, |
||
+ | # PRIMARY KEY (seq), |
||
+ | # KEY host (host), |
||
+ | # KEY seq (seq), |
||
+ | # KEY program (program), |
||
+ | # KEY time (time), |
||
+ | # KEY date (date), |
||
+ | # KEY priority (priority), |
||
+ | # KEY facility (facility)) |
||
+ | # TYPE=MyISAM; |
||
+ | # |
||
+ | #---------------------------------------------------------------------- |
||
+ | # Piping method |
||
+ | #---------------------------------------------------------------------- |
||
+ | #destination database { pipe("/tmp/mysql.pipe" |
||
+ | # template("INSERT INTO logs (host, facility, |
||
+ | # priority, level, tag, date, time, program, |
||
+ | # msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', |
||
+ | # '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', |
||
+ | # '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") |
||
+ | # template-escape(yes)); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Buffer file method |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination database { file("/var/log/dblog/fulllog.$YEAR.$MONTH.$DAY.$HOUR.$MIN.$SEC" |
||
+ | template("INSERT INTO logs (host, facility, |
||
+ | priority, level, tag, date, time, program, |
||
+ | msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', |
||
+ | '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', |
||
+ | '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") |
||
+ | owner(root) group(root) perm(0600) |
||
+ | dir_perm(0700) create_dirs(yes) |
||
+ | template-escape(yes)); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Program method (alternate using sqlsyslogd): |
||
+ | # |
||
+ | # Notes: This is not a bad process, but lacks very much flexibility |
||
+ | # unless more changes are made to the source of sqlsyslogd. |
||
+ | # This is because sqlsyslogd assumes the data in a larger |
||
+ | # object style instead of breaking it down into smaller |
||
+ | # columnar pieces. |
||
+ | #---------------------------------------------------------------------- |
||
+ | #destination database { program("/usr/local/sbin/sqlsyslogd -u |
||
+ | # sqlsyslogd -t logs sqlsyslogs2 -p"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Since we probably will not be putting ALL of our logs in the database |
||
+ | # we better plan on capturing that data that we will be discarding for |
||
+ | # later review to insure we did not throw anything away we really |
||
+ | # should have captured. |
||
+ | #---------------------------------------------------------------------- |
||
+ | destination db_discard { file("/var/log/discard.log"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Filters |
||
+ | #---------------------------------------------------------------------- |
||
+ | # |
||
+ | # Functions: |
||
+ | # |
||
+ | # Name Synopsis Description |
||
+ | # -------------- ------------------------------ -------------------- |
||
+ | # facility facility(facility[,facility]) Match messages |
||
+ | # having one of the |
||
+ | # listed facility code. |
||
+ | # filter Call another filter rule and |
||
+ | # evaluate its value |
||
+ | # host host(regexp) Match messages by |
||
+ | # using a regular |
||
+ | # expression against |
||
+ | # the hostname field |
||
+ | # of log messages. |
||
+ | # level/priority level(pri[,pri1..pri2[,pri3]]) Match messages based |
||
+ | # on priority. |
||
+ | # match Tries to match a regular |
||
+ | # expression to the message |
||
+ | # itself. |
||
+ | # program program(regexp) Match messages by |
||
+ | # using a regular |
||
+ | # expression against |
||
+ | # the program name |
||
+ | # field of log messages |
||
+ | #---------------------------------------------------------------------- |
||
+ | # NOTES: |
||
+ | # |
||
+ | # Getting filtering to work right can be difficult because while the |
||
+ | # syntax is fairly simple, it is not well documented. To illustrate |
||
+ | # a brief lesson on filtering and to explain the majority of the |
||
+ | # mechanics, we shall use the filter from the PostgreSQL database |
||
+ | # how-to page found at: http://www.umialumni.com/~ben/SYSLOG-DOC.html |
||
+ | # |
||
+ | # This is a perfect and somewhat complex example to use. In its |
||
+ | # original form it resembles: |
||
+ | # |
||
+ | # filter f_postgres { not( |
||
+ | # (host("syslogdb") and facility(cron) and level(info)) |
||
+ | # or (facility(user) and level(notice) |
||
+ | # and ( match(" gethostbyaddr: ") |
||
+ | # or match("last message repeated ") |
||
+ | # ) |
||
+ | # ) |
||
+ | # or ( facility(local3) and level(notice) |
||
+ | # and match(" SYSMON NORMAL ")) |
||
+ | # or ( facility(mail) and level(warning) |
||
+ | # and match(" writable directory") |
||
+ | # ) |
||
+ | # or ( ( host("dbserv1.somecompany.com") |
||
+ | # or host("dbserv2.somecompany.com") |
||
+ | # ) |
||
+ | # and facility(auth) and level(info) |
||
+ | # and match("su oracle") and match(" succeeded for root on /dev/") |
||
+ | # ) |
||
+ | # ); }; |
||
+ | # |
||
+ | # While in this form, it does not induce a tremendous amount of |
||
+ | # insight on what the specific filter is attempting to accomplish. In |
||
+ | # reformatting the filter to resemble something a bit more human |
||
+ | # readable, it would look like: |
||
+ | # |
||
+ | # filter f_postgres { not |
||
+ | # ( |
||
+ | # ( |
||
+ | # host("syslogdb") and |
||
+ | # facility(cron) and |
||
+ | # level(info) |
||
+ | # ) or |
||
+ | # ( |
||
+ | # facility(user) and |
||
+ | # level(notice) and |
||
+ | # ( |
||
+ | # match(" gethostbyaddr: ") or |
||
+ | # match("last message repeated ") |
||
+ | # ) |
||
+ | # ) or |
||
+ | # ( |
||
+ | # facility(local3) and |
||
+ | # level(notice) and |
||
+ | # match(" SYSMON NORMAL ") |
||
+ | # ) or |
||
+ | # ( |
||
+ | # facility(mail) and |
||
+ | # level(warning) and |
||
+ | # match(" writable directory") |
||
+ | # ) or |
||
+ | # ( |
||
+ | # ( |
||
+ | # host("dbserv1.somecompany.com") or |
||
+ | # host("dbserv2.somecompany.com") |
||
+ | # ) and |
||
+ | # facility(auth) and |
||
+ | # level(info) and |
||
+ | # match("su oracle") and |
||
+ | # match(" succeeded for root on /dev/") |
||
+ | # ) |
||
+ | # ); |
||
+ | # }; |
||
+ | # |
||
+ | # Now in this form we can now begin to see what this filter has been |
||
+ | # attempting to accomplish. We can now further breakdown each logical |
||
+ | # section and explain the different methods: |
||
+ | # |
||
+ | # [1] As in all statements in syslog-ng, each of the beginnings and |
||
+ | # endings must be with a curly bracket "{" "}" to clearly denote |
||
+ | # the start and finish. |
||
+ | # |
||
+ | # In this filter, the entire filter is preferred by a "not" to |
||
+ | # indicate that these are the messages that we are NOT interested |
||
+ | # in and should be the ones filtered out. All lines of logs that |
||
+ | # do not match these lines will be sent to the destination. |
||
+ | # |
||
+ | # { not |
||
+ | # |
||
+ | # [2] The first major part of the filter is actually a compound |
||
+ | # filter that has two parts. Because the two parts are separated |
||
+ | # by an "or", only one of the two parts must be matched for that |
||
+ | # line of log to be filtered. |
||
+ | # |
||
+ | # [2a] In the first part of this filter there are three requirements |
||
+ | # to be met for the filter to take affect. These are the host |
||
+ | # string "syslogdb". the facility "cron", and the syslog level |
||
+ | # of info. |
||
+ | # |
||
+ | # ( |
||
+ | # ( |
||
+ | # host("syslogdb") and |
||
+ | # facility(cron) and |
||
+ | # level(info) |
||
+ | # ) or |
||
+ | # |
||
+ | # [2b] In the second part of the filter, which in itself is a |
||
+ | # compound filter, there are three requirements as well. These |
||
+ | # are that the facility of "user", and the log level of "notice" |
||
+ | # are met in addition to one of the two string matches that are |
||
+ | # shown in the example. |
||
+ | # |
||
+ | # ( |
||
+ | # facility(user) and |
||
+ | # level(notice) and |
||
+ | # ( |
||
+ | # match(" gethostbyaddr: ") or |
||
+ | # match("last message repeated ") |
||
+ | # ) |
||
+ | # ) or |
||
+ | # |
||
+ | # [3] In the section of the filter there are once again three |
||
+ | # requirements to fire off a match which are a facility of "level3" |
||
+ | # a log level of "notice" and a sting match of " SYSMON NORMAL ". |
||
+ | # |
||
+ | # ( |
||
+ | # facility(local3) and |
||
+ | # level(notice) and |
||
+ | # match(" SYSMON NORMAL ") |
||
+ | # ) or |
||
+ | # |
||
+ | # [4] This part of the filter is very similar to the previous |
||
+ | # filter, but with different search patterns. |
||
+ | # |
||
+ | # ( |
||
+ | # facility(mail) and |
||
+ | # level(warning) and |
||
+ | # match(" writable directory") |
||
+ | # ) or |
||
+ | # |
||
+ | # [5] The last section of the filter is also a compound filter |
||
+ | # that to take affect will require that one of two hosts |
||
+ | # are matched, the facility of "auth", and log level of |
||
+ | # "info" occur in addition to the two string matches. |
||
+ | # |
||
+ | # ( |
||
+ | # ( |
||
+ | # host("dbserv1.somecompany.com") or |
||
+ | # host("dbserv2.somecompany.com") |
||
+ | # ) and |
||
+ | # facility(auth) and |
||
+ | # level(info) and |
||
+ | # match("su oracle") and |
||
+ | # match(" succeeded for root on /dev/") |
||
+ | # ) |
||
+ | # |
||
+ | # [6] As in all command sets in syslog-ng, each of the statements |
||
+ | # must be properly closed with the correct ending punctuation |
||
+ | # AND a semi-colon. Do not forget both, or you will be faced with |
||
+ | # an error. |
||
+ | # |
||
+ | # ); }; |
||
+ | # |
||
+ | # While this may not be the most complete example, it does cover the |
||
+ | # majority of the options and features that are available within the |
||
+ | # current version of syslog-ng. |
||
+ | #---------------------------------------------------------------------- |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Standard filters for the standard destinations. |
||
+ | #---------------------------------------------------------------------- |
||
+ | filter f_auth { facility(auth, authpriv); }; |
||
+ | filter f_authpriv { facility(authpriv); }; |
||
+ | filter f_cron { facility(cron); }; |
||
+ | filter f_daemon { facility(daemon); }; |
||
+ | filter f_kern { facility(kern); }; |
||
+ | filter f_local1 { facility(local1); }; |
||
+ | filter f_local2 { facility(local2); }; |
||
+ | filter f_local3 { facility(local3); }; |
||
+ | filter f_local4 { facility(local4); }; |
||
+ | filter f_local5 { facility(local5); }; |
||
+ | filter f_local6 { facility(local6); }; |
||
+ | filter f_local7 { facility(local7); }; |
||
+ | filter f_lpr { facility(lpr); }; |
||
+ | filter f_mail { facility(mail); }; |
||
+ | filter f_messages { facility(daemon, kern, user); }; |
||
+ | filter f_news { facility(news); }; |
||
+ | filter f_spooler { facility(uucp,news) and level(crit); }; |
||
+ | filter f_syslog { not facility(auth, authpriv) and not facility(mail); }; |
||
+ | filter f_user { facility(user); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Other catch-all filters |
||
+ | #---------------------------------------------------------------------- |
||
+ | filter f_crit { level(crit); }; |
||
+ | #filter f_debug { not facility(auth, authpriv, news, mail); }; |
||
+ | filter f_debug { level(debug); }; |
||
+ | filter f_emergency { level(emerg); }; |
||
+ | filter f_err { level(err); }; |
||
+ | filter f_info { level(info); }; |
||
+ | filter f_notice { level(notice); }; |
||
+ | filter f_warn { level(warn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Filer for the MySQL database pipe. These are things that we really |
||
+ | # do not care to see otherwise they may fill up our database with |
||
+ | # garbage. |
||
+ | #---------------------------------------------------------------------- |
||
+ | #filter f_db { not facility(kern) and level(info, warning) or |
||
+ | # not facility(user) and level(notice) or |
||
+ | # not facility(local2) and level(debug); }; |
||
+ | # |
||
+ | #filter f_db { not match("last message repeated ") or |
||
+ | # not match("emulate rawmode for keycode"); }; |
||
+ | # |
||
+ | #filter f_discard { facility(kern) and level(info, warning) or |
||
+ | # facility(user) and level(notice) or |
||
+ | # facility(local2) and level(debug); }; |
||
+ | # |
||
+ | #filter f_discard { match("last message repeated ") or |
||
+ | # match("emulate rawmode for keycode"); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | # |
||
+ | # Notes: When applying filters, remember that each subsequent filter |
||
+ | # acts as a filter on the previous data flow. This means that |
||
+ | # if the first filter limits the flow to only data from the |
||
+ | # auth system, a subsequent filter for authpriv will cause |
||
+ | # no data to be written. An example of this would be: |
||
+ | # |
||
+ | # log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_auth); |
||
+ | # filter(f_authpriv); destination(authlog); }; |
||
+ | # |
||
+ | # So, one can cancel out the other. |
||
+ | # |
||
+ | # There are also certain flags that can be attached to each of the log |
||
+ | # statements: |
||
+ | # |
||
+ | # Flag Description |
||
+ | # -------- ---------------------------------------------------------- |
||
+ | # catchall This flag means that the source of the message is ignored, |
||
+ | # only the filters are taken into account when matching |
||
+ | # messages. |
||
+ | # fallback This flag makes a log statement 'fallback'. Being a |
||
+ | # fallback statement means that only messages not matching |
||
+ | # any 'non-fallback' log statements will be dispatched. |
||
+ | # final This flag means that the processing of log statements ends |
||
+ | # here. Note that this doesn't necessarily mean that |
||
+ | # matching messages will be stored once, as they can be |
||
+ | # matching log statements processed prior the current one. |
||
+ | #---------------------------------------------------------------------- |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Standard logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_auth); destination(authlog); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_local7); destination(bootlog); }; |
||
+ | #log{ source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_debug); destination(debug); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_local1); destination(explan); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_local5); destination(routers); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_messages); destination(messages); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_authpriv); destination(secure); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_spooler); destination(spooler); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_kernel); |
||
+ | source(s_tcp); filter(f_syslog); destination(syslog); }; |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); destination(syslog); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_user); destination(user); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Special catch all destination sorting by host |
||
+ | #---------------------------------------------------------------------- |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_kernel); |
||
+ | source(s_tcp); destination(hosts); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Send to a loghost |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); destination(loghost); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Mail subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_mail); destination(mail); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_mail); |
||
+ | filter(f_err); destination(mailerr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_mail); |
||
+ | filter(f_info); destination(mailinfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_mail); |
||
+ | filter(f_notice); destination(mailinfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_mail); |
||
+ | filter(f_warn); destination(mailwarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # INN subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_news); |
||
+ | filter(f_crit); destination(newscrit); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_news); |
||
+ | filter(f_err); destination(newserr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_news); |
||
+ | filter(f_notice); destination(newsnotice); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_news); |
||
+ | filter(f_warn); destination(newswarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Cron subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_cron); destination(crondebug); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_cron); |
||
+ | filter(f_err); destination(cronerr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_cron); |
||
+ | filter(f_info); destination(croninfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_cron); |
||
+ | filter(f_warn); destination(cronwarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # LPR subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_lpr); destination(lpr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_lpr); |
||
+ | filter(f_err); destination(lprerr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_lpr); |
||
+ | filter(f_info); destination(lprinfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_lpr); |
||
+ | filter(f_warn); destination(lprwarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Kernel subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_kern); destination(kern); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_kernel); |
||
+ | source(s_tcp); filter(f_kern); |
||
+ | filter(f_err); destination(kernerr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_kernel); |
||
+ | source(s_tcp); filter(f_kern); |
||
+ | filter(f_info); destination(kerninfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_kernel); |
||
+ | source(s_tcp); filter(f_kern); |
||
+ | filter(f_warn); destination(kernwarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Daemon subsystem logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_tcp); |
||
+ | # source(s_udp); filter(f_daemon); destination(daemon); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_daemon); |
||
+ | filter(f_err); destination(daemonerr); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_daemon); |
||
+ | filter(f_info); destination(daemoninfo); }; |
||
+ | log { source(s_dgram); |
||
+ | source(s_internal); |
||
+ | source(s_tcp); filter(f_daemon); |
||
+ | filter(f_warn); destination(daemonwarn); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Console logging |
||
+ | #---------------------------------------------------------------------- |
||
+ | # 16-Mar-03 - REP - Removed logging to the console for performance |
||
+ | # reasons. Since we are not really going to be |
||
+ | # looking at the console all the time, why log there |
||
+ | # anyway. |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); filter(f_syslog); destination(console); }; |
||
+ | |||
+ | #---------------------------------------------------------------------- |
||
+ | # Logging to a database |
||
+ | #---------------------------------------------------------------------- |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); filter(f_db); destination(database); }; |
||
+ | #log { source(s_dgram); |
||
+ | # source(s_internal); |
||
+ | # source(s_kernel); |
||
+ | # source(s_tcp); filter(f_discard); destination(db_discard); }; |
||
+ | </pre> |
||
+ | |||
+ | * Fő bajom a szintaxissal: ha egy-egy objektumot (source, filter vagy destination) csak egyszer haszálunk, akkor is, mielőtt használhatnánk, explicite deklarálni kell valahol és nevet adni neki. Nem csinálhatunk pl. ilyet: |
||
+ | |||
+ | <pre> |
||
+ | # EZ SAJNOS ÉRVÉNYTELEN |
||
+ | log { |
||
+ | source(pipe("/proc/kmsg" log_prefix("kernel: "))); |
||
+ | filter(not level(debug)); |
||
+ | destination(file("/var/log/kern.log")); |
||
+ | }; |
||
+ | </pre> |
||
+ | |||
+ | === socklog === |
||
+ | |||
+ | * Egyszerű, mint a százas szög: olvas "valahonnan", és stdoutra ír. |
||
+ | * Kb. 260kB memóriát foglal (a hozzá tartozó svlogd-vel együtt kb. 300kB-ot). |
||
+ | ** Ehhez képest egy nem túl összetett konfigurációjú syslog-ng 5+ MB-ot, és 80MB fölötti virtuális tárterületet. |
||
+ | * A "valahonnan" lehet: |
||
+ | ** unix domain socket |
||
+ | ** udp socket |
||
+ | ** [[UCSPI]] (tcp socket, unix domain socket, ssl socket) (később lesz róla szó) |
||
+ | * Ha több helyről is szeretnénk olvasni, egyszerűen több példányt futtatunk. |
||
+ | * Az üzenetek szétválogatása az [[A_runit_működése#svlogd|svlogd]] feladata. |
||
+ | |||
+ | Előnyök: |
||
+ | |||
+ | * Kicsi, aranyos, egyszerű |
||
+ | * Majdnem olyan rugalmas, mint a syslog-ng |
||
+ | |||
+ | A socklog+svlogd kombináció hátrányai: |
||
+ | |||
+ | * A konfiguráció elaprózódik (sok kis file, amit ugyan generálhatunk egy nagyból, de ahhoz dolgozni kell) |
||
+ | * Egy konfigurációs változtatás adott esetben több file módosítását jelenti (pl. az egyikben kizárunk egy üzenettípust, a másikban engedélyezzük) |
||
+ | * Sok hoszttól üzeneteket fogadni és azokat szétválogatni bonyolult, rosszul skálázódik (bár scriptelhető) |
||
+ | |||
+ | == Ajánlott irodalom == |
||
+ | |||
+ | * http://www.campin.net/syslog-ng/faq.html - syslog-ng faq |
||
+ | * http://sial.org/howto/logging/syslog-ng/ - egy esettanulmány syslog-ng-vel |
||
+ | |||
+ | == Potenciális zh-kérdések == |
||
+ | |||
+ | * Mik a főbb problémák a hagyományos unixos sysloggal? |
||
+ | * Mik a syslog-ng fő jellemzői, előnyei, hátrányai? |
||
+ | * Hogyan működik a socklog? Hogyan építhetünk vele teljes rendszernapló-megoldást? Mik az előnyei/hátrányai egy syslog-ng-alapú megoldáshoz képest? |
||
+ | * Mik az előnyei/hátrányai az udp ill. a tcp fölötti hálózati naplózásnak? |
||
+ | * Mi a különbség a logrotate és az svlogd naplórotációs mechanizmusa között? Melyik miért jobb? |
||
+ | * Mi a különbség az alábbi két syslog-ng config-részlet között? |
||
+ | : a) |
||
+ | <pre> |
||
+ | source local { unix-stream("/dev/log"); internal(); }; |
||
+ | source jail1 { unix-stream("/jail/dns/dev/log"); }; |
||
+ | source jail2 { unix-stream("/jail/www/dev/log"); }; |
||
+ | </pre> |
||
+ | : b) |
||
+ | <pre> |
||
+ | source local { |
||
+ | unix-stream("/dev/log"); |
||
+ | internal(); |
||
+ | unix-stream("/jail/dns/dev/log"); |
||
+ | unix-stream("/jail/www/dev/log"); |
||
+ | }; |
||
+ | </pre> |
||
+ | * Mik a bajok a logcheck-kel? (Az előadáson elhangzott, valaki könyörüljön meg a többieken és írja fel a wikire...) |
||
+ | * Mi a TAI64N? Milyen előnyei vannak más, ugyanezt a célt szolgáló megoldásokkal szemben? Van-e hátránya? |
||
+ | * Mitől jó vagy rossz egy naplófájl-formátum? |
A lap jelenlegi, 2012. október 31., 11:32-kori változata
Általában naplózni akarunk minden, a rendszer szempontjából fontos eseményt. A naplózással kapcsolatban a következő - részben filozófiai - kérdéseket kell feltennünk magunknak:
- Mit akarunk naplózni?
- Az "esemény" tetszőlegesen szűken vagy tágan értelmezhető; lehet "kicsi" vagy "nagy"; tartozhat hozzá sok vagy kevés információ:
- "Elmozdult az egér"
- "A 42-es PID-jú folyamat TERM signal hatására kilépett"
- "Új ARP-bejegyzést helyeztünk el az ARP-táblában"
- "A gonosz crackerek betörtek és az összes adatot ellopták"
- "Minden rendben van"
- "Megint eltelt 10 perc"
- "Belépett valaki ssh-val"
- "Józsi belépett ssh-val"
- "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről"
- "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában a 2.3.4.5-ről szokott belépni"
- "Józsi ma este 8-kor belépett ssh-val az 1.2.3.4-ről, de általában csak munkaidőben lép be (99%), a 2.3.4.5-ről (95%)"
- A két utóbbi példa csak nagy jóindulattal nevezhető "eseménynek"; a naplóüzenetekben az "elemi eseményhez" egy összetett utófeldolgozás eredményét is csatoltuk.
- stb.
- Az "esemény" tetszőlegesen szűken vagy tágan értelmezhető; lehet "kicsi" vagy "nagy"; tartozhat hozzá sok vagy kevés információ:
- Mi a célunk a naplózással? (Ez befolyásolja, mi lesz az "esemény".)
- Biztonsági audit
- Pl.: ssh log, filesystem audit trail
- Teljesítmény-finomhangolás előkészítése
- Pl.: vmstat
- A rendszer állapotváltozásainak nyomon követése későbbi reprodukálhatóság vagy visszacsinálhatóság érdekében
- Pl.: dpkg.log
- Értesülés a rendszer működését veszélyeztető eseményekről, folyamatokról
- Pl.: kernellog, smartmontools logjai
- Üzleti döntések támogatása
- Pl.: apache accesslog
- Utólagos diagnosztika megkönnyítése
- Pl.: ppp log, cron log, maillog, udev log, általában a "debug" típusú logok, ...
- Szolgáltatások használati profiljának feltérképezése
- Pl.: dnscache query log, webszerver access-logja
- Accounting
- Pl.: CUPS page log
- Személyes
- irclog, IM log
- Csak úgy, majd csak jó lesz valamire
- Pl.: mp3-lejátszó által lejátszott számok naplója
- Biztonsági audit
- Mihez fogunk kezdeni a naplóval?
- Csak legyen, egyszer majd talán belenézünk
- Adatokat bányászunk belőle (pl. accesslogból)
- "Mely országokból jöttek azok a felhasználók, akik az X és az Y terméket is megnézték, de utána a Z-t vették meg?"
- "Azok közül, akik a promócióban terjesztett URL-re ellátogatnak, hányan vásárolnak később a webshopban?"
- "Vásárlóink hány százalékát teszik ki a Linux-felhasználók?"
- "A múlt havi promóció melyik országban volt a legsikeresebb?"
- "Hányan jöttek vissza az oldalunkra a promóciót követő három hónapon belül kettőnél többször?"
- "Melyik user melyik napszakban honnan szokott be-ssh-zni? Volt-e bárkinél számottevő eltérés ettől a profiltól? (Mi a számottevő?)"
- "Milyen folyamatok futottak akkor, amikor a 300-as load-tüskét mértük?"
- Stb.
- Policy-konformanciát ellenőrzünk (ha nem tudjuk vagy akarjuk eleve kikényszeríteni)
- "Tényleg nem játszott éjfél után a gyerek?"
- "Igaz-e, hogy főleg/csak a munkával összefüggő weboldalakat nézegetnek az alkalmazottak?"
- "Igaz-e, hogy Gipsz Jakab csak az ebédszünetben foglalkozott magánlevelezéssel?"
- "Igaz-e, hogy Gáz Géza autóbuszvezető csak a kijelölt útvonalon közlekedik a gondjaira bízott Ikarus-szal, vagy taxizgat is vele?"
- "Tényleg nem futtatott senki p2p filecserélőt a Schönherzben?"
- Gyanús mintákat keresünk benne
- Tároljuk, mert muszáj (törvényi előírás)
- Átadjuk az ügyfélnek
- Aki majd mit csinál vele?
- Erős lehet a kísértés a kozmetikázásra
- Sose fizessünk marketingcéget a visszatérő userek száma alapján...
- Időnként szabad szemmel beleolvasunk
- Célzottan keresünk benne
- "Kik voltak belépve múlt kedden este 6 és 8 között?"
- Stb.
Tartalomjegyzék |
[szerkesztés] 1 Naplózóprogramok
Sok van.
Rendszernaplóra legelterjedtebb:
- syslog
- syslog-ng
- socklog+svlogd
- (klogd)
- új default a Debianban: rsyslog (még nem volt időm megnézni; potenciális házi feladat)
De persze minden második alkalmazás saját naplófájlt ír, és persze mindegyik más és más formátumban. Még a dátumformátum sem egyezik.
Néhány példa:
- samba:
[2007/10/20 22:56:13, 0] nmbd/nmbd_become_lmb.c:become_local_master_stage2(396) ***** Samba name server FOOBAR is now a local master browser for workgroup WORKGROUP on subnet 1.2.3.4 ***** [2007/10/20 22:58:26, 1] lib/util_sock.c:open_socket_out(896) timeout connecting to 2.3.4.5:139
- Egy esemény több sor, még csak nem is mindig ugyanannyi.
- Hogyan fésülünk össze két ilyen logot?
- A dátumnál a lexikografikus sorrend megegyezik az időbelivel, ez jó.
- Fejlesztőknek szóló napló (megmondja, a forráskód melyik fájljának melyik sorában generálják az adott üzenetet).
- Ez az üzemeltetőnek nem feltétlenül hasznos.
Hogyan lehetett volna jobban?
2007-10-20 22:56:13 foobar nmbd: info: becomemaster: stage2: FOOBAR now local master for WORKGROUP on 1.2.3.4 2007-10-20 22:58:26 foobar nmbd: warn: remoteannounce: timeout connecting to 2.3.4.5:139
- squid access.log:
1193373264.480 43 192.168.0.4 TCP_MISS/200 1334 GET cache_object://192.168.0.4/server_list - NONE/- text/plain 1193373264.536 1 192.168.0.4 TCP_MISS/200 1334 GET cache_object://192.168.0.4/server_list - NONE/- text/plain
- Dátum helyett UNIX epoch seconds, pont, ms.
- A milliszekundum-mezőben szerencsére megvannak a kezdő nullák a 100-nál kisebb értékek előtt (ez sajnos nem magától értetődő).
- Rendezhető és összefésülhető, de nem túl olvasható.
- Mezőkre osztott sor, de a mezők száma nem állandó (vagy nem fix számú szóköz választja el őket egymástól).
- Az awk-nál egyszerűbb eszközökkel (pl. cut) való feldolgozás nehézkes.
- squid cache.log:
2007/10/26 06:26:03| helperOpenServers: Starting 10 'adzapper.wrapper' processes 2007/10/26 06:27:49| CACHEMGR: <unknown>@192.168.0.4 requesting 'counters'
- Még egy alkalmazáson belül sem egységes a dátumformátum...
- És minek oda az a pipe karakter?
- De legalább nagyjából használható tag ("címke") van a sor elején.
- rinetd.log:
21/Sep/2007:19:54:43 127.0.0.1 127.0.0.1 111 192.168.0.4 111 44 72 done-remote-closed
- Újabb kiválóan rendezhető dátumformátum.
- Viszont TAB van a mezők között, úgyhogy a cut is boldogul vele.
- apache "combined" accesslog:
lj511257.crawl.yahoo.net - - [02/Oct/2007:10:53:05 +0200] "GET /~korn/nws2002/djbdns-DNS-BIND-nelkul.png/Slide05.html HTTP/1.0" 200 1215 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
- Újabb dátumszörny, bár ebben legalább benne van, melyik időzónában kell értelmezni.
- Rendezni szeretnénk? Mi sem egyszerűbb! sort -t ' ' -k 4.9,4.12n -k 4.5,4.7M -k 4.2,4.3n -k 4.14,4.15n -k 4.17,4.18n -k 4.20,4.21n (bár többféle időzónából származó logokra ez sem elég).
- Némelyik mező belsejében lehet szóköz, bár szerencsére főleg az utolsóban (User-Agent) jellemző.
- apache errorlog:
[Tue Oct 2 12:53:54 2007] [error] [client 65.92.126.129] File does not exist: /var/www/links
- Miért is lenne ugyanolyan formátumú a dátum?
- syslogd:
Oct 25 23:04:33 kernel: [810505.320000] tap2: received packet with own address as source address
- Időzóna?
- Sőt: évszám?!
Na de nézzük meg részletesebben a syslogd-t:
[szerkesztés] 1.1 syslogd
- Történelmi okokból ez a legelterjedtebb.
- bár Debianon mára kiszorította az rsyslog, ami viszont 100%-ig kompatibilis akar lenni vele.
- "facility" és "severity" alapján tudja célobjektumokba (l. alant) válogatni az üzeneteket
- tud figyelni:
- unix domain socketen (/dev/log)
- egyszerre csak egyen, chroot esetén gond lehet (a linuxos és az openbsd-s syslogd tud több socketet is kezelni, a régebbiek általában nem)
- mount --bind a socketet tartalmazó könyvtárra jó workaround lehet; a /dev/log ebben az esetben symlink kell, hogy legyen.
- Hátrány: a naplóban nem fog látszani, melyik chrootból jött egy-egy üzenet.
- egyszerre csak egyen, chroot esetén gond lehet (a linuxos és az openbsd-s syslogd tud több socketet is kezelni, a régebbiek általában nem)
- udp socketen (514-es port)
- unix domain socketen (/dev/log)
- célobjektumok:
- file (választhatóan pufferelt vagy sync)
- nem feltétlenül hozza létre!
- udp
- named pipe (pl. /dev/xconsole
- bejelentkezett felhasználó terminálja
- file (választhatóan pufferelt vagy sync)
Példa /etc/syslog.conf-ra:
# /etc/syslog.conf Configuration file for syslogd. # # For more information see syslog.conf(5) # manpage. # # First some standard logfiles. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log uucp.* /var/log/uucp.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # Logging for INN news system # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice # # Some `catch-all' logfiles. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages kern.warn @loghost # # Emergencies are sent to everybody logged in. # *.emerg * # Messages of the priority alert will be directed # to the operator # *.alert root,joey # This will store all messages with the priority crit in the file /var/adm/critical, except for any kernel message. *.=crit;kern.none /var/log/critical # # I like to have messages displayed on the console, but only on a virtual # console I usually leave idle. # #daemon,mail.*;\ # news.=crit;news.=err;news.=notice;\ # *.=debug;*.=info;\ # *.=notice;*.=warn /dev/tty8 # The named pipe /dev/xconsole is for the `xconsole' utility. To use it, # you must invoke `xconsole' with the `-file' option: # # $ xconsole -file /dev/xconsole [...] # # NOTE: adjust the list below, or you'll go crazy if you have a reasonably # busy site.. # daemon.*;mail.*;\ news.crit;news.err;news.notice;\ *.=debug;*.=info;\ *.=notice;*.=warn |/dev/xconsole
Problémák:
- Több implementáció, közös gyökerekkel, de eltérő fejlődéssel => a felszín alatt megbúvó inkompatibilitások.
- Pl. van olyan verzió, amelyiknek nem mindegy, hogy a configfile-ban szóközök vagy TABok vannak.
- Nem minden implementáció hozza létre a logfile-okat, ha nem léteznek - némelyik egyszerűen nem logolja az adott fájlba írandó üzeneteket, ha nem létezik a fájl.
- Szűkös konfigurációs lehetőségek:
- Csak facility és severity/priority alapján lehet szétválogatni a logokat.
- Pedig rengeteg daemon.* üzenet van, jó lenne ezeket külön logolni.
- A hagyományos dátumformátum rossz:
- "Oct 23 12:03:42"
- Nem rendezhető lexikografikusan => logok összefésülése körülményes
- Nincs benne évszám
- Nincs benne az időzóna
- Még a dátumban sem állandó a mezők száma ("Jan 1" vs. "Jan 10": "Jan__1" - három mező, "Jan_10" - két mező)
- Az udp-s logolás megbízhatatlan, üzenetek veszhetnek el.
- Nincs hitelesítés, hálózati naplózás esetén könnyű hamis naplóüzeneteket beszúrni.
- A sorok struktúrája változó, nehézkes az egységes kezelésük.
- Csak egy IP-nek van hely a logolt sor formátumában, így proxy esetén ott mindig a proxy IP-je lesz,
- hacsak a proxy bele nem írja az üzenet szövegébe a kliens IP-jét;
- így viszont változó számú IP-prefix kerülhet a sorok elejére.
- Rotáció problémás (nem beépített)
- PID-lottó...
- Egy esemény egy sor, ráadásul általában max. 1024 karakter (ennél egy java stack trace kapásból több).
- "Tömörítés": ha ugyanaz az üzenet sokszor ismétlődik, a syslogd nem logolja, csak egy idő után azt, hogy hány ismétlődés volt.
- Így pl. nem nagyon tudjuk, az egyes ismétlődések mikor voltak.
- Általában rootként futtatják.
- pl. mert egyébként HUP signal (logok újramegnyitása) után esetleg már nem tudna írni a logokba, mert csak a root számára írhatók.
- Implementációfüggő, hogy melyik mezők szerepelnek az udp-n átküldött üzenetekben:
- pl. Solaris: <13>Oct 23 12:34:56 last message repeated 15 times.
- Linux: <13>last message repeated 4 times
- Cisco: 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16
- Ez nem is hasonlít egy syslogd-üzenetre, így a syslogd elécsap egy saját fejlécet, és így logolja: Aug 23 03:04:05 switch.company.com 2005 Aug 23 03:04:05 UTC +00:00 %PAGP-5-PORTFROMSTP:Port 4/16 left bridge port 4/16
[szerkesztés] 1.2 syslog-ng
- Hasonlít a syslogra, annak a hibáit próbálja javítani.
- Több generációváltáson van túl; az egyes generációk részben inkompatibilisek egymással.
- Tud SOCK_STREAM és SOCK_DGRAM típusú Unix domain socketet is (legalábbis Linuxon)
- Tud TCP-n is figyelni
- Nemcsak megbízható átvitel, hanem
- SSL-esíthető is
- Testreszabható a naplózott sorok formátuma (elsősorban a mezők sorrendje: dátum, hoszt, üzenet)
- Viszonylag egyszerűen tudjuk sok különböző gép logjait fogadni (nem kell mindegyikhez külön konfiguráció: a naplófájl neve függhet a kliens IP-jétől vagy hosztnevétől)
- Tud több Unix domain socketen figyelni, így:
source local { unix-stream("/dev/log"); internal(); }; source jail1 { unix-stream("/jail/dns/dev/log"); }; source jail2 { unix-stream("/jail/www/dev/log"); };
- és így:
source local { unix-stream("/dev/log"); internal(); unix-stream("/jail/dns/dev/log"); unix-stream("/jail/www/dev/log"); };
Hátrányok:
- Bonyolult.
- Statikusan linkelik a GLibhez, ami egy viszonylag nagy library - dinamikusan viszont nem lehet, mert a /usr alá szokás telepíteni a GLibet, így a syslog-ng indulásakor esetleg nem áll rendelkezésre.
- Aránylag összetett konfigurációs szintaxis, ráadásul szószátyár is és nem is túl átlátható.
- DNS-feloldáskor blokkolódik (ez 2007. körül volt így; lehet, hogy már javították).
- A reguláris kifejezések illesztése lassú, az alternatíva viszont bonyolult:
Reguláris kifejezéssel:
filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error .* is way too large .* set clock manually"); };
Ehelyett így kéne:
filter f_xntp_filter_no_regexp { # original line: "xntpd[1567]: time error -1159.777379 is way too large (set clock manually); program("xntpd") and match("time error") and match("is way too large") and match("set clock manually"); };
Ez egyrészt nem ugyanazt jelenti (illeszkedik arra is, hogy "'set clock manually' is way too large for a time error", bár ez itt nem túl nagy gond), másrészt nem kifejezetten karbantartásbarát.
Példakonfiguráció (ha valakinek van kedve magyarul felkommentezni, ne habozzon):
#---------------------------------------------------------------------- # Program: syslog-ng.conf # Notes: Embedded most of the manual notes within the configuration # file. The original manual can be found at: # # http://www.balabit.com/products/syslog_ng/reference/book1.html # http://www.campin.net/syslog-ng/faq.html # # Many people may find placing all of this information in a # configuration file a bit redundant, but I have found that # with a little bit of extra comments and reference, # maintaining these beasties is much easier. # # This particular log file was taken from the examples that # are given at the different web sites, and made to emulate # the logs of a Mandrake Linux system as much as possible. # Of course, Unix is Unix, is Linux. It should be generic # enough for any Unix system. #---------------------------------------------------------------------- # 16-Mar-03 - REP - Added some extra definitions to the file. # 15-Mar-03 - REP - Added back the comments on filtering. # 27-Feb-03 - REP - Further modified for local environment. # 27-Feb-03 - REP - Updated for new configuration and version 1.6.0 # 12-Dec-02 - REP - Continued updates for writing to databases. # 30-Nov-02 - REP - Initial creation for testing. #---------------------------------------------------------------------- # Options #---------------------------------------------------------------------- # # Name Values Description # ------------------------- ------- ------------------------------------ # bad_hostname reg exp A regexp which matches hostnames # which should not be taken as such. # chain_hostnames y/n Enable or disable the chained # hostname format. # create_dirs y/n Enable or disable directory creation # for destination files. # dir_group groupid # dir_owner userid # dir_perm perm # dns_cache y/n Enable or disable DNS cache usage. # dns_cache_expire num Number of seconds while a successful # lookup is cached. # dns_cache_expire_failed num Number of seconds while a failed # lookup is cached. # dns_cache_size num Number of hostnames in the DNS cache. # gc_busy_threshold num Sets the threshold value for the # garbage collector, when syslog-ng is # busy. GC phase starts when the number # of allocated objects reach this # number. Default: 3000. # gc_idle_threshold num Sets the threshold value for the # garbage collector, when syslog-ng is # idle. GC phase starts when the number # of allocated objects reach this # number. Default: 100. # group groupid # keep_hostname y/n Enable or disable hostname rewriting. # This means that if the log entry had # been passed through at least one other # logging system, the ORIGINAL hostname # will be kept attached to the log. # Otherwise the last logger will be # considered the log entry owner and # the log entry will appear to have # come from that host. # log_fifo_size num The number of lines fitting to the # output queue # log_msg_size num Maximum length of message in bytes. # long_hostnames on/off This options appears to only really # have an affect on the local system. # which removes the source of the log. # As an example, normally the local # logs will state src@hostname, but # with this feature off, the source # is not reported. # mark num The number of seconds between two # MARK lines. NOTE: not implemented # yet. # owner userid # perm perm # stats num The number of seconds between two # STATS. # sync num The number of lines buffered before # written to file # time_reap num The time to wait before an idle # destination file is closed. # time_reopen num The time to wait before a died # connection is reestablished # use_dns y/n Enable or disable DNS usage. # syslog-ng blocks on DNS queries, # so enabling DNS may lead to a # Denial of Service attack. To # prevent DoS, protect your # syslog-ng network endpoint with # firewall rules, and make sure that # all hosts, which may get to # syslog-ng is resolvable. # use_fqdn y/n Add Fully Qualified Domain Name # instead of short hostname. # use_time_recvd y/n Use the time a message is # received instead of the one # specified in the message. #---------------------------------------------------------------------- # 15-Mar-03 - REP - Since some of the clocks are not quite right, we # are going to go ahead and just use the local time # as the master time. # 12-Mar-03 - REP - We have taken a few configuration options from the # newer Solaris configuration because some of the # reasons are valid for us as well. We have increased # the log_msg_size and log_fifo_size to increase the # amount of buffering that we do. While for most # systems this may not have a noticeable affect, it # will for systems that are at the end of a lot of # logging systems. # 20-Dec-02 - REP - Changed the stat() time from the default of 10 # minutes to once an hour. #---------------------------------------------------------------------- options { chain_hostnames(no); create_dirs (no); dir_perm(0755); dns_cache(yes); keep_hostname(yes); log_fifo_size(2048); log_msg_size(8192); long_hostnames(on); perm(0644); stats(3600); sync(0); time_reopen (10); use_dns(yes); use_fqdn(yes); }; #---------------------------------------------------------------------- # Sources #---------------------------------------------------------------------- # # fifo/pipe - The pipe driver opens a named pipe with the # specified name, and listens for messages. It's # used as the native message getting protocol on # HP-UX. # file - Usually the kernel presents its messages in a # special file (/dev/kmsg on BSDs, /proc/kmsg on # Linux), so to read such special files, you'll need # the file() driver. Please note that you can't use # this driver to follow a file like tail -f does. # internal - All internally generated messages "come" from this # special source. If you want warnings, errors and # notices from syslog-ng itself, you have to include # this source in one of your source statements. # sun-streams - Solaris uses its STREAMS API to send messages to # the syslogd process. You'll have to compile # syslog-ng with this driver compiled in (see # ./configure --help). # # Newer versions of Solaris (2.5.1 and above), uses a # new IPC in addition to STREAMS, called door to # confirm delivery of a message. Syslog-ng supports # this new IPC mechanism with the door() option. # # The sun-streams() driver has a single required # argument, specifying the STREAMS device to open and # a single option. # tcp/udp - These drivers let you receive messages from the # network, and as the name of the drivers show, you # can use both UDP and TCP as transport. # # UDP is a simple datagram oriented protocol, which # provides "best effort service" to transfer # messages between hosts. It may lose messages, and # no attempt is made to retransmit such lost # messages at the protocol level. # # TCP provides connection-oriented service, which # basically means a flow-controlled message pipeline. # In this pipeline, each message is acknowledged, and # retransmission is done for lost packets. Generally # it's safer to use TCP, because lost connections can # be detected, and no messages get lost, but # traditionally the syslog protocol uses UDP. # # None of tcp() and udp() drivers require positional # parameters. By default they bind to 0.0.0.0:514, # which means that syslog-ng will listen on all # available interfaces, port 514. To limit accepted # connections to one interface only, use the # localip() parameter as described below. # # Options: # # Name Type Description Default # -------------- ------ -------------------------------- -------- # ip or local ip string The IP address to bind to. Note 0.0.0.0 # that this is not the address # where messages are accepted # from. # keep-alive y/n Available for tcp() only, and yes # specifies whether to close # connections upon the receival # of a SIGHUP signal. # max-connections number Specifies the maximum number of 10 # simultaneous connections. # port or local port number The port number to bind 514 # to. # -------------- ------ -------------------------------- -------- # # unix-stream - unix-dgram - These two drivers behave similarly: # they open the given AF_UNIX socket, and start # listening on them for messages. unix-stream() is # primarily used on Linux, and uses SOCK_STREAM # semantics (connection oriented, no messages are # lost), unix-dgram() is used on BSDs, and uses # SOCK_DGRAM semantics, this may result in lost # local messages, if the system is overloaded. # # To avoid denial of service attacks when using # connection-oriented protocols, the number of # simultaneously accepted connections should be # limited. This can be achieved using the # max-connections() parameter. The default value of # this parameter is quite strict, you might have to # increase it on a busy system. # # Both unix-stream and unix-dgram has a single # required positional argument, specifying the # filename of the socket to create, and several # optional parameters. # # Options: # # Name Type Description Default # -------------- ------ -------------------------------- -------- # group string Set the gid of the socket. root # keep-alive y/n Selects whether to keep yes # connections opened when # syslog-ng is restarted, can be # used only with unix-stream(). # max-connections numb Limits the number of 10 # simultaneously opened # connections. Can be used only # with unix-stream(). # owner string Set the uid of the socket. root # perm num Set the permission mask. For 0666 # octal numbers prefix the number # with '0', e.g. use 0755 for # rwxr-xr-x. #---------------------------------------------------------------------- # Notes: For Linux systems (and especially RedHat derivatives), # they have a second logging process for kernel messages. # This source is /proc/kmsg. If you are running this on a # system that is not Linux, then the source entry for this # should be removed. # # It seems that there is some performance questions related # to what type of source stream should be used for Linux # boxes. The documentation states the /dev/log should use # unix-stream, but from the mailing list it has been # strongly suggested that unix-dgram be used. # # WARNING: TCP wrappers has been enabled for this system, and unless # you also place entries in /etc/hosts.allow for each of the # devices that will be delivering logs via TCP, you will # NOT receive the logs. # # Also note that if there is any form of a local firewall, # this will also need to be altered such that the incoming # and possibly outgoing packets are allowed by the firewall # rules. #---------------------------------------------------------------------- # There has been a lot of debate on whether everything should be put # to a single source, or breakdown all the sources into individual # streams. The greatest flexibility would be in many, but the most # simple is the single. Since we wrote this file, we have chosen the # route of maximum flexibility. # # For those of you that like simplicity, this could have also been # done as the follows: # # source src # { # internal(); # pipe("/proc/kmsg" log_prefix("kernel: ")); # tcp(ip(127.0.0.1) port(4800) keep-alive(yes)); # udp(); # unix-stream("/dev/log"); # }; # # You would also have to change all the log statements to only # reference the now single source stream. #---------------------------------------------------------------------- # 16-Mar-03 - REP - The default number of allowed TCP connects is set # very low for a logserver. This value should only # be set greater than the default for servers that # will actually be serving that many systems. #---------------------------------------------------------------------- source s_dgram { unix-dgram("/dev/log"); }; source s_internal { internal(); }; source s_kernel { pipe("/proc/kmsg" log_prefix("kernel: ")); }; source s_tcp { tcp(port(4800) keep-alive(yes) max_connections(100)); }; #---------------------------------------------------------------------- # Destinations #---------------------------------------------------------------------- # # fifo/pipe - This driver sends messages to a named pipe like # /dev/xconsole # # The pipe driver has a single required parameter, # specifying the filename of the pipe to open, and # no options. # file - The file driver is one of the most important # destination drivers in syslog-ng. It allows you to # output messages to the named file, or as you'll see # to a set of files. # # The destination filename may include macros which # gets expanded when the message is written, thus a # simple file() driver may result in several files # to be created. Macros can be included by prefixing # the macro name with a '$' sign (without the quotes), # just like in Perl/PHP. # # If the expanded filename refers to a directory # which doesn't exist, it will be created depending # on the create_dirs() setting (both global and a per # destination option) # # WARNING: since the state of each created file must # be tracked by syslog-ng, it consumes some memory # for each file. If no new messages are written to a # file within 60 seconds (controlled by the time_reap # global option), it's closed, and its state is freed. # # Exploiting this, a DoS attack can be mounted against # your system. If the number of possible destination # files and its needed memory is more than the amount # your logserver has. # # The most suspicious macro is $PROGRAM, where the # possible variations is quite high, so in untrusted # environments $PROGRAM usage should be avoided. # # Macros: # # Name Description # ----------------- ----------------------------------------------- # DATE Date of the transaction. # DAY The day of month the message was sent. # FACILITY The name of the facility, the message is tagged # as coming from. # FULLDATE Long form of the date of the transaction. # FULLHOST Full hostname of the system that sent the log. # HOST The name of the source host where the message # is originated from. If the message traverses # several hosts, and chain_hostnames() is on, # the first one is used. # HOUR The hour of day the message was sent. # ISODATE Date in ISO format. # MIN The minute the message was sent. # MONTH The month the message was sent. # MSG or MESSAGE Message contents. # PRIORITY or LEVEL The priority of the message. # PROGRAM The name of the program the message was sent by. # SEC The second the message was sent. # TAG The priority and facility encoded as a 2 digit # hexadecimal number. # TZ The time zone or name or abbreviation. e.g. 'PDT' # TZOFFSET The time-zone as hour offset from GMT. e.g. # '-0700' # WEEKDAY The 3-letter name of the day of week the # message was sent, e.g. 'Thu'. # YEAR The year the message was sent. Time expansion # macros can either use the time specified in # the log message, e.g. the time the log message # is sent, or the time the message was received # by the log server. This is controlled by the # use_time_recvd() option. # ----------------- ----------------------------------------------- # # Options: # # Name Type Description Default # -------------- ------ -------------------------------- -------- # compress y/n Compress the resulting logfile global # using zlib. NOTE: this is not setting # implemented as of 1.3.14. # reate_dirs y/n Enable creating non-existing no # directories. # dir_perm num The permission mask of 0600 # directories created by # syslog-ng. Log directories are # only created if a file after # macro expansion refers to a # non-existing directory, and dir # creation is enabled using # create_dirs(). # encrypt y/n Encrypt the resulting file. global # NOTE: this is not implemented as setting # of 1.3.14. # fsync y/n Forces an fsync() call on the # destination fd after each write. # Note: this may degrade # performance seriously # group string Set the group of the created root # filename to the one specified. # log_fifo_size num The number of entries in the global # output fifo. setting # owner string Set the owner of the created root # filename to the one specified. # perm num The permission mask of the file 0600 # if it is created by syslog-ng. # remove_if_older num If set to a value higher than 0, 0 # before writing to a file, # syslog-ng checks whether this # file is older than the specified # amount of time (specified in # seconds). If so, it removes the # existing file and the line to # be written is the first line in # a new file with the same name. # In combination with e.g. the # $WEEKDAY macro, this is can be # used for simple log rotation, # in case not all history need to # be kept. # sync_freq num The logfile is synced when this global # number of messages has been setting # written to it. # template string Specifies a template which # specifies the logformat to be # used in this file. The possible # macros are the same as in # destination filenames. # template_escape y/n Turns on escaping ' and " in yes # templated output files. It is # useful for generating SQL # statements and quoting string # contents so that parts of your # log message don't get # interpreted as commands to the # SQL server. # -------------- ------ -------------------------------- -------- # # program - This driver fork()'s executes the given program with # the given arguments and sends messages down to the # stdin of the child. # # The program driver has a single required parameter, # specifying a program name to start and no options. # The program is executed with the help of the current # shell, so the command may include both file patterns # and I/O redirection, they will be processed. # # NOTE: the program is executed once at startup, and # kept running until SIGHUP or exit. The reason is to # prevent starting up a large number of programs for # messages, which would imply an easy DoS. # tcp/udp - This driver sends messages to another host on the # local intranet or internet using either UDP or TCP # protocol. # # Both drivers have a single required argument # specifying the destination host address, where # messages should be sent, and several optional # parameters. Note that this differs from source # drivers, where local bind address is implied, and # none of the parameters are required. # # Options: # # Name Type Description Default # -------------- ------ -------------------------------- -------- # localip string The IP address to bind to before 0.0.0.0 # connecting to target. # localport num The port number to bind to. 0 # port/destport num The port number to connect to. 514 # -------------- ------ -------------------------------- -------- # usertty - This driver writes messages to the terminal of a # logged-in user. # # The usertty driver has a single required argument, # specifying a username who should receive a copy of # matching messages, and no optional arguments. # unix-dgram - unix-stream - This driver sends messages to a unix # socket in either SOCK_STREAM or SOCK_DGRAM mode. # # Both drivers have a single required argument # specifying the name of the socket to connect to, and # no optional arguments. #---------------------------------------------------------------------- #---------------------------------------------------------------------- # Standard Log file locations #---------------------------------------------------------------------- destination authlog { file("/var/log/auth.log"); }; destination bootlog { file("/var/log/boot.log"); }; destination debug { file("/var/log/debug"); }; destination explan { file("/var/log/explanations"); }; destination messages { file("/var/log/messages"); }; destination routers { file("/var/log/routers.log"); }; destination secure { file("/var/log/secure"); }; destination spooler { file("/var/log/spooler"); }; destination syslog { file("/var/log/syslog"); }; destination user { file("/var/log/user.log"); }; #---------------------------------------------------------------------- # Special catch all destination sorting by host #---------------------------------------------------------------------- destination hosts { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; #---------------------------------------------------------------------- # Forward to a loghost server #---------------------------------------------------------------------- #destination loghost { udp("10.1.1.254" port(514)); }; #---------------------------------------------------------------------- # Mail subsystem logs #---------------------------------------------------------------------- destination mail { file("/var/log/mail.log"); }; destination mailerr { file("/var/log/mail/errors"); }; destination mailinfo { file("/var/log/mail/info"); }; destination mailwarn { file("/var/log/mail/warnings"); }; #---------------------------------------------------------------------- # INN news subsystem #---------------------------------------------------------------------- destination newscrit { file("/var/log/news/critical"); }; destination newserr { file("/var/log/news/errors"); }; destination newsnotice { file("/var/log/news/notice"); }; destination newswarn { file("/var/log/news/warnings"); }; #---------------------------------------------------------------------- # Cron subsystem #---------------------------------------------------------------------- destination cron { file("/var/log/cron.log"); }; destination crondebug { file("/var/log/cron/debug"); }; destination cronerr { file("/var/log/cron/errors"); }; destination croninfo { file("/var/log/cron/info"); }; destination cronwarn { file("/var/log/cron/warnings"); }; #---------------------------------------------------------------------- # LPR subsystem #---------------------------------------------------------------------- destination lpr { file("/var/log/lpr.log"); }; destination lprerr { file("/var/log/lpr/errors"); }; destination lprinfo { file("/var/log/lpr/info"); }; destination lprwarn { file("/var/log/lpr/warnings"); }; #---------------------------------------------------------------------- # Kernel messages #---------------------------------------------------------------------- destination kern { file("/var/log/kern.log"); }; destination kernerr { file("/var/log/kernel/errors"); }; destination kerninfo { file("/var/log/kernel/info"); }; destination kernwarn { file("/var/log/kernel/warnings"); }; #---------------------------------------------------------------------- # Daemon messages #---------------------------------------------------------------------- destination daemon { file("/var/log/daemon.log"); }; destination daemonerr { file("/var/log/daemons/errors"); }; destination daemoninfo { file("/var/log/daemons/info"); }; destination daemonwarn { file("/var/log/daemons/warnings"); }; #---------------------------------------------------------------------- # Console warnings #---------------------------------------------------------------------- destination console { file("/dev/tty12"); }; #---------------------------------------------------------------------- # All users #---------------------------------------------------------------------- destination users { usertty("*"); }; #---------------------------------------------------------------------- # Examples of programs that accept syslog messages and do something # programatically with them. #---------------------------------------------------------------------- #destination mail-alert { program("/usr/local/bin/syslog-mail"); }; #destination mail-perl { program("/usr/local/bin/syslog-mail-perl"); }; #---------------------------------------------------------------------- # Piping to Swatch #---------------------------------------------------------------------- #destination swatch { program("/usr/bin/swatch --read-pipe=\"cat /dev/fd/0\""); }; #---------------------------------------------------------------------- # Database notes: # # Overall there seems to be three primary methods of putting data from # syslog-ng into a database. Each of these has certain pros and cons. # # FIFO file: Simply piping the template data into a First In, First # Out file. This will create a stream of data that will # not require any sort of marker or identifier of how # much data has been read. This is the most elegant of # the solutions and probably the most unstable. # # Pros: Very fast data writes and reads. Data being # inserted into a database will be near real # time. # # Cons: Least stable of all the possible solutions, # and could require a lot of custom work to # make function on any particular Unix system. # # Loss of the pipe file will cause complete # data loss, and all following data that would # have been written to the FIFO file. # # Buffer file: While very similar to a FIFO file this is would be a # text file which would buffer all the template # output information. Another program from cron or # similar service would then run and source the buffer # files and process the data into the database. # # Pros: Little chance of losing data since everything # will be written to a physical file much like # the regular logging process. # # This method gives a tremendous amount of # flexibility since there would be yet another # opportunity to filter logs prior to inserting # any data into the database. # # Cons: Because there must be some interval between # the processing of the buffer files, there will # be a lag before the data is inserted in to the # database. # # There is also a slight chance of data corruption # (ie bad insert command) if the system crashes # during a write, although this scenero is very # unlikely. # # Another possible issue is that because multiple # buffer files be written, the previously run # sourcing file could get behind the data # insertion if there is a very large quantity of # logs being written. This will totally depend # on the system that this is running on. # # Program: The least elegant of the solutions. This method is to # send the stream of data through some further interrupter # program such as something in Perl or C. That program # will then take some action based off the data which # could include writing to a database similarly to the # program "sqlsyslogd". # # Pros: Allows complete control of the data, and as much # post processing as required. # # Cons: Slowest of all the forms. Since the data will # have to go through some post processing it will # cause data being written to the database to # remain behind actual log records. This could # cause a race condition in that logging is lost # either due to system crash, or high load on # the logging system. # #---------------------------------------------------------------------- #---------------------------------------------------------------------- # Writing to a MySQL database: # # Assumes a table/database structure of: # # CREATE DATABASE syslog; # USE syslog; # # CREATE TABLE logs ( host varchar(32) default NULL, # facility varchar(10) default NULL, # priority varchar(10) default NULL, # level varchar(10) default NULL, # tag varchar(10) default NULL, # date date default NULL, # time time default NULL, # program varchar(15) default NULL, # msg text, seq int(10) unsigned NOT NULL auto_increment, # PRIMARY KEY (seq), # KEY host (host), # KEY seq (seq), # KEY program (program), # KEY time (time), # KEY date (date), # KEY priority (priority), # KEY facility (facility)) # TYPE=MyISAM; # #---------------------------------------------------------------------- # Piping method #---------------------------------------------------------------------- #destination database { pipe("/tmp/mysql.pipe" # template("INSERT INTO logs (host, facility, # priority, level, tag, date, time, program, # msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', # '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', # '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") # template-escape(yes)); }; #---------------------------------------------------------------------- # Buffer file method #---------------------------------------------------------------------- destination database { file("/var/log/dblog/fulllog.$YEAR.$MONTH.$DAY.$HOUR.$MIN.$SEC" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) template-escape(yes)); }; #---------------------------------------------------------------------- # Program method (alternate using sqlsyslogd): # # Notes: This is not a bad process, but lacks very much flexibility # unless more changes are made to the source of sqlsyslogd. # This is because sqlsyslogd assumes the data in a larger # object style instead of breaking it down into smaller # columnar pieces. #---------------------------------------------------------------------- #destination database { program("/usr/local/sbin/sqlsyslogd -u # sqlsyslogd -t logs sqlsyslogs2 -p"); }; #---------------------------------------------------------------------- # Since we probably will not be putting ALL of our logs in the database # we better plan on capturing that data that we will be discarding for # later review to insure we did not throw anything away we really # should have captured. #---------------------------------------------------------------------- destination db_discard { file("/var/log/discard.log"); }; #---------------------------------------------------------------------- # Filters #---------------------------------------------------------------------- # # Functions: # # Name Synopsis Description # -------------- ------------------------------ -------------------- # facility facility(facility[,facility]) Match messages # having one of the # listed facility code. # filter Call another filter rule and # evaluate its value # host host(regexp) Match messages by # using a regular # expression against # the hostname field # of log messages. # level/priority level(pri[,pri1..pri2[,pri3]]) Match messages based # on priority. # match Tries to match a regular # expression to the message # itself. # program program(regexp) Match messages by # using a regular # expression against # the program name # field of log messages #---------------------------------------------------------------------- # NOTES: # # Getting filtering to work right can be difficult because while the # syntax is fairly simple, it is not well documented. To illustrate # a brief lesson on filtering and to explain the majority of the # mechanics, we shall use the filter from the PostgreSQL database # how-to page found at: http://www.umialumni.com/~ben/SYSLOG-DOC.html # # This is a perfect and somewhat complex example to use. In its # original form it resembles: # # filter f_postgres { not( # (host("syslogdb") and facility(cron) and level(info)) # or (facility(user) and level(notice) # and ( match(" gethostbyaddr: ") # or match("last message repeated ") # ) # ) # or ( facility(local3) and level(notice) # and match(" SYSMON NORMAL ")) # or ( facility(mail) and level(warning) # and match(" writable directory") # ) # or ( ( host("dbserv1.somecompany.com") # or host("dbserv2.somecompany.com") # ) # and facility(auth) and level(info) # and match("su oracle") and match(" succeeded for root on /dev/") # ) # ); }; # # While in this form, it does not induce a tremendous amount of # insight on what the specific filter is attempting to accomplish. In # reformatting the filter to resemble something a bit more human # readable, it would look like: # # filter f_postgres { not # ( # ( # host("syslogdb") and # facility(cron) and # level(info) # ) or # ( # facility(user) and # level(notice) and # ( # match(" gethostbyaddr: ") or # match("last message repeated ") # ) # ) or # ( # facility(local3) and # level(notice) and # match(" SYSMON NORMAL ") # ) or # ( # facility(mail) and # level(warning) and # match(" writable directory") # ) or # ( # ( # host("dbserv1.somecompany.com") or # host("dbserv2.somecompany.com") # ) and # facility(auth) and # level(info) and # match("su oracle") and # match(" succeeded for root on /dev/") # ) # ); # }; # # Now in this form we can now begin to see what this filter has been # attempting to accomplish. We can now further breakdown each logical # section and explain the different methods: # # [1] As in all statements in syslog-ng, each of the beginnings and # endings must be with a curly bracket "{" "}" to clearly denote # the start and finish. # # In this filter, the entire filter is preferred by a "not" to # indicate that these are the messages that we are NOT interested # in and should be the ones filtered out. All lines of logs that # do not match these lines will be sent to the destination. # # { not # # [2] The first major part of the filter is actually a compound # filter that has two parts. Because the two parts are separated # by an "or", only one of the two parts must be matched for that # line of log to be filtered. # # [2a] In the first part of this filter there are three requirements # to be met for the filter to take affect. These are the host # string "syslogdb". the facility "cron", and the syslog level # of info. # # ( # ( # host("syslogdb") and # facility(cron) and # level(info) # ) or # # [2b] In the second part of the filter, which in itself is a # compound filter, there are three requirements as well. These # are that the facility of "user", and the log level of "notice" # are met in addition to one of the two string matches that are # shown in the example. # # ( # facility(user) and # level(notice) and # ( # match(" gethostbyaddr: ") or # match("last message repeated ") # ) # ) or # # [3] In the section of the filter there are once again three # requirements to fire off a match which are a facility of "level3" # a log level of "notice" and a sting match of " SYSMON NORMAL ". # # ( # facility(local3) and # level(notice) and # match(" SYSMON NORMAL ") # ) or # # [4] This part of the filter is very similar to the previous # filter, but with different search patterns. # # ( # facility(mail) and # level(warning) and # match(" writable directory") # ) or # # [5] The last section of the filter is also a compound filter # that to take affect will require that one of two hosts # are matched, the facility of "auth", and log level of # "info" occur in addition to the two string matches. # # ( # ( # host("dbserv1.somecompany.com") or # host("dbserv2.somecompany.com") # ) and # facility(auth) and # level(info) and # match("su oracle") and # match(" succeeded for root on /dev/") # ) # # [6] As in all command sets in syslog-ng, each of the statements # must be properly closed with the correct ending punctuation # AND a semi-colon. Do not forget both, or you will be faced with # an error. # # ); }; # # While this may not be the most complete example, it does cover the # majority of the options and features that are available within the # current version of syslog-ng. #---------------------------------------------------------------------- #---------------------------------------------------------------------- # Standard filters for the standard destinations. #---------------------------------------------------------------------- filter f_auth { facility(auth, authpriv); }; filter f_authpriv { facility(authpriv); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_local1 { facility(local1); }; filter f_local2 { facility(local2); }; filter f_local3 { facility(local3); }; filter f_local4 { facility(local4); }; filter f_local5 { facility(local5); }; filter f_local6 { facility(local6); }; filter f_local7 { facility(local7); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_messages { facility(daemon, kern, user); }; filter f_news { facility(news); }; filter f_spooler { facility(uucp,news) and level(crit); }; filter f_syslog { not facility(auth, authpriv) and not facility(mail); }; filter f_user { facility(user); }; #---------------------------------------------------------------------- # Other catch-all filters #---------------------------------------------------------------------- filter f_crit { level(crit); }; #filter f_debug { not facility(auth, authpriv, news, mail); }; filter f_debug { level(debug); }; filter f_emergency { level(emerg); }; filter f_err { level(err); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; #---------------------------------------------------------------------- # Filer for the MySQL database pipe. These are things that we really # do not care to see otherwise they may fill up our database with # garbage. #---------------------------------------------------------------------- #filter f_db { not facility(kern) and level(info, warning) or # not facility(user) and level(notice) or # not facility(local2) and level(debug); }; # #filter f_db { not match("last message repeated ") or # not match("emulate rawmode for keycode"); }; # #filter f_discard { facility(kern) and level(info, warning) or # facility(user) and level(notice) or # facility(local2) and level(debug); }; # #filter f_discard { match("last message repeated ") or # match("emulate rawmode for keycode"); }; #---------------------------------------------------------------------- # Logging #---------------------------------------------------------------------- # # Notes: When applying filters, remember that each subsequent filter # acts as a filter on the previous data flow. This means that # if the first filter limits the flow to only data from the # auth system, a subsequent filter for authpriv will cause # no data to be written. An example of this would be: # # log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); # source(s_udp); filter(f_auth); # filter(f_authpriv); destination(authlog); }; # # So, one can cancel out the other. # # There are also certain flags that can be attached to each of the log # statements: # # Flag Description # -------- ---------------------------------------------------------- # catchall This flag means that the source of the message is ignored, # only the filters are taken into account when matching # messages. # fallback This flag makes a log statement 'fallback'. Being a # fallback statement means that only messages not matching # any 'non-fallback' log statements will be dispatched. # final This flag means that the processing of log statements ends # here. Note that this doesn't necessarily mean that # matching messages will be stored once, as they can be # matching log statements processed prior the current one. #---------------------------------------------------------------------- #---------------------------------------------------------------------- # Standard logging #---------------------------------------------------------------------- log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_auth); destination(authlog); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_local7); destination(bootlog); }; #log{ source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); # source(s_udp); filter(f_debug); destination(debug); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_local1); destination(explan); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_local5); destination(routers); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_messages); destination(messages); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_authpriv); destination(secure); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_spooler); destination(spooler); }; log { source(s_dgram); source(s_internal); source(s_kernel); source(s_tcp); filter(f_syslog); destination(syslog); }; #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); # source(s_udp); destination(syslog); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_user); destination(user); }; #---------------------------------------------------------------------- # Special catch all destination sorting by host #---------------------------------------------------------------------- log { source(s_dgram); source(s_internal); source(s_kernel); source(s_tcp); destination(hosts); }; #---------------------------------------------------------------------- # Send to a loghost #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); destination(loghost); }; #---------------------------------------------------------------------- # Mail subsystem logging #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); # source(s_udp); filter(f_mail); destination(mail); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_mail); filter(f_notice); destination(mailinfo); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_mail); filter(f_warn); destination(mailwarn); }; #---------------------------------------------------------------------- # INN subsystem logging #---------------------------------------------------------------------- log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_news); filter(f_crit); destination(newscrit); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_news); filter(f_err); destination(newserr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_news); filter(f_notice); destination(newsnotice); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_news); filter(f_warn); destination(newswarn); }; #---------------------------------------------------------------------- # Cron subsystem logging #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_tcp); # source(s_udp); filter(f_cron); destination(crondebug); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_cron); filter(f_err); destination(cronerr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_cron); filter(f_info); destination(croninfo); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_cron); filter(f_warn); destination(cronwarn); }; #---------------------------------------------------------------------- # LPR subsystem logging #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_tcp); # source(s_udp); filter(f_lpr); destination(lpr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_lpr); filter(f_err); destination(lprerr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_lpr); filter(f_info); destination(lprinfo); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_lpr); filter(f_warn); destination(lprwarn); }; #---------------------------------------------------------------------- # Kernel subsystem logging #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); # source(s_udp); filter(f_kern); destination(kern); }; log { source(s_dgram); source(s_internal); source(s_kernel); source(s_tcp); filter(f_kern); filter(f_err); destination(kernerr); }; log { source(s_dgram); source(s_internal); source(s_kernel); source(s_tcp); filter(f_kern); filter(f_info); destination(kerninfo); }; log { source(s_dgram); source(s_internal); source(s_kernel); source(s_tcp); filter(f_kern); filter(f_warn); destination(kernwarn); }; #---------------------------------------------------------------------- # Daemon subsystem logging #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_tcp); # source(s_udp); filter(f_daemon); destination(daemon); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_daemon); filter(f_err); destination(daemonerr); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_daemon); filter(f_info); destination(daemoninfo); }; log { source(s_dgram); source(s_internal); source(s_tcp); filter(f_daemon); filter(f_warn); destination(daemonwarn); }; #---------------------------------------------------------------------- # Console logging #---------------------------------------------------------------------- # 16-Mar-03 - REP - Removed logging to the console for performance # reasons. Since we are not really going to be # looking at the console all the time, why log there # anyway. #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); filter(f_syslog); destination(console); }; #---------------------------------------------------------------------- # Logging to a database #---------------------------------------------------------------------- #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); filter(f_db); destination(database); }; #log { source(s_dgram); # source(s_internal); # source(s_kernel); # source(s_tcp); filter(f_discard); destination(db_discard); };
- Fő bajom a szintaxissal: ha egy-egy objektumot (source, filter vagy destination) csak egyszer haszálunk, akkor is, mielőtt használhatnánk, explicite deklarálni kell valahol és nevet adni neki. Nem csinálhatunk pl. ilyet:
# EZ SAJNOS ÉRVÉNYTELEN log { source(pipe("/proc/kmsg" log_prefix("kernel: "))); filter(not level(debug)); destination(file("/var/log/kern.log")); };
[szerkesztés] 1.3 socklog
- Egyszerű, mint a százas szög: olvas "valahonnan", és stdoutra ír.
- Kb. 260kB memóriát foglal (a hozzá tartozó svlogd-vel együtt kb. 300kB-ot).
- Ehhez képest egy nem túl összetett konfigurációjú syslog-ng 5+ MB-ot, és 80MB fölötti virtuális tárterületet.
- A "valahonnan" lehet:
- unix domain socket
- udp socket
- UCSPI (tcp socket, unix domain socket, ssl socket) (később lesz róla szó)
- Ha több helyről is szeretnénk olvasni, egyszerűen több példányt futtatunk.
- Az üzenetek szétválogatása az svlogd feladata.
Előnyök:
- Kicsi, aranyos, egyszerű
- Majdnem olyan rugalmas, mint a syslog-ng
A socklog+svlogd kombináció hátrányai:
- A konfiguráció elaprózódik (sok kis file, amit ugyan generálhatunk egy nagyból, de ahhoz dolgozni kell)
- Egy konfigurációs változtatás adott esetben több file módosítását jelenti (pl. az egyikben kizárunk egy üzenettípust, a másikban engedélyezzük)
- Sok hoszttól üzeneteket fogadni és azokat szétválogatni bonyolult, rosszul skálázódik (bár scriptelhető)
[szerkesztés] 2 Ajánlott irodalom
- http://www.campin.net/syslog-ng/faq.html - syslog-ng faq
- http://sial.org/howto/logging/syslog-ng/ - egy esettanulmány syslog-ng-vel
[szerkesztés] 3 Potenciális zh-kérdések
- Mik a főbb problémák a hagyományos unixos sysloggal?
- Mik a syslog-ng fő jellemzői, előnyei, hátrányai?
- Hogyan működik a socklog? Hogyan építhetünk vele teljes rendszernapló-megoldást? Mik az előnyei/hátrányai egy syslog-ng-alapú megoldáshoz képest?
- Mik az előnyei/hátrányai az udp ill. a tcp fölötti hálózati naplózásnak?
- Mi a különbség a logrotate és az svlogd naplórotációs mechanizmusa között? Melyik miért jobb?
- Mi a különbség az alábbi két syslog-ng config-részlet között?
- a)
source local { unix-stream("/dev/log"); internal(); }; source jail1 { unix-stream("/jail/dns/dev/log"); }; source jail2 { unix-stream("/jail/www/dev/log"); };
- b)
source local { unix-stream("/dev/log"); internal(); unix-stream("/jail/dns/dev/log"); unix-stream("/jail/www/dev/log"); };
- Mik a bajok a logcheck-kel? (Az előadáson elhangzott, valaki könyörüljön meg a többieken és írja fel a wikire...)
- Mi a TAI64N? Milyen előnyei vannak más, ugyanezt a célt szolgáló megoldásokkal szemben? Van-e hátránya?
- Mitől jó vagy rossz egy naplófájl-formátum?