NDL2DB History 21.8.2011 v2.11 * fixed db functions inclusion 16.8.2011 v2.10 * php 5.3 compatibility fixes * configurator2: sub class open fixed * replaced ereg() with preg_match() 28.4.2010 v2.03-RC1 * fixed php5 errors and warnings (KvE) - replaced eregi_replace("..." by prep_replace("/.../i" - replace eregi("..." by preg_match("/.../i" - Where necessary / in expressions escaped by changing to \/ - comment out register_long_arrays and magic_quotes_gpc in php.ini 26.9.2008 v2.02-beta01 * statistics: NL_LINE reads is the same as total reads. corrected counting. 25.09.2008 v2.02-beta01 * func_nf1: $st = $state||'Online'; line 765 seems to be that doesnt works. state Down hasnt been inserted into the db. Keyword parsing and State parsing now fixed. * record hashes: nl_node includes keword and nl_line includes state 02.06.2008 v2.01-alpha10 * Once more NL_ENTRY insert errors: - Warning: record allways exists in NL_ENTRY: 127, 714056. - ERROR: insert record failed NL_ENTRY: insert into nl_entry (id_nodelist, id_nl_line) VALUES (127,684034) - ERROR: SQL error: (1062) Duplicate entry '127-714094' for key 1 Problem occures when running 2 tasks and 2 tasks works on the same nodelist: # 01:43:46 [1] nodelist import (1) of: 'nodelist.006' (1995) # 03:20:54 [1] Warning: One or more record insert errors. Possible DB inconsistency. # 01:44:53 [0] nodelist RE-import (1) of: 'nodelist.006' (1995) # 03:21:57 [0] Warning: One or more record insert errors. Possible DB inconsistency. The workqueue update routine now checks the sql_affectedrows() after changing the state of the affected nodelist to prevent duplicate execution on the same nodelist at the same time. State: fixed. * nl_line dupes: id_nl_line 766420 + 766421 2x 358895:74521f51f7adb69789351bbce29f17db same id_nl_node, but only one id_nl_line in nl_entry. procedure nl_line produces dupe entries (about 10%, 5800 and more within half a million nl_line records), that needs to checked and eliminated (dupeelim maintenance) - results in a table restructure: nl_line index set to unique multiple inserts ends in an sql insert error that can be detected. State: fixed. 28.04.2008 v2.01-alpha09 * Performance counters analyze results, that minimum time that is needed to read hash array values compares to the maximum readin time that is needed to readin a db record. This results in code rewrite by removing hash arrays (reduced memory usage, reduced access time). 27.04.2008 v2.01-alpha08 * runtime conflicts inserting new records and identifying the IDs of the inserted records. Relates also to the nl_entry dupes. Search actual records by their hash key. Fixed. * hash statistics: add read hash from disc counter 26.04.2008 v2.01-alpha07 * Undefined variable: sres1 func_nf1.php on line 529 Fixed. * Undefined offset: 10339 in func_nf1.php on line 529 under some conditions. Fixed by isset(). 25.04.2008 v2.01-alpha06 * changed table structure: nl_wqueue, add #nodelist lines + if multiple same sort order level exists, added addtl. sort order by year, day 25.04.2008 v2.01-alpha05 * modified sord order to 1 for single update to workqueue. 22.04.2008 v2.01-alpha05 * multitasking import: new import and re-import at the same time occures. A timing problem in search next workqueue entry and set it to under construction. Recheck state on updating record. 21.04.2008 v2.01-alpha05 * Re-import of nodelist creates 2nd record in nodelist with new id_nodelist. Nodelist-found flag name mistyped. Fixed. 20.04.2008 v2.01-alpha05 * using strict in array_search() to prevent insert into db errors. * on insert into db failures skip following db inserts for a single nodelist line because related keys are missing and inserts fails too. + added a FORCE import switch + added type 4 insert: insert a single nodelist to DB w/o workqueue handling 12.04.2008 v2.01-alpha04 * Notice: Use of undefined constant PROG - assumed 'PROG' in func_lib line 462 func_lib and constants inclusion needed before starting logfile() * Notice: Undefined offset: 7497 in ndl2db line 402 now checking array index with isset(). Fixed. * import_file: unable to identify '...' packeridentifier() => current(), next(), key() array functions doesn't work under php 4.4.1 replaced by foreach() loop. Fixed. 10.04.2008 v2.01-alpha04 + Problem: execute workqueue deletes all records in workqueue no log message why records are deleted. The machine has rebooted, the drive mapping hasn't been reestablished. Missing source drive prevents the script to import the source files from within the workqueue handle loop, so the records gets deleted w/o any further infos. Added global check that the source path is accessible otherwise the script terminates. Fixed. 09.04.2008 v2.01-alpha4 * multiple task run produces nl_entry sql insert with duplicate id_nodelist, id_nl_line pairs. As of a result, the array_push()/array_slice()/array_splice() combination handled thru an bulk array of 5000 elements hasn't been inserted and updated. About 20 of 5000 pairs was dupes. Now inserting each single id_nodelist, id_nl_line pair with checking if pair is allways in the db. Fixed. 08.04.2008 v2.01-alpha04 * type 2+3: check if allways in workqueue results 0 forever as a wrong resultset were used that produced dupe entries. fixed. - removed inclusion for unused library func_dli - wqorder: removed inclusion for unused library func_dli 07.04.2008 v2.01-alpha03 + add SID to log lines to identify session# in multitasking environments * type 2+3: search or add nodelists to workqueue added check in nfmt1() if nodelist allways in workqueue 07.04.2008 v2.01-alpha02 * create flag dir if not exists + add install.txt after some failures and lost records in a production nl_wqueue 06.04.2008 v2.01-alpha01 + added wqorder.php Workqueue ordering script 05.04.2008 v2.01-alpha01 + added workqueue handling - scan to workqueue - execute from workqueue (*default) - add single file to workqueue + identify type 1 nodelist 4d (new format, with zones) or nodelist 2d (old format, w/o zones) * simplier process indicator + added session handling and process termination handling to interrupt long scan processes by semaphore files (needs directory 'flag') 30.03.2008 + added AKA, Sysops name and Systemname queries. + added addtl. query links into the output form for related queries. 29.03.2008 i first web query plattform (NLARCHIVE2 script) 02.03.2008 * modified nfmt1(), added save records nl_node, nl_line, nl_entry controlled by hashes. i full Nodelist import tests successful. i started full NODELIST import from Fido-History-Project Archiv 01.03.2008 + added func_lib.php, moved out global sub functions from func_dli + added save records nl_nodelist (includes hashes) + added doy2date() (daynumber and year to date) 26.02.2008 * modified extractLastname() 24.02.2008 + added extractLastname() 22.02.2008 * import_if_nodelist() + added inspectfmt(), different nodelist formats switch + added subfunction func_nf1.php nfmt1() 21.02.2008 * import_if_nodelist() + added Nodelist admin line check + added CRC16() calculation and check 20.02.2008 + added import_subdir() 19.02.2008 i Starting 2nd port to PHP + added global logfile() + added compress.cfg parser + added packeridentifier() + added tempdir handling, create, cleanup, remove + added unpackndl2tmp() 14.02.2008 i Found Sources from 4th Dec 07 09.02.2008 i Harddisc crash, all sources v1.0 lost. 31.01.2007 i Version 1.0 of Port Perl to PHP Nodelist Importer Started Nodelists import to FidoHis2 database. 04.12.2007 i Version 1.0 of Port Perl to PHP Nodelist Importer for Fido-History-Project Node- and Pointlists to MySQL5 30.11.2007 - v1.0 (PHP) i Port to PHP 22.02.2007 i Suggestion by Alex Woick of DB structure with a PL script for import nodelist data