PHP-Nuke Download Section vs. FileBase
How to import from Filebase ...
- Section 1 - Analyse I, Filebase Synchronisation
[15.6.2005]
Download Section Update Prozess
Gegeben: AMBROSIA60 Filebase Q:\dirs\subdirs filebase files.bbs format
BBS2WEB Q:\BBS2WEB\www HTM files
Erstellung:
CATEGOR.LST list of htm files
CATEGORY.TXT list of MySQL Nuke Download Categories
DOWNLOAD.SQL update list for import to MySql
categor.lst
cd q:\bbs2web\www
dir *.htm>cateor.lst
category.txt
I:\Apache\mysql\bin>mysqldump --host=localhost --user=name --password=name
--result-file=category.txt nuke nuke_downloads_categories
download.sql x1)
[new program]
Import to MySQL:
I:\Apache\mysql\bin>mysql --host=localhost --user=name
--password=name nuke<download.sql
x
1) download.sql Format:
..............................................................................
insert into nuke_downloads_categories values (NULL, 'Box_Info',
'AMBROSIA60 Box Infos','39');
insert into nuke_downloads_downloads values (NULL,'categorie#','0','Filename',
'url x2)','description x3)','filedate yyyy-mm-dd hh:mm:ss x4)','Allfix','',
'0','uli','0.0000','0','0','filesize x5)','version x6)','');
quit
..............................................................................
x
2) URL: Link to filebase i.e.
http://ambrosia60.dd-dns.de/files/service/24000_fd/r24pnt.z40
Link aus *.htm files
x
3) description from htm, 2nd line
x
4) date + time from htm, fields Date: + Time:
copy Date: yyyy-mm-dd + " " + Time: hh:mm:ss
x
5) filesize from htm, field Size:
x
6) version from htm, field description, 2nd line
if day-### use ### with '#' and year-field -> i.e. day-161 -> #161/2005
Alternative?
direkt aus FILES.BBS auslesen?
weiterhin noch unklar: URL Link
URL Link Base Path
http://ambrosia60.dd-dns.de/files
ist klar, aber danach ... wie setzt sich
/service/24000_fd/r24pnt.z40
zusammen?
BBS2WEB greift auf MAX FAREAS.DAT zu.
FAREAS.DAT setzt sich aus Definitionen aus FILEAREA.CTL zusammen,
das wiederum die Orginal Pfade enthaelt: i.e. Q:\service\24000_fd
entweder [l:\max\]FILEAREA.CTL oder [q:\ra\]FLSEARCH.CTL auslesen
basepath + path + filename = url-link
Auslesen FAREA.DAT ? (seq. file)
offset 70 (ab pos 70 beginnt erste area definition)
areaname + chr(0) + path + chr(0) + accesslevel (klartext!) + chr(0)
+ description + 4x 16 bytes + 2 ...
chr(0) + type datelist, type free ?!?
28 36 30 29 ³ 5F 42 6F 78 ³ 5F 49 6E 66 ³ 6F 00 40 00 (60)_Box_Info.@.
^^
00 00 29 00 ³ 00 00 17 00 ³ 01 00 0A 00 ³ 00 00 00 00 ................
1E 00 00 00 ³ 00 00 00 00 ³ 00 00 7A 00 ³ 00 00 40 03 ..........z...@.
^^
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ................
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 46 ...............F
49 4C 45 4C ³ 49 53 54 00 ³ 51 3A 5C 66 ³ 69 6C 65 6C ILELIST.Q:\filel
69 73 74 5C ³ 00 50 72 69 ³ 76 69 6C 00 ³ 46 69 6C 65 ist\.Privil.File
6C 69 73 74 ³ 65 6E 00 40 ³ 00 00 00 3F ³ 00 00 00 13 listen.@...?...#
00 01 00 05 ³ 00 00 00 00 ³ 00 1A 00 00 ³ 00 00 00 00 ................
00 00 00 69 ³ 00 00 00 40 ³ 03 00 00 00 ³ 00 00 00 00 ...i...@........
00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ................
00 00 00 00 ³ 00 00 00 00 ³ 57 49 32 00 ³ 51 3A 5C 44 ........WI2.Q:\D
3x16 + 8 + 8
last record:
057A5 20 20 00 40 ³ 00 00 00 4D ³ 00 00 00 1C ³ 00 01 00 08 .@...M........
^start
057B5 00 00 00 00 ³ 00 23 00 00 ³ 00 00 00 00 ³ 00 00 00 8D .....#..........
057C5 00 00 00 40 ³ 03 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ...@............
057D5 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ³ 00 00 00 00 ................
057E5 00 00 00 00 ³ 54 52 4E 5F ³ 41 52 00 51 ³ 3A 5C 54 52 ....TRN_AR.Q:\TR
057F5 45 49 42 45 ³ 52 4E 5C 50 ³ 41 43 4B 45 ³ 52 5C 00 50 EIBERN\PACKER\.P
05805 72 69 76 69 ³ 6C 00 54 72 ³ 65 69 62 65 ³ 72 4E 65 74 rivil.TreiberNet
05815 5F 2D 5F 50 ³ 61 63 6B 70 ³ 72 6F 67 72 ³ 61 6D 6D 65 _-_Packprogramme
05825 20 20 20 20 ³ 20 20 20 20 ³ 20 20 20 20 ³ 20 20 20 00 .
????
Yep, Offset faengt nicht bei 70 oder so an, sondern bei 0 ...
es sind jeweils komplette Records, mit binaerem VOrspann und
nachfolgendem PATH und DESCRIPTION. Laenge des Records scheint abhaengig
von der Pathlaenge und der Description ...
crawl thru files.bbs'ses
use files.bbs size as update indicator (filedate/time permanently changes!)
if filesize([path]files.bbs) <> stamp
analyze files.bbs
update sql script
flsearch.ctl alleine hilft nicht, da die struktur nicht
unbedingt mit der von fareas.dat uebereinstimmt -> use filearea.ctl
instead
weiterhin gruppierung und subcategories:
fidonet, webportale, filebase
fidonet ..... use list
webportale ... use list
filebase ..... div13, div3, div27, service,point sonderfaelle
ansonsten subcategory by root path
ie. sds, pdn, gnf, gfd, win32 usw.
groups.lst
analyze:
readin groups and dirs
exclude list [EXCLUDE]
ansonsten, areagroup
[CATEGORIE]
or
[CATEGORIE\subcategorie]
and
[CATEGORIE]
id=##
zu allen CATEGORIE und CATEGORIE\subcategorie muessen
IDs vorhanden sein, andernfalls
step1, create import.sql fuer main CATEGORIE
if all CATEGORIE includes id
step2, create import.sql fuer subcategorie's
analyze filearea.ctl
use path i.e. Q:\DIV13\box_info
search groups, if found path Q:\DIV13\BOX_INFO use this group
otherwise reduce path next lower level Q:\DIV13
search groups, if found path Q:\DIV13 use this group
otherwise reduce path ... no more? stop, skip
program? clipper ... mysql import is needed
php, without mysql import (see search index process) ???
search index uses perl ! (i:\apache2\apache\fbmaint\fbmaint.cmd
-> perl index.pl)
aber dafuer sitesearch
cd \Apache\Apache2\sitesearch\admin
i:\apache\php\php -f mspider.php i:/apache/apache2/sitesearch/admin/spider2.txt
spider2.txt includes links to htm to analyse (update)
variant I
1. import areas import areas to categories
from mx.fareas.dat, flsearch.ctl, config, ....
2. admin module sort/move from root areas to sub areas
variant II
1. import areas using "routing config" and
from mx.fareas.dat, flsearch.ctl, config, ....
configure root areas, use directorys to parentid's
see groups.lst
2. optional admin module
3. import areas2cat (sync/save ipaths to categories)
4. import files using db info
sync by compare filesize files.bbs
db: prefix_downloads_ipaths
farea area, displayareaname, path, level, description, parentid, fbsize
zu 4)
insert into nuke_downloads_downloads values (
NULL,
'categorie#', for downloads_ipath 1 to end, farea area -> #
'0',
'Filename', file(path+filename)
'url x2)', basepath + path + filename
'description x3)', from files.bbs
'filedate yyyy-mm-dd hh:mm:ss x4)', filedate(path+filename)
'Allfix',
'',
'0',
'uli', submitter
'0.0000',
'0',
'0',
'filesize x5)', filesize(path+filename)
'version x6)', from description?
''
);
ACHTUNG: PHP.INI config parameter
register_argc_argv = OFF -> ON !!!
output_buffering=4096 -> 0
php program:
1. read own config
2. read phpnuke config
3. read phpnuke db config
4. rest
http or console mode see mspider.php
CATEGOR.LST, BBS2WEB Output Directory List:
-------------------------------------------
Verzeichnis von Q:\BBS2WEB\WWW
06ER HTM 5.345 11.06.05 13:44
24000ZIP HTM 31.283 11.06.05 13:44
24000_V7 HTM 1.999 11.06.05 13:44
ALLFIX HTM 4.244 11.06.05 13:44
ALLFIXUT HTM 4.275 11.06.05 13:44
[...]
CATEGORY.TXT, MySqlDump _prefix_Downloads_* Output:
---------------------------------------------------
-- MySQL dump 9.11
--
-- Host: localhost Database: nuke
-- ------------------------------------------------------
-- Server version 4.0.20a-nt
--
-- Table structure for table `nuke_downloads_categories`
--
CREATE TABLE nuke_downloads_categories (
cid int(11) NOT NULL auto_increment,
title varchar(50) NOT NULL default '',
cdescription text NOT NULL,
parentid int(11) NOT NULL default '0',
PRIMARY KEY (cid),
KEY cid (cid),
KEY title (title)
) TYPE=MyISAM;
--
-- Dumping data for table `nuke_downloads_categories`
--
INSERT INTO nuke_downloads_categories VALUES (1,'Fidonet','Fidonet related Downloads',0);
INSERT INTO nuke_downloads_categories VALUES (3,'Webportale','Alles rund um Webinterface und
Webportale\r\n',0);
INSERT INTO nuke_downloads_categories VALUES (4,'PHP-Nuke','PHP-Nuke Installations- und
Upgrade Pakete. Auch Bundles.\r\n',3);
INSERT INTO nuke_downloads_categories VALUES (5,'Region24','Region 24 related files\r\n',1);
INSERT INTO nuke_downloads_categories VALUES (6,'Pointlisten','Region 24 Pointlisten\r\n',5);
INSERT INTO nuke_downloads_categories VALUES (7,'Scripts','Interessante Script Pakete\r\n',3);
INSERT INTO nuke_downloads_categories VALUES (45,'SERVICE','Globales Service Verzeichnis
(Nodelisten, Diffs,u.a.)',1);
INSERT INTO nuke_downloads_categories VALUES (44,'FNEWSFRQ','Latest FIDONEWS Fido Newsletters'
,1);
INSERT INTO nuke_downloads_categories VALUES (43,'FNEWS','FIDONEWS Fido Weekly Newsletters',1);
INSERT INTO nuke_downloads_categories VALUES (42,'WI2','Webinterface II Project Files',3);
INSERT INTO nuke_downloads_categories VALUES (41,'Filelist','Filelisten',39);
INSERT INTO nuke_downloads_categories VALUES (40,'Box_Info','AMBROSIA60 Box Infos',39);
INSERT INTO nuke_downloads_categories VALUES (39,'FileBase','AMBROSIA60 FileBase Mirror',0);
INSERT INTO nuke_downloads_categories VALUES (46,'NODELIST','Latest Fidonet Nodelist',1);
INSERT INTO nuke_downloads_categories VALUES (47,'NODEDIFF','Latest Fidonet Nodediff',1);
INSERT INTO nuke_downloads_categories VALUES (48,'R24PNT','R24PNT and R24PNT_D, Alle Region24
4D Pointlists Boss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (49,'R24PFQ','Latest R24PNT and R24PNT_D',1);
INSERT INTO nuke_downloads_categories VALUES (50,'Z2PNT','Z2PNT, Alle Zone2 4D Pointlists
Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (51,'Z2PntFRQ','Latest Z2PNT Zone2 4D Pointlists
Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (52,'XPNTZ2DI','Z2PNT_D, Alle Zone2 4D PointDiffs
Poss-Format',1);
INSERT INTO nuke_downloads_categories VALUES (53,'XPZ2DFRQ','Latest Z2PNT_D, Zone2 4D
PointDiff Poss-Format',1);
FILEAREA.CTL, Maximus Filebase Definition File:
-----------------------------------------------
FileArea BOX_INFO
DownLoad Q:\DIV13\box_info
acs Normal
Desc AMBROSIA(60)_Box_Info
Type DateList
Type Free
End FileArea
[...]
'ROUTING' Config File for Areas to Categories relations
[EXCLUDE] Paths to exclude
Q:\X
[ROOT] no Areas to Root Path !
[FIDONET] Root Categorie, ParentID=0 (!)
parent=0
Q:\DIV13\netz244 single, ungrouped areas
Q:\news
Q:\service
Q:\DIV13\region24
Q:\DIV27\I-BINKD
Q:\DIV13\P24CVRT
Q:\DIV13\fd
Q:\DIV13\mailutil
Q:\DIV13\fidosoft.244
Q:\DIV3\rules24
Q:\DIV3\RECLISTE
Q:\DIV27\I-UTIL
Q:\point
Q:\DIV13\FASTECHO
Q:\DIV27\ITRACK
Q:\div13\allfix
Q:\div13\allfixut
Q:\div27\ttick
Q:\SDS\ftsc
[WEBPORTALE] 2nd Main Categorie, ParentID=0 !
parent=0
Q:\DIV13\wi2 only one single Area
[FILEBASE] 3rd Main Categorie, ParentID=0 !
parent=0
Q:\DIV13 rest of areas in Q:\DIV13 directory
Q:\filelist
Q:\DIV27 rest of areas in Q:\DIV27 directory
Q:\DIV3 rest of areas in Q:\DIV3 directory
[XP] Subgroup in Maingroup "FILEBASE" (see above)
parent=FILEBASE ParentID <> 0, ParentID unknown, use Category Name
Q:\xp XP (Crosspoint) Group Areas, starting with Directory Q:\XP\...
[PDN] same as XP, Subgroup in Maingroup "FILEBASE" (see above)
parent=FILEBASE ParentID <> 0, ParentID unknown, use Category Name
Q:\PDN PDN Group Areas, starting with Directory Q:\PDN\...
[...]
1. Step, import to TEMP_DOWNLOADS_TABLE
records after 1st step import
downloads_temp
--------------
area parentid parentname path
--------- ----------- ----------- --------------------
EXCLUDE Q:\X\
ROOT
FIDONET 0
FIDONET Q:\DIV13\netz244\
FIDONET Q:\news\
FIDONET Q:\service\
FIDONET Q:\DIV13\region24\
FIDONET Q:\DIV27\I-BINKD\
FIDONET Q:\DIV13\P24CVRT\
FIDONET Q:\DIV13\fd
FIDONET Q:\DIV13\mailutil
FIDONET Q:\DIV13\fidosoft.244
FIDONET Q:\DIV3\rules24
FIDONET Q:\DIV3\RECLISTE
FIDONET Q:\DIV3\echolist
FIDONET Q:\point
FIDONET Q:\DIV13\FASTECHO
FIDONET Q:\DIV27\ITRACK
FIDONET Q:\div13\allfix
FIDONET Q:\div13\allfixut
FIDONET Q:\div27\ttick
FIDONET Q:\SDS\ftsc
WEBPORTALE 0
WEBPORTALE Q:\DIV13\wi2
FILEBASE 0
FILEBASE Q:\DIV13
FILEBASE Q:\filelist
...
2nd step: merge FILEAREA.CTL or FAREAS.DAT (Maximus) thru TEMP_DOWNLOADS_TABLE filter
area parentid parentname path MX.Areaname PHPNUKE-Areaname AccessLevel
--------- ----------- ----------- --------------------- --------------- ---------------- -----------
EXCLUDE Q:\X\ SECURE PROTECTED
ROOT
FIDONET 0
FIDONET Q:\DIV13\netz244\ NET244 NET244 NORMAL
FIDONET Q:\news\ NEWS NEWS NORMAL
FIDONET Q:\service\ SERVIC SERVIC x2) NORMAL
FIDONET Q:\DIV13\region24\ REGION24 REGION24 NORMAL
FIDONET Q:\DIV27\I-BINKD\ I-BINKD I-BINKD NORMAL
FIDONET Q:\DIV13\P24CVRT\ P24CVRT P24CVRT x2) NORMAL
FIDONET Q:\DIV13\fd FD FD NORMAL
FIDONET Q:\DIV13\mailutil MAILUTIL MAILUTIL NORMAL
FIDONET Q:\DIV13\fidosoft.244 SOFT244 SOFT244 NORMAL
FIDONET Q:\DIV3\rules24 RULES24 RULES24 NORMAL
FIDONET Q:\DIV3\RECLISTE RECLISTE RECLISTE NORMAL
FIDONET Q:\DIV3\echolist ECHOLIST ECHOLIST NORMAL
FIDONET Q:\point POINT POINT NORMAL
FIDONET Q:\DIV13\FASTECHO FASTECHO FASTECHO NORMAL
FIDONET Q:\DIV27\ITRACK ITRACK ITRACK NORMAL
FIDONET Q:\div13\allfix ALLFIX ALLFIX NORMAL
FIDONET Q:\div13\allfixut ALLFIXUT ALLFIXUT NORMAL
FIDONET Q:\div27\ttick TTICK TTICK NORMAL
FIDONET Q:\SDS\ftsc FTSC FTSC x3) NORMAL
WEBPORTALE 0
WEBPORTALE Q:\DIV13\wi2 WI2 WI2 NORMAL
FILEBASE 0
FILEBASE Q:\DIV13
FILEBASE Q:\filelist
...
x1) FILEBASE Q:\DIV13\MSDOS\ MSDOS MSDOS NORMAL
x1) FILEBASE Q:\DIV13\PRINTER\ PRINTER PRINTER NORMAL
x1) new areas inserted by 'routing' FILEBASE Q:\DIV13\ line to Maingroup FILEBASE
x2) names can be corrected in step 3 by PHPNUKE Admin Console
i.e. correct SERVIC to SERVICE, correct P24CVRT to PNTLCVRT etc.
x3) later move to Maingroup SDS, in PHPNUKE Admin Console
Maximus Access Levels:
-------------------------
Class Name Privilege Level
Hidden 65535
SysOp 100
AsstSysOp 90
Clerk 80
Extra 70
Favored 60
Privil 50
Worthy 40
Normal 30
Limited 20
Demoted 10
Transient 0
