GE Database Toolkit

CrownClown

Well-known member
Mar 29, 2020
88
34
Hello everyone. I`m just re-posting my tools from GEEU forums (as it`s gonna shutdown soon, think).
In short- that`s a couple of short scripts that `ll generate a "jupath-like" (currently maintained as ge.sytes.net) database in one single .html file. This .html will be BIG (~7 mb with all options- chars, items (actually 50+% of total file volume), etc.
It basically copies what you can see in ingame information center, and uses game translation data to work, so ofc what`s bugged in game (like missing AR\DR stats in buff description, for example) -will STILL BE bugged in .html
You can read more about it (old version) here: http://forum.eurogamez.eu/showthread.php?7823
Usage is basically the same. Place in /ge/, run prepare.bat, run parse(English).bat. Or run .php files directly.
Current changelist and roadmap for it:

Link: (Current images included, will re-upload with "clean" image folder in a couple of days).

Changes:
Compared to GEEU forums release:
1) Parse more information -not only characters, but currently also:
-Achievments (sorted by category)
-Weapons&Armors&Accessoires&Medals (Each on separate tab, sorted by sub-classes, like weapon => Swords\Sabres\..., Armors=> Leather, Metal, Robe, Coat....
-Craftable equipment is parsed with recipe. Enchantable equipment is parsed with option list and chances.
2) Some mid-class errors is fixed. Way more clean script output (less errors, mainly due to fixed errors with missing files. Not 100% fixed, planned 100% fix with next update.
3) More durable to incorrect xml markup , thx Hand of Vecna & Prefix from Starstorm-ge for pointing: https://starstorm-ge.blogspot.com/2020/05/ge-html-offline-database.html
4) Re-structured main script, to make it a bit easier to read. Commented a bit.

Roadmap
Disclaimer:
No ETA,i`m doing it for fun when i`m interested in it. As i`m not playing GE for some time -my interest is in the scripts itself, not in the game. There can be very different time gaps between any changes. You can motivate me a bit by showing your interest in this toolkit, but do not expect anything.

ChangeAmount of work\timeDetailsPossible options:
Change whole script design to be more modular & controllable with CLI or small web GUI1-2 full days.While it`s easy to simply make script modular (aka make it possible to select what kind of content you do want to parse, and what you don`t), and it can be done within a couple of hours right now- i want this kit to be interactive in the end, so such control can be applied with command line, without messing with files. Doable, but can wait until next release, as currently amount of options if bearable, it`ll only rise with next updates.SEMI-DONE.
Fancy GUI, maybe with some HTML framework???As it`s a kind of thing that involves personal tastes more then technical questions- currently it`s as "clean" as possible, 90% is pure clean HTML elements without any decorative styles. But as i want it to be usable "out of the box"- i`ll have to decorate basic template at least a bit.
Even more options to parsefrom 1 hr\option to ???Currently planned stuff:
1) Expedition page (one big table, same as ach. page
2) "Mission" tab from information center. Basic mission info.
3) Questlines info - basically "Quest" tab from information center.
4) Some static information that can be useful, like accesoire enhancement chances
Currently done in separate scripts. Need to be integrated into the toolkit, which is easy. More complex options can require more time. Open for suggestions.
Page structure changes to reduce file size1-2 full daysCurrently some stuff is done in a bit better way (skills& stances, where it was critical NOT to store everything as extra-long string on character page, as many chars have same stances, and many stances have same skills.
But changes done last week (basically anything aside character pages) is not so modular. It`s possible to reduce total file size by ~30-40% by simply replacing repeatative content with "shortcuts". But atm it`s not considered 1st priority .
GUI commfort changes1-2 full daysAnchors where it`s needed (like anchors on stance names on char pages, which`ll scroll the page to the stance), maybe tooltips instead of plain texts for some elements.
Next Major Update: Implement custom static fields from internet source (like Pastebin), to store pre-defined info. Like links to Fatsnake yt for stances.1-2 weeks (depends on amount of content to be manually processed)
Minor: move illust files automatically1 hrCurrently you`ll have to move /ui/illust/ to /images/ manually.DONE
 
Last edited:

CrownClown

Well-known member
Mar 29, 2020
88
34
Minor update:
1)Fixed a couple of parsing errors.
2).HTML is now semi-modular. Script will ask you for categories that you want\do not want to parse. You can answer "N" or "n" to skip this category (it`ll skip everything- side tabs, header link, content), or anything else (including simply spamming enter) to include category.
3) No more ./illust/ folder moving needed. Script will put item icons into /image/Items/ folder. That includes recipe items icons and parsable item categories icons (weapons\armors\medals\accs).
4) A bit more proper layout for item tables, to be more compact in case item have both chipping options and crafting recipe.
5) Now replaces {br} with HTML <br> tag. Basically new line.
6) Cleaned /image/ folder, which reduced archive volume to 24MB from 100+.
DL link:
Achievment for 170x 36 AR weapons craft is "semi-fixed" (reported to @Dianmir , "fixed" on server-side).
 

Calliostoma

Member
Aug 3, 2020
10
0
Hi Thank you for your work.
However I have a question.
I used your toolkit in the official server. However, I encountered a problem.
Indeed in their database, the job description is in the dictionary_local.xml and no more in the skill.xml.
I'm not that good in programmation but could you give me a tips to fix this ? Right now, the database I obtained gives number which I need to refer in dictionary_local files. An example of what I got is in the enclosed files.
Thanks
 

Attachments

CrownClown

Well-known member
Mar 29, 2020
88
34
Which server are you on? Steam` GE need additional preparation for .ipf files, as they`re split into .*number*ipf. Script for that can be found in Ashardalon blog, tho i can`t really remember if it worked for me 100%. I remember that i did some additional preps.
In short- when you extract .ipf, you also need to extract all the .*number*ipf, and recursively replace all the files from .ipf to their versions from .numberipf.
Example:
extract ies000000.ipf . Rename folder /ies000000/ to /ies/
extract ies000001.ipf . Replace all files from /ies/ to their versions from /ies000001/
extract ies000002.ipf.....
extract ies000004.ipf. Replace all files from /ies/ to their versions from /ies000004/

Same need to be done to dictionary, and ui.
Ashardalon wrote a script for that, also including regional version (subfolders like /DEU/).
Only after that proceed to ix3.exe use.
 

Calliostoma

Member
Aug 3, 2020
10
0
Hi ! Thanks for your reply !
I'm on Steam GE and indeed, I managed to extract the ies, ui and dictionary correctly.
I'm actually stuck with the Parse(English).bat where I need to change a bit the English.php you created because the description of the skills are, contrary to files in Andromeda, in the dictionary_local.xml files.
More specifically, by running the actual script, for example for the line
echo $Skill['SpecDesc10'][$CurrSkill]
- if used with the Andromeda files, it gives the proper description.
- But for the Steam version, it refers to a number with the format <$>XXXXX.
This is because the datatable_stance is written as SkillName2=<$>XXXXX; where you need to refer this number to the dictionary_local.xml to have the skill description.
I was thinking about creating a new datatable_stance.xml by replacing the numbers with the corresponding text. But I have no knowledge on php so I'm getting hard time.
 

CrownClown

Well-known member
Mar 29, 2020
88
34
Thing is ix3.exe do the same stuff, replacing <$>XXXXXX</> with the strings from dictionary.
But it`s kinda whinny about directory paths - it`ll replace both if there`re both /ies/*file*.ies, dictionary.ipf and /dictionary/dictionary_local.xml files.
it SHOULD do the thing if your folder structure is right, tho.
Gonna look at it a bit later.

If you want to do so manually with PHP- you can go like this:
1) Parse whole dictionary into PHP array - will consume a lot of RAM, but allow faster addressing later. You can probably use Saxta function i wrote for that, if you apply right arguments.
2) use preg_replace for <$>*number*</> to $dictionary[*number*] for each SkillName2.
 

CrownClown

Well-known member
Mar 29, 2020
88
34
Just checked. Everything worked fine for me (win10, x64, actual Steam SEA client updated today).
What i did:
1) Downloaded toolkit from 2nd post of this thread.
2) Extracted to /ge/
3) Copied Starstorm script to /ge/parse.cmd
4) Renamed GE folder without spaces (GranadoEspadaForSEA)
5) Lauched Powershell with admin rights.
6) cd D:\Games\Steam\steamapps\common\GranadoEspadaForSEA\ge
7)./parse.cmd

result attached. Seems fine.
 

Attachments

CrownClown

Well-known member
Mar 29, 2020
88
34
Yes, but you'll have to manually manage dictionary_local.xml
In Andromida there're separate files for each language.
You'll have to modify cmd from Ashardalon or prepare.php to grab dictionary-local_*lang*, and replace dictionary_local.xml with it BEFORE you do the ix3.exe part.
Other then that- there're no restrictions. Maybe you will want to add some other replacements to SaXTA.php if your language have any important unicode symbols thats replaced within xml.
Btw, im gonna rewrite it in a couple of days. Found a bug that skips adding some stances to js strings, and apostrophe replacement needed. Plus ill probably upgrade it a bit. Just for fun, no deep reasons.
 

WatchMan

Member
Mar 31, 2020
24
1
Yes, but you'll have to manually manage dictionary_local.xml
In Andromida there're separate files for each language.
You'll have to modify cmd from Ashardalon or prepare.php to grab dictionary-local_*lang*, and replace dictionary_local.xml with it BEFORE you do the ix3.exe part.
Other then that- there're no restrictions. Maybe you will want to add some other replacements to SaXTA.php if your language have any important unicode symbols thats replaced within xml.
Btw, im gonna rewrite it in a couple of days. Found a bug that skips adding some stances to js strings, and apostrophe replacement needed. Plus ill probably upgrade it a bit. Just for fun, no deep reasons.
thanks bro w8 for you new file
 

CrownClown

Well-known member
Mar 29, 2020
88
34
Gotta publish the next version of kit till this Friday.
Currently done:
-Cleaner CLI output, less errors due to missing files.
-Fixed a bug with stances nested at 6th slot of character (like Unlimited Shot on Musks) producing JS errors and incorrect page behaviour.
- Prepare.bat will be fully replaced with Ashardalon script FOR STEAM CLIENT.
- Original Prepare.bat will be split into several language files, to swap dictionaries for Andromida lang packs.
- A bit more advanced GUI, style from Ge.Jupath (or modern Sytes DB)
- ID sorting for characters (basically "sort by release order" in most cases. That`s what is there by default until you sort the list any other way.)
fixed some development tails, like demi-russian texts.
- Separate file will be written to allow embedding stance videos from FatSnake YT channel. It will be manually produced by me once, with a bit of manual and simple method for whoever is willing to use the kit to add it later.


======
Separate Golang version was intended (.exe, no need to carry PHP distro along), but after some consideration i decided to not waste time developing it. While Golang is waaaaaay better in terms of speed and resource management compared to PHP, it won`t change the bottleneck (which is IPF => ZIP=>IES=>XML conversion) in any significant way (maybe only by organizing the multiple threads to do so). And for majority of 2010+ PCs wasting a couple gigabytes of RAM for like 30 sec is not really huge load. In total- reducing whole scenario from like 10 min to 9:45 don`t really feel worth rewriting it from a scratch.

Also, while everyone can open & check PHP code, doing so with EXE will make the kit look more suspicious, as .exe is way less safe.
So kit will stay .php+distro for at least near future.
 

CrownClown

Well-known member
Mar 29, 2020
88
34
No. I never developed anything Android\Ios related. This kit, old non-oficial GEEU blog, a couple of tools to maintain translation for GEEU (i used to translate ENG=>RUS for them), and some non-GE projects.
 

CrownClown

Well-known member
Mar 29, 2020
88
34
I`ll take a couple of days more to polish stuff a bit, but here`s a peek of current progress:
1) Script is restructured. Instead of one BIG HMTL file, currently it`ll be:
DBChars.HTML- whole info about chars, skills and stances.
DBAch.HTML- whole info about achievements
...
and so on.
JS scripts is currently nested in separate files in /scripts/ folder. Same with CSS styling.
/Templates/ contain all the parts of HTML pages.
Script do not ask if you want to parse\skip parts anymore.

What it was done for: Instead of loading 7MB file into RAM each time you run DB, currently the only big file is DBWeapons.HTML(~3.8MB), others being ~300-500kb each. Plus i can now control which JS library is needed for each file (for example- there`re no need to keep table sorting JS on any page but characters, as there`re simply no sorted tables. In result - it now takes way less time for file to load.

Fixed some major bugs (mentioned above bug with 6th stance slot, plus &Apos; was resulting some bugs. Script is a bit more trouble-proof.

"Custom content" script is introduced. Currently it`s only used for stances, allowing you to add YT video to any stance. Video is shown in "spoiler" kind of button, so it`s not interrupting the page until you show the video.
Syntax is as simple as possible. Just open CustomContent.php with notepad, you`ll see the following

Add string on the same template and run parse.cmd. DB will be updated and YT player will be embedded.



Currently i begin to work on page styling, it`ll keep me a couple of days, i think, but probably i`ll pause until NY. If i decide to pause- i`ll upload current version tomorrow.
 

Attachments