Technical Discussions on the next versions

Archive of notes and cross input on Dev versions of Coranto (1.25.1 and Older)

Moderators: Dale Ray, SrNupsen, Bluetooth, Jackanape

Technical Discussions on the next versions

Postby cerberos76_ » Wed Jul 28, 2004 9:54 pm

For everyone information, me and parahead have started discussing some of the major new features of the next Coranto release.

The major elements of discussions are:
a) how to implement the split for flat databases
b) what to do to try and make life easier for anu future upgrade of the MySQL (CSQL) database (or even support for other databases)
c) keep compatibility.

To do a), my current progress is based on splitting the newsdat.txt in time intervals, e.g. from one single big file to a set of files for months, or weeks and maintain the same format to keep as much compatibility as possible (e.g. the last month will still be called newsdat.txt, the others might be placed in a associated folder).
I am implementing a function that does the rebuild in batches with a small time interval each (like a repetitive Automated Partial build). The batches are forked out for max speed (as I did with other addons) but can also be programmed in successive time intervals for minimum processor load.
The plan is to let the user select the database split by time/week/day intervals or by size (e.g. max 1 Mb), and try to have this function as integrated as possible in the build process so not to have to do too many operations manually.
The system is controlled by an external flat-file index that keeps track of where the parts/newsids are at all times.
Some of the problems are:
- make sure archives are correctly built
- the modify window should be transparent to user
- modifications to (past) news must go in the right files

This will certainly not break any addon, but might cause reduction of functionality in some if the main database is reduced (of course the database-split function will anyway have an option to recompact it all and therefore go back to original state and have full functionality).

b) The second idea is an abstraction layer that Parahead is proposing like a package (Coranto::Newsdata) for the access of newsdata which contains a number of methods for read/write/delete. Since all calls must pass through that module it is easy to implement it against one (or more) flat files, MySQL or other... It is just to replace the Coranto::Newsdata module with the one that the user prefer to work with. If one would like to take this another step the DBI/DBD scenario would be case...

Once this abstraction layer is written, that can be interchanged (flat/MySQL) more easily and the core becomes cleaner.
For example instead of explicitly looping in like this to find a newsid with flat file (and have a different function for the MySQL):
my $fh = CRopen("$CConfig{'htmlfile_path'}/newsdat.txt");
NCLOOP: while (<$fh>) {
chomp($_);
SplitDataFile($_);
if ($in{'ncid'} eq $newsid) {
last NCLOOP;
}
}
close($fh);


you would write:

use Coranto::Newsdata;
my $DBHandle = Coranto::Newsdata->getHandle("FLATFILE");
$DBHandle->getNewsItem($in{'ncid'});


of course the loop would still be needed, but would be implemented in the external Newsdata package rather than tighly coupled with the core build as it is now (which makes difficult to keep the support of MySQL at changes of the core).

Of course we need to avoid overheads and tru to go step by step

c) For compatibility, the problem would be that once the abstraction layer is introduced, the addons making use of the core might need to change their code to continue working.
This might be avoided with a very careful coding, so we will have to see how this progress.

For the moment the focus is on the area a), with some preliminary ideas on b) and some other optimisation of the current build based on selective build as done in Extractor and PartialBuild.

This is a small summary of some of the discussions between me and Parahead in the last few days. Anyone is welcome to add his opinion or to pass suggestions, ideas or nicely written code for any of the above (or other Coranto improvements)!
Coranto addons:
* GuestPost, one click guestposting
* MoveIt!, Advanced Backup and site transfer
* PartialRebuild, CPU-friendly enhancement of FullBuild
* URLColumn, URLs for multiple Categories/Profiles
* Tickit!, Javascript News Tickers/Faders for Coranto!
* AddonEnabler, enables or disables single, multiple or all addons in one go
* SubmitMultiple, post consecutive news in an automatic loop!
* RSSAdvanced, RelatedNews ....

Get them at: http://coranto.gweilo.org/addons/
cerberos76_
 
Posts: 1339
Joined: Sun Mar 10, 2002 1:21 am

Postby muttly » Thu Jul 29, 2004 4:51 am

2 minor requests/suggestions -

- option to put Create New Profile at the top of the page above the list

- rename classes in CSS to describe function rather than describe physical features .yellow and .whitebg doesn't have much meaning outside the original color scheme
muttly
 
Posts: 178
Joined: Mon Jun 10, 2002 8:19 pm

Postby cerberos76_ » Thu Jul 29, 2004 11:31 am

NitroX_infinity wrote:If you could backup per profile/style/template you could take that backup and use it for another site too :)

Why have it in the core?
Well, compatibility is one thing.

Nice metaphore: you don't sell a car and sell the petrol tank as an addon which has to be installed manually.
Okay, that metaphore isn't exactly correct but you know what I mean?


Hi Nitrox. Probably you misunderstand the concept of newsdat.txt (as I believe also from the earlier post). it does contain ALL your database (all profiles etc). Moveit selects the files that are necessary for you. With the files that are (automatically or manually according to your choises) put in the Moveit directory, you can close down your site and move where you want!!! To reconstruct the entire site, you just log-in the new site and do a full build. All the profile files will be reconstructed automatically!

In fact one of its options is exactly to Move the site away (and it will set all parameters for you)... that is why it is called moveit!

As for compatibility: putting a routine in the core does not ensure anything! Be sure Moveit is compatible with all versions as it just copies files across at regular intervals... If we change the name of the critical files I will also change Moveit to support it.

But please let's keep this out of this thread. if you have suggestions for Moveit, go to the Addon section.

Here please let's discuss critical technical enhancement for the core (also for smaller/layout enhancement, open a suggestions in the Core Requestsarea, not here)
Coranto addons:
* GuestPost, one click guestposting
* MoveIt!, Advanced Backup and site transfer
* PartialRebuild, CPU-friendly enhancement of FullBuild
* URLColumn, URLs for multiple Categories/Profiles
* Tickit!, Javascript News Tickers/Faders for Coranto!
* AddonEnabler, enables or disables single, multiple or all addons in one go
* SubmitMultiple, post consecutive news in an automatic loop!
* RSSAdvanced, RelatedNews ....

Get them at: http://coranto.gweilo.org/addons/
cerberos76_
 
Posts: 1339
Joined: Sun Mar 10, 2002 1:21 am

Postby Parahead » Fri Jul 30, 2004 6:03 am

Like cerberos76 is saying, we have had some mail correspondens during the last couple of days and we agree about the fact that improving the build routine can be done, but I am not so sure as cerberos76 about the part where splitting the newsdat.txt file will not brake any existing addons nor require zero recoding of them. I may be missunderstanding you cerberos and what you are trying to say to me, but I know for a fact that there are certain addons that just will not work in an expected way if newsdat.txt don´t include all newsitems.

To give you an example, the iSay addon would not be able to function on all news since it would not find the requested newsitem in newsdat.txt and would simply produce a "Specified ID does not exist" message when such a newsitem where requested... Other addons I know would be affected is ShadowMail and CopyNews.

But this is not a showstopper for me, I have made some investigation and the only addons that I have been able to find that would require changes to comply with a splitted newsdat.txt file is the ones I mentioned above. And I agree with you cerberos that it is better to do things step-by-step, so I will lay low (for a while) with the module approach similar to DBI/DBD, but my personal belief is that this is the way to go to truly get independence between the logic and the database layers and if the addons must be changed anyway (which you say they don´t) I think the module approach would require *less* changing than taking into considiration having code in each addon that must keep track of several newsdat.txt files in order to find the requested newsitem...

Hmmm... cerberos, since we have couple of other suggestions presented in this thread and also the request I made here about the improved fault tolerance for nsettings.cgi, which you have approved to, maybe we should divide the work amongst us so that I look into if it is possible to do these things instead of we both look at improving the build process, since you have more experience in that area anyway (Extractor)... But of course I will keep tossing around (stupid) ideas regarding the build and investigate/test that part if you like...
Yes, I am still around...
www.parahead.com/coranto/
User avatar
Parahead
 
Posts: 4837
Joined: Fri Jan 12, 2007 8:54 pm
Location: Stockholm - Sweden

Postby faithless » Tue Aug 03, 2004 11:34 pm

Worry not about iSay's compatibilty. It could be easily recoded to work with the new database system if any.

Currently, it reads crcfg.dat and reads newsdat.txt directly without going through any other parts of Coranto. But if the data layer was to be queried via a module package, I'm sure iSay can be easily recoded to work with that too
faithless
 
Posts: 1335
Joined: Wed Oct 16, 2002 4:17 pm
Location: Infront of the Computer

Postby Parahead » Wed Aug 04, 2004 7:15 am

faithless wrote:Worry not about iSay's compatibilty. It could be easily recoded to work with the new database system if any.

Currently, it reads crcfg.dat and reads newsdat.txt directly without going through any other parts of Coranto. But if the data layer was to be queried via a module package, I'm sure iSay can be easily recoded to work with that too
This is kind of my point, that there are certain addons, like iSay and ShadowMail, that needs recoding *regardless* of if we only split up newsdat.txt or make the database accessible through a package.

cerberos76 wrote:of course the database-split function will anyway have an option to recompact it all and therefore go back to original state and have full functionality
Unless this option is there... :-D
Yes, I am still around...
www.parahead.com/coranto/
User avatar
Parahead
 
Posts: 4837
Joined: Fri Jan 12, 2007 8:54 pm
Location: Stockholm - Sweden

Postby SwItCh » Sat Oct 16, 2004 12:20 pm

Question: what about upgradability... If I am currently working on my site, and installing addons etc, posting news/reviews etc. and in 6 months time you come out with a new version with new features and possible coding differenes, will it be possible to upgrade... saying that, is it EVER possible that you'll be able to upgrade once you've started using addons?

Also, this is VERY minor, but on installtion of Coranto and the automatic display of "Powered by Coranto" which is displayed ast the bottom of the script. It leads to a dead page, as amphibianweb must be gone or something. It just didn't seem proffesional I guess, It would be best having a new domain name that linked to here.

I know I can't contribute to coding, but I love Coranto, and everything about it, so I wouldn't mind buying a new domain name and having it redirected to this site.

SOrry for not haivng a better input... *walks off in shameless slump*
User avatar
SwItCh
 
Posts: 44
Joined: Thu Oct 14, 2004 12:44 pm

Postby faithless » Sat Oct 16, 2004 12:28 pm

There's already quite a few "Coranto" domain names pointed to here already. :D
faithless
 
Posts: 1335
Joined: Wed Oct 16, 2002 4:17 pm
Location: Infront of the Computer

Postby Parahead » Sat Oct 16, 2004 12:57 pm

SwItCh wrote:Question: what about upgradability... If I am currently working on my site, and installing addons etc, posting news/reviews etc. and in 6 months time you come out with a new version with new features and possible coding differenes, will it be possible to upgrade... saying that, is it EVER possible that you'll be able to upgrade once you've started using addons?
Yes, many addons work independently of which Coranto version you have installed... Having that said it is of course possible that new features in Coranto can break old addons, but maintaining backward compatibility is always desired when adding functionality. Currently the only major addon that hasn´t been upgraded to keep up with the current release is the CSQL addon.

SwItCh wrote:Also, this is VERY minor, but on installtion of Coranto and the automatic display of "Powered by Coranto" which is displayed ast the bottom of the script. It leads to a dead page, as amphibianweb must be gone or something. It just didn't seem proffesional I guess, It would be best having a new domain name that linked to here.
It is possible to choose this domain as the link as well you know... The amphibianweb link is there because it was Elvii's domain, and since Coranto technically isn´t Open Source it has been decided that it should remain because his licence agreement stated that an businessinstallation of Coranto should have a link to that site... Yes, maybe dumb to link to a dead domain, but still, that is the reason....

SwItCh wrote:I know I can't contribute to coding, but I love Coranto, and everything about it, so I wouldn't mind buying a new domain name and having it redirected to this site.
The CTUS is now *the* Coranto place, buying a new domain linking here don´t add that much of a value... The domain issue has been discussed quite many times before.
Yes, I am still around...
www.parahead.com/coranto/
User avatar
Parahead
 
Posts: 4837
Joined: Fri Jan 12, 2007 8:54 pm
Location: Stockholm - Sweden

Postby SwItCh » Sat Oct 16, 2004 1:01 pm

Oh okay, was just trying to see if I could possibly do anything at all lol

and I wa just looking at Coranto, I didn't realise you could direct it here also... how dumb of me.

Will the new Coranto actually have the current addons which you guys think is efficient intergrated into the new version? or will people still get the news system and be required to download the addons?
User avatar
SwItCh
 
Posts: 44
Joined: Thu Oct 14, 2004 12:44 pm

Postby Parahead » Sat Oct 16, 2004 1:09 pm

SwItCh wrote:Will the new Coranto actually have the current addons which you guys think is efficient intergrated into the new version? or will people still get the news system and be required to download the addons?
This is kind of the spirit of Coranto. Things that can be done via addons should be done via addons. Keep the development distributed and more people can take part of adding features. It also helps each installation being as slim as possible, only adding the functionality needed...
Yes, I am still around...
www.parahead.com/coranto/
User avatar
Parahead
 
Posts: 4837
Joined: Fri Jan 12, 2007 8:54 pm
Location: Stockholm - Sweden

Postby shanemckiness » Thu May 12, 2005 2:44 am

How about integrating maginot into the core? :)
shanemckiness
 
Posts: 234
Joined: Fri May 21, 2004 5:14 pm

Postby Parahead » Fri May 13, 2005 1:47 pm

shanemckiness wrote:How about integrating maginot into the core? :)
You didn´t read my post just above, now did you? ;-)

Parahead wrote:Things that can be done via addons should be done via addons. Keep the development distributed and more people can take part of adding features. It also helps each installation being as slim as possible, only adding the functionality needed...
Yes, I am still around...
www.parahead.com/coranto/
User avatar
Parahead
 
Posts: 4837
Joined: Fri Jan 12, 2007 8:54 pm
Location: Stockholm - Sweden


Return to Coranto Development Archives -- 1.25.1 and Older

Who is online

Users browsing this forum: No registered users and 0 guests

cron