RSS feed creation - Discussion and Solution

[Dale Ray]

There has been some discussion of RSS feeds and creating them in various threads on this forum. A forum search for RSS finds 125 matches. The subject has caught my eye and I have started researching into RSS - what it is used for and how it can be implemented in Coranto.

Here are two links to the RSS 2.0 specification:

http://blogs.law.harvard.edu/tech/rss
http://feedvalidator.org/docs/rss2.html

Here is a discussion of the various earlier versions of RSS:

There is also another format for syndicating news called ATOM. Read a comparison of the two written by a supporter of ATOM.

More about ATOM:

http://www.atomenabled.org/
http://www.intertwingly.net/wiki/pie/FrontPage

So - in one sentence- RSS and ATOM feeds provide a summary of content on a site in a very specific format that can be retrieved by an aggregator.

All of this is probably of little interest to the user who just wants to create a feed to syndicate a few headlines and draw traffic to his/her site. The user just wants to be able to easily create a feed and syndicate it.

There are two addons for Coranto for creating RSS feeds:

RSS File Advanced beta 5d
RSS File Beta Build 4

both of these date from 2002.

There is also RSS Parser 8 beta for collecting and displaying RSS feeds on your Coranto site.

I am starting this discussion to get feedback from users about what they would like in an updated RSS feed creation system. I did not say addon in the previous sentence because I am not convinced (yet) that an addon is needed. I feel that you can create an RSS feed with the use of styles and templates.

I do not want to build upon either of the previous addons because of licensing issues. One is by elvii, who is long gone from these forums, and the other by cerberos76, who has been known to change his mind about the licensing model for his released work. I don't want to deal with either of these issues.

I welcome your comments and input on this project. There is no road-map for getting something that works published, if you are currently working in your secret lab on this same thing - go ahead - you'll probably be finished before I am.

If you are using one of the previous addons for creating an RSS feed in Coranto please check your feed with a feed validator and let me know what errors you get.

[Parahead]

I am starting this discussion to get feedback from users about what they would like in an updated RSS feed creation system. I did not say addon in the previous sentence because I am not convinced (yet) that an addon is needed. I feel that you can create an RSS feed with the use of styles and templates.

Since I have been slightly involved in the other threads you mention I just want to give my view on the subject.

For example this thread contains a suggestion from shanemckiness about a simple RSS feed and if only using the required channel elements (eg. <title>, <link> and <description>) the approach of using templates for the top/bottom info (including the elements above) and a style for the items should work just fine since the required elements can more or less be hardwired into the template.

However, if a user wants to start using some of the optional elements for the channel, it gets a little more complicated to pull it off using a template/style approach. For example <pubDate> and <lastBuildDate> would need to be dynamically updated at build time. Yes, quite possible to create a combination of profile/template/style and a little bit of PerlCode that makes this happen, so it *may* be so that all that is needed is to publish a documentation for how to do that including examples of profiles/templates/styles.

However, creating an addon for it seems more userfriendly in the long run, having inputfields for the optional channel elements in the RSS Profile and so on. Also, using the addon in combination with Maginot (or viewnews.cgi) lets you use <guid isPermaLink="true"> for the item element.

It doesn't seem to be a very complicated addon to create from a technical perspective either, more or less a standard profile with a bunch of inputfields at the profile settings page for the channel elements and maybe a "Default RSS Style" for the item element, to keep things simple yet flexible?

[shanemckiness]

I think what we found so far is that RSS advanced can produce ok results, but can only use 1 Field at a time, which makes is impossible to use. Also, it is coded in .91 format I believe, which is outdated. Using profiles is a bit tricky. For one, the page should be created without any php includes or such, because the file will be accessed a ton by other sites. Parahead reccomends a maginot sliced profile rather than a regular static profile because static profiles cannot include more than one newitem at a time. I have yet to doodle around to see how well a sliced profile works. I found that a regular coranto profile would not be a good choice because you need to create a file with header information and footer infromation that cannot repeat, with ITEMS in between. I believe an addon would be best due the complications of setting this up, even though it's really a very simple thing..

[Parahead]

Parahead reccomends a maginot sliced profile rather than a regular static profile because static profiles cannot include more than one newitem at a time.

You make it sound like this is a shortcomming of the Static Profile, when it in fact the hole purpose of that particular profile type is to place one newsitem in one file. Then instead of using viewnews.cgi to present an individual newsitem the static files is used to ease the burden on the webserver when there is many visitors. This is why the addon was named Maginot Line to begin with (I think anyway). However, this is offtopic in this thread so please lets drop it...

I found that a regular coranto profile would not be a good choice because you need to create a file with header information and footer infromation that cannot repeat, with ITEMS in between.

And I can't understand how you have come to this conclusion, since a template/style approach together with a standard profile would solve this. I get the feeling that you haven't got your head arround everything in regards to templates/styles and the use of certain profiles. And I truly don't mean anything bad with it, it just sounds like you make things so complicated?

I believe an addon would be best due the complications of setting this up, even though it's really a very simple thing..

I also believe an addon would be best for the very same reason, you are kind of living proof of that thesis...

[shanemckiness]

I have yet to see someone who has done it without an addon or without server or client side includes.

[Dale Ray]

Using Coranto 1.31.2 it is relatively easy to produce a valid RSS document with the use of a template and style.

Here is the template:

Code: Select all

<?xml version="1.0" ?>
<?xml-stylesheet type="text/css" href="rss.css" ?>
<rss version="2.0">
<channel>
<title>Test RSS Feed</title>
<link>http://www.corantodemo.org</link>
<description>Test of an RSS feed generated by Coranto.</description>
<language>en-us</language>
<Field: Content>
</channel>
</rss>


Here is the style:

Code: Select all

<item>
  <title><Field: Subject></title>
  <link>http://t3/coranto/viewnews.cgi?id=<Field: newsid></link>
  <description><WordSnip Field: Text></description>
  <pubDate><Field: Date></pubDate>
</item>


The <WordSnip> tag requires the Snipper addon by Parahead.

You must set the HTML User File Name for the profile to end with the extension .xml.

Setting the General Date Format to:

Code: Select all

<Field: Abbrev_Weekday>, <Field: TwoDigitDay> <Field: Abbrev_Month_Name> <Field: Year> <Field: TwoDigitHour>:<Field: Minute>:<Field: Second> CST


allows you to simply use <Field: Date> for the pubDate. Some of these date fields are not available in Coranto 1.24.

This is just about the simplest RSS feed you can make.

This isn't as easy in 1.24 because you can not set an extension for each profile and because of the missing date fields to create a date compliant with the standard.

I created a test document with this setup and the file validated at http://feedvalidator.org/

The links to the individual items don't work - the file was created on a local test setup and the links point to that.

If you use Coranto's native snip feature and not the Snipper addon the file can be created with NO addons. It would be simple to add

Code: Select all

<category><Field: Category></category

to the style if you enabled the Coranto category addon.

Other optional elements can be added the same way. You are essentially mapping the RSS elements to one or more Coranto fields.

The profile/template/style sheet approach is not the best since all of the elements defined in the template can only be changed by editing the template. I think that a new profile type for RSS would work better. It would need to:

Allow the user to set values for title, description, link and other elements that are hard-coded in the template.

Allow the user to map the RSS item elements to one or more Coranto fields. Allowing field codes to be entered on the settings page would accomplish this.

Either circumvent the normal Coranto build process or upon saving the settings on the profile page create the style for the user.

Enough thinking out loud for now, comments welcome.

FYI: Here is the cascading style sheet I used:

Code: Select all

* {
display: block;
}

:root {
margin: 50px;
}

channel > title {
font-size: x-large;
text-align: center;
color: red;
}

item {
margin: 25px 0 20px 0;
}

item > title {
font-size: medium;
margin-bottom: 20px;
}

item > link {
font-size: small;
margin-top: 6px;
margin-bottom: 6px;
}

item > description {
font-size: small;
}

item > pubDate {
font-size: small;
}

language {
   display: none;
}


[Parahead]

Enough thinking out loud for now, comments welcome.

I think you have proven the concept of using a template/style to create a simple RSS Feed to shane now at least.

It also sounds like we have pretty much the same view on how to solve this for little more complex feeds, I say go ahead and do some coding...

[Dale Ray]

Enough thinking out loud for now, comments welcome.

I think you have proven the concept of using a template/style to create a simple RSS Feed to shane now at least.

It also sounds like we have pretty much the same view on how to solve this for little more complex feeds, I say go ahead and do some coding...

It is a very simple feed AND if you have multiple posters you would have to make sure that you validate the output (take out HTML, escape certain characters) before the file is created. You can use Anne Rice or Search And Replace to escape special characters and use the field settings to strip HTML.

I am going to tackle this as an addon, but if anyone needs it tomorrow They will be very disappointed.

[Wpeckham]

It works beautifully!

To create my RSS feed, I used this news style

<item>
<title><Field: Subject></title>
<link>http://www.peckham.dynu.com/cgi-bin/coranto/viewnews.cgi?id=<Field: newsid></link>
<description><TextField: Text></description>
<category><Field: Category></category>
<pubDate><Field: Date></pubDate>
<guid IsPermaLink="false">http://www.peckham.dynu.com/cgi-bin/coranto/viewnews.cgi?id=<Field: newsid></guid>
</item>

RSS Style Notes:
1. the guid element is required by the standard, and should be unique. Using a link to your site with the unique news item field makes it satisfy the standard, and also gets around a problem with some feed readers that use this element instead of the link element for the click-through or open action. The IdPermaLink="false" should prevent this, but not every reader gets it right.
2. If you want to reduce the size of your feed file and reduce its overhead, or force readers to open your news item, you can "snip" instead of including the entire text. The change would be at the "TextField: Text". I prefer to include the entire text.

I set up an rss profile using this style and the template rss.tmpl:

<?xml version="1.0" ?>
<rss version="2.0">
<channel>
<title>W.Peckham RSS new feed</title>
<link>http://www.peckham.dynu.com</link>
<description>News from William Peckham</description>
<language>en-us</language>
<Field: Content>
</channel>
</rss>

to create the feed. Since this was created in my data directory in the coranto tree, I had to either make an ssi file (a cgi link using php or perl would have worked as well) to dump the data out to the client on demand, or make a hard link. I chose to make the link, which is the simple thing to do under unix.

This seems to be VERY slick, very FAST (the addon was slow at times) and works with NO EXTRA ADDONS! Since it uses nothing more than the template and style that is not either installed with the Coranto beta or required for running it, there is no bloat.

The single downside: using "TextField:" any formatting or coding in your news is stripped. If they follow the link and view it on your site it will be formatted, but your rss feed will only contain the stripped text.

[leeerickson2050]

You must set the HTML User File Name for the profile to end with the extension .xml.

Ahhhh Man!!! This makes me want to put the BETA on my main site.....

[Psykosys]

You must set the HTML User File Name for the profile to end with the extension .xml.

Ahhhh Man!!! This makes me want to put the BETA on my main site.....

You could take a look at my tiny, UI-free addon here, which I've been using for my feeds. It changes the extension of whatever files you specify from one extension to another after ever build.
Just copy+paste to notepad, and save as cra_something.pl in your addon directory

Obviously, a very roundabout way of doing things, but it works.

[DRaines]

This thread has helped me set up a RSS 2.0 compliant feed from my site (almost). However, the publication dates are not fully RFC-822 date-time formatted.

I am using Coranto 1.24 and have modified the styles and templates in this thread to create my own stream. However, I cannot get two features of the time to be correct:

1) <Field: Weekday> gives me a full weekday name. The spec. calls for a three character weekday format.
2) <Field: Month_Name> gives me a full month name. The spec. calls for a three character month format.

I tried various forms of "snipping", but those seem to only work with text fields. Are there any other time/date fields that meet the specification? Is there another way to get the time that will meet the specification?

The use of Coranto's standard news publishing for an RSS feed really appeals to me. I would like to get this one last thing working.

Thanks in advance for any help you can give,
Darrel

[Dale Ray]

This thread has helped me set up a RSS 2.0 compliant feed from my site (almost). However, the publication dates are not fully RFC-822 date-time formatted.

I am using Coranto 1.24 and have modified the styles and templates in this thread to create my own stream. However, I cannot get two features of the time to be correct:

1) <Field: Weekday> gives me a full weekday name. The spec. calls for a three character weekday format.
2) <Field: Month_Name> gives me a full month name. The spec. calls for a three character month format.

Darrel

Try this:

Code: Select all

<PerlCode>
$pubDate = substr ($Weekday,0,3).", " . $TwoDigitDay . " " . substr ($Month_Name,0,3) ." " . $Year ." " . $TwoDigitHour . ":" . $Minute . ":" . $Second . " CST";
</PerlCode>
<Field: pubDate>
</p>

Note: the perl code should all be on one line - the forum may put a linebreak in.

Replace the timezone at the end with the proper one for your server. Use the field pubDate in your atyle where you were putting the date fields.

Let us know how it works.

Are you using the Post-Build File Renamer addon?

[DRaines]

Try this:

Oh, sure, show me how it is done by writing Perl!?

He, he. I guess I should have seen that coming. Well, since you are going to force me to do Perl, I guess I will go all the way.

In order to use the <PerlCode> tag, you have to enable it in the cruser.pl file. While I was there, I noticed that the file will allow you to include your own subroutines. I decided that it would be easier to do it this way in case I need to use the code again in another style definition. So I did not enable <PerlCode>, but added the following:

Code: Select all

sub getPubDate {
  return substr($Weekday,0,3).", ".$TwoDigitDay." ".$Abbrev_Month_Name." ".$TwoDigitYear." ".$TwoDigitHour.":".$Minute." ".$Time_Zone;
}

There are a couple of variances from your suggestion. The seconds are not included as they are optional. The time zone is actually available in a string, $Time_Zone. You will also note that I used another, undocumented string, $Abbrev_Month_Name. More on this later.

Now I add a <Sub: getPubDate> to my style definition, and the date is correctly formatted.

Let us know how it works.

This works very nicely and is valid for a feed date!

Are you using the Post-Build File Renamer addon?

Yes, and it works great.

Now, my RSS style and template work well except for one thing. I cannot put a pubDate or lastBuildDate in the template file. You see, the time fields are only available when I process news entries. I cannot put those fields or the subroutine call in the template file.

What to do?

I decided to dig deeper by browsing through the rest of the code. I noted that crlib.pl has the time functions set up in a subroutine. It was this code reading that made clear that the function $Abbrev_Month_Name was available. Another variable could be added (the code is almost all there already) to produce $Abbrev_Weekday. However, I don't want to change the baseline code.

I went back to cruser.pl and did a little playing around. It turns out that you can create global variables as well as subroutines in this file! Aha. The answer is in sight. I added the following code in the subroutine section:

Code: Select all

$pubDate = makePubDate();

sub makePubDate {
  @Abbrev_Week_Days = ('Sun','Mon','Tue','Wed','Thu','Fri','Sat');
  @Abbrev_Months = ('Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec');
  @Week_Days = ('Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday');
  @Months = ('January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December');
   ($Second,$Minute,$Hour,$Month_Day, $Month,$Year,$Week_Day,$IsDST) = (gmtime(time()))[0,1,2,3,4,5,6,8];
   $Month_Number = $ActualMonth = $Month + 1;
   $Year += 1900;
   $TwoDigitYear = substr($Year,2,2);
   $TwoDigitDay = sprintf('%.2d', $Month_Day);
   $TwoDigitMonth = sprintf('%.2d', $Month_Number);
   $Minute = sprintf('%.2d', $Minute);
   $Second = sprintf('%.2d', $Second);
   $TwoDigitHour = sprintf('%.2d', $Hour);
   $Month_Name = $Months[$Month];
   $Abbrev_Month_Name = $Abbrev_Months[$Month];
   $Day = $Month_Day;
   $Weekday = $Week_Days[$Week_Day];
   $Abbrev_Weekday = $Abbrev_Week_Days[$Week_Day];
   return $Abbrev_Weekday.", ".$TwoDigitDay." ".$Abbrev_Month_Name." ".$TwoDigitYear." ".$TwoDigitHour.":".$Minute." GMT";
}

Most of this code is stolen from the crlib.pl subroutines, with a little of my own thrown in. Now the global variable $pubDate contains the proper RFC-822 string for the last time that Coranto was run. I add a <Field: pubDate> tag to both the pubDate and lastBuildDate entries in the feed, and check the results.

Ta Da! I have a valid feed without sacrificing the channel publication date.

Notes:
1) You may notice that I used GMT for the time zone in the makePubDate subroutine. This was easier than figuring out the local time zone.
2) It may be nice to provide the $Abbrev_Weekday string in a later release of Coranto.

Okay, I may have made some serious mistakes along the way, but everything seems to be working for now.

Comments?

[faithless]

... This isn't as easy in 1.24 because you can not set an extension for each profile and because of the missing date fields to create a date compliant with the standard....

Using a standard profile type in 1.24, you can change the file extension type generated by changing the value of the filename generated from news.txt to news.xml or any filename.ext you so desire.