#1
  1. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69

    WikiWiki data store..


    Hi,

    Intrested in WikiWiki, One thing in particular, where does it store the pages it creates, I'm thinking either it has its own database, dmbfile or just stores them as plain text. Could somone comferm this?

    Thanks alot,
    Mark.
  2. #2
  3. Just another guy
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Jun 2003
    Location
    Wisconsin
    Posts
    2,953
    Rep Power
    262
    From reading around a bit on WikiWiki, it looks like they have their own database. (refers to file backups and restoration from the database)
  4. #3
  5. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Thanks for the reply. I downloaded the file a while ago and couldnt see any C code.. Which would mean that the database is implimented in Python.

    Although I've since desided to use a flat file design rather than using any databases as it makes is alot easier to download and or import data from into other websites using the software. For backups I'm thinking about implimenting a module to store the last version of the page in a backup directory for restoral.

    Any thoughts would be welcome.

    thanks,
    Mark.
  6. #4
  7. Wacky hack
    Devshed Novice (500 - 999 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    513
    Rep Power
    14
    If you're going for a hand-rolled flatfile database, I'd look into XML. Whatever people say, it will force you to think carefully about the way in which you store the data, the fields you use, the relations you set-up, etc, and it will make fetching and storing the data easy using Python's XML support. XML is also well supported in almost all other relevant applications.

    What exactly are you coding for?
  8. #5
  9. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Hi telex, thanks for the input.

    Cant really see myself using XML for this but i'll give it a look.

    Ok well its a project i've set myself is.. I dont really know what to call it. For all intents and perposes it's a content managment system but i guess it could also be discribed as a templating engine too.

    The basic idea is to store page content and templates seperatly then compile and publish the files as plain html. By editing the stored content file you can update a page easily and punt it into a custom template before making it viewable from the web, allowing previewing and editing by anybody with a user account and from any web brower. Hopefully when a template is edited you will be able to recompile all the pages which use it making a global change across the website .

    Anyway thats the idea, whether or not its a good idea remains to be seen. What you think?

    Hope you found this interesting,

    Have fun,
    Mark.
  10. #6
  11. Wacky hack
    Devshed Novice (500 - 999 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    513
    Rep Power
    14
    That is quite an interesting idea

    So you can create a set of templates, and a set of pages whose content will then conform to those templates? Well here XML does sound like quite it could be a useful solution, depending on the complexity of the templates. If the end-user simply makes templates by putting preset blocks together (like header, left menu, right text block, etc) and then applying styles to them, then you can have one XML file which defines which blocks are to be used in particular template, what style is used, etc. In the other XML file you have the data associated with each block. You then merge the two together when the user compiles all the pages.

    The reason XML would be handy here is that you could abstract away from the mundane details of data storage, and quickly make data objects that hold the template and contents.

    Then again, if the end-user is going to have more control over the template, like directly editing the HTML, this solution could be quite complicated to implement, or quite complicated for the end user.
  12. #7
  13. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69

    Templates


    Thanks .

    XML is very intersting but I really dont know that much about how it works or what you can do with it, i've read a few articles but I wouldnt say i know enough to use it compitantly.

    The templates would be created in html and them uploaded to the server, aswell as html you'll be able to call prebuilt objects into the page using some kind of secial markup (syntax is undesided but im leaning towards ASP/JSP tag style, the content will be inserted into the template in place of one of these tags as will the title and a few other template spacific tags.) Of course if i deside to use XML then these prebuilt objects would be kept within XML but i'm leaning towards a custom aproch to storing the data in files simply for ease.

    I'd also like to use a custom structured text like interface for the content files but i would need html to still be usable.

    I dont know, this what you'd call complex? I'm thinking about setting up a source forge project, do you think this would be a good idea?

    Mark.
  14. #8
  15. Wacky hack
    Devshed Novice (500 - 999 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    513
    Rep Power
    14
    Hmm, OK well your idea is subtly different to mine, which was geared more towards an online content management system that didn't require knowledge of HTML. Yours seems more for experienced users who want more control.

    So... putting special markup in HTML code is no problem - I've done this in Perl before - just so long as you make some strict parsing rules so it is easy to detect the tags and insert the prebuilt objects without creating HTML errors. It might be an idea to check the validity of the HTML too... I'm not sure if there's a tool to do this, but if there isn't, just sending the output to http://validator.w3.org and scanning the results page for the "no errors" message would do it.

    As for the content, how you store it depends mostly on how it will be viewed and edited. Will they get an online form, so the storage format is hidden to them, or will they also open up the content files and upload them properly formatted? To make things easy for yourself, I'd suggest the latter. How you then store it is up to you... you could learn how to use XML.Parser or roll your own system. Either way you'll end up with a class with methods to access the data you're storing, and a well thought out file format that makes writing this class easy.

    Just for comparison, you might have this special tag in your HTML file:

    ----
    some text...
    <*1*>
    some more text...
    ----

    And then in your XML content file have:

    ----
    <object type="textblock">
    <id>1</id>
    <style>light blue</style>
    <contents>Wahey, my first textblock!</contents>
    </object>
    ----

    Or in your hand-rolled one, perhaps have:

    ----
    id: 1
    type: textblock
    style: light blue
    contents: wahey my first textblock!
    ----

    And then either way you'd have a class method that can read that bit of content from the file, and then associate that bit of content with the right template, like so:

    content = readContent(file="index")
    merge(content, "template1")

    Where readContent returns a list of the content bits, each object in the list being a dictionary containing the data for that bit. The merge method then takes each item in the list in turn, takes its id, finds the corresponding tag in the template and merged the content in with the appropriate style.

    So you see, XML or your own storage format, it makes little difference except for hand-rolling your own solution can be more fun, education, but bug prone.

    Am I making sense? Just thinking aloud here...

    As for making a Sourceforge project, again it depends on whether this is some code you're just going to use on one site, or whether it's an idea you want to code and let others use on their own web sites. If the latter, then there's no harm in setting up an SF project.
  16. #9
  17. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    ok, not that different. The idea came from.. (well a fall on my head while blading) I was asked to design a webpage for a furniture company, they wanted to be able to manage there own content and etc. obviously didnt know enough to handle the design and implementation of the, so i would design the template and the system, and they would update there own pages..

    I also liked the way Zope allows for different people to work on different parts of the site i.e. graphic desginers and db managers do two different things, but Zope let's them work together easily and eficently.

    One question though, I'm doing alot with directories and files, do you think this would be better done with perl? The code would of course be available for others to use, change and etc. so I may as well create the project when i have a working beta!

    Thanks for the great feedback telex You've given me allot of good advice and insite its very much apreciated. Sorry for the small responce but its getting kida late..

    Gotta agree with you, writing your own custom code is more fun.. and bug prone, but then why do soemthing if it's not fun? maybe thats the age talking

    Mark.
    Last edited by netytan; July 23rd, 2003 at 07:55 PM.
  18. #10
  19. Wacky hack
    Devshed Novice (500 - 999 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    513
    Rep Power
    14
    I'd do it in whichever language you're more familiar with. I doubt it'd be much easier in Perl, to be honest, though I've never tried anything like this in Python.

    The main thing is just to work out your data storage schema carefully in advance, write a few nice classes as the backend of your code so the rest is very modular and easy, and then it shouldn't be too hard. Just a quick example, you could make a class called Page, and then have methods to fill that page with contents (merging the files, adding content here and then), and then have a method to print that page to the browser. The worst thing is if you end up with lots of "print" lines outputting HTML yourself.
  20. #11
  21. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    I see what your saying, I'm probably going to write it in Python, its my fav language and I'm quite confident using it.

    The only thing I'm going to use print statments for is creating the interface (presently looking kinda nice). I will also have to think about how i choose to show the files, it has to be easy to use and navagate. The data storage is the most important part i agree but doesnt seem too hard. Once the frame work is finished then i think adding more functionality to it wont be a problem.

    For the actual pages (viewed from the web) i wont even touch, the program (dubed geko) will create pure html pages in the root directory of the web server.

    Looking forward to starting this now . Do like the names, definatly gonna use merge in there, such a nice word, but i wouldnt have thought of it. Thanks

    Take care,
    Mark.
  22. #12
  23. Wacky hack
    Devshed Novice (500 - 999 posts)

    Join Date
    Apr 2001
    Location
    London, England
    Posts
    513
    Rep Power
    14
    Well good luck with it, and do update this thread when you're getting somewhere, because it's an interesting idea.

IMN logo majestic logo threadwatch logo seochat tools logo