Major update for JowettTalk

[Forumadmin]

After many hours sorting various issues and anomalies I managed to reconstruct the album tree into a tree under the current forum called Library which is made up of 103 forum branches containing 551 topic twigs that will contain the 15679 item leaves. 325 empty albums will be deleted.
Next step is to build the topics and fill them with posts containing the items.
I expect much trial and error over the next few days.

[Forumadmin]

For the last couple of weeks I have been busy programming the process to move the Gallery items into posts. There were slight interludes for rugby and straightening bumpers with Bill Lock and visits to Keith Jones rebuilding his Bradford.
There were many issues including creating thumbnails for the pdf files, Gallery users not being in JowettTalk and timeouts because the transfer takes a long time. I had to modify the process so that it could be restarted at any point and pick up where it left off
The process is currently chugging away on the development system running under WAMP which is a basically a Unix lookalike webserver running under Windows. This server has 8 processors chugging away but is limited by disk speed in creating the 16000 thumbnails and posts.
When I am happy with the process I will try the process on the Ubuntu linux system that runs in a virtual machine that is more like the live server. This was used to test the new version of JowettTalk and it runs on a solid state disk so should be a lot faster doing the transfer. It maybe I can do the process directly on the new version which will save a few steps, but I doubt it.
A few of the items may not be transferred as they are in unusual formats and these will have to be handled manually. These include games, videos and sound tracks uploaded by some of the users.
To do is the updating of all the existing links in JowettTalk and some fettling of the Gallery to make it look more presentable, although most of that I will leave to do on the live system as I do not want Jowett Talk to be down for too long.

[Forumadmin]

With all the gallery, bar a few miscreants, moved into JowettTalk, the next step was to update all the links in the JT posts that referenced items in the Gallery.

This tasked really tested my patience because over the 12 years various methods for linking were used. Also some links went to derivatives of images in the Gallery so the the original image had to be found as that is what is now in JT.
There was a lot of trial and error but I think the majority have been converted- over 1000 posts were processed, some with 50 links in them. About 80 links will need to be checked out manually.

The program did find links that also did not work in the live system, so perhaps I can resolve those issues in the future.
I am trying to reduce the amount of manual work that has to be done as this will reduce the downtime of JT. I hope to get it down to only a couple of days downtime.

[Forumadmin]

I have now started to write the checking program that will tell me what has not been transferred correctly. It has found some issues so they will need to be corrected.
I am putting enough back into the posts to allow rebuilding the Gallery tree structure and to find, in future, references should someone find an old link into the Gallery. The JowettTalk search function should be able to find the relevant item referenced.

[Forumadmin]

One of the issues I am battling with is how to present the new library within the JowettTalk structure of forums, topics and posts . So if you have any suggestions I would be pleased to consider.
If I stick with the existing tree structure of the Gallery, items become attachments in posts which can either be standalone topics or posts can be combined within a topic, or indeed items could be combined within the same post.
Relating a story, such as a car rebuild, lends itself to the latter layout of combining within the same post. There is a necessary limit on the size of each post so about a dozen items could be sensibly appended to the same post with additional posts being added within the same topic to make up the whole story. (This is similar to the Gallery layout of 12 items per page).

I have decided to make albums which just contain items to become topics. Albums that contain albums become forums which thus forms the new tree structure. The only issue here is that some forums will then have items that need storing, so I decided to make a new topic within the forum with the same title as the forum in which to put these 'stand alone' items. The downside of this is that the items/albums may not appear in the same order as they did in the Gallery.

Provided your User Control Panel, Board preferences , Edit Display Options and set in 'Display Topic' and 'Display post' ' in 'post time' order 'ASCENDING' then things should remain in the desired order. BUT many prefer to have the time order DESCENDING so that the latest posts are displayed first. You can also set your UCB to display in 'subject' order which might help in the Personal Albums forum to find a member's personal album.

An issue with combining items in one post is that maintenance is higher if additions need to be made, so I decided to have each Gallery item to be contained within its own post initially. Curators can later combine posts into one if they desire to construct the story more compactly.

Gallery albums in some cases have been reorganised to display items in a specific order. This might be because say manuals are in car age order. Replicating that order in the new Library can be done my manipulating the last post time in the Topics and Posts database tables. This not only requires additional programming at transfer time but also subsequent maintenance. I could save the existing order in the subject of the post/topic by prefixing with a sequence number. The UCB can then be used to choose to order by 'subject'. Clearly this method has serious maintenance issues.
I am currently constructing a database table to capture the existing state of the Gallery so that , if required in future, some reorganisation of the topics and posts could be done. Initially though the new Library will not be perfectly matched to the old Gallery order.

[Forumadmin]

Today I completed the revamp of the build process which I decided to do because the previous process was difficult to restart if an error was found.
The comparison scripts that I wrote to check completeness and accuracy of transfer found many different types of issues which is not surprising with about 200 people adding and commenting on 16000 items in about 30 different formats.
For instance some data contained Finnish characters that caused an issue. Many people gave filenames that did not have the correct type such as -docx rather than .docx. Some people use filenames that were more like a paragraph of text. Although accepted by the Gallery they are not by the Forum software.
I have reduced issues now down to about 90 which will need manual attention and transfer. Not bad with 16000 items.
Today I also wrote the script to add the Gallery comments to the items (posts) in the new Jowett Archive. I also stress tested by doubling the size of the archive to make sure the system could cope with another 12 years of growth.
So I think the process is ready to test one final time before upgrading to the new software on the Test server.
There is still some work to be done checking access controls on the new system before transferring over to the live system.
I have yet to decide whether I do some reorganisation and housekeeping before the transfer or after it as there is some information which is out of date, superfluous or needs re-filing. There may also be things I would like to do to take advantage of the new facilities with the new software. Login will be the standard out of the box method as single-sign-on is no longer necessary. I hope not to have to modify the new software so that future upgrades will be a breeze.


Happy New Year.

[Forumadmin]

Hurray!

After 5 months of fairly intensive work the job is done. Well almost...
At least the site now has the new Library, the new software for the forum, and a slicker login and sign-on.
I have to test the system on the update of the underlying software which the hosting providers will shortly insist on doing. This platform is the Linux operating system, Apache web server, PHP program preprocessor and MYSQL database --each of which has been regularly updated over the 22 years the site has been in existence. On top of that was the phpbb Bulletin Board software and the Gallery 2 media software which also have a revision every few months. The latter however stopped being updated and would not work on the latest php version. For this platform upgrade to go ahead I need to check everything works on the new platform which I reproduce in a virtual server at home. I have already updated the Jowett Member Services to this platform.

This diagram shows the live system which is regularly backed up to my PC in the CYGWIN virtual server. The WAMP virtual server was used to install, develop and test the new system before uploading to a clean next generation system in the UBUNTU virtual server. Notepad++ and Dreamweaver are used to modify code. Eclipse Integrated development environment is used to test code against the next generation platform before loading it onto the UBUNTU test server.

As well as this testing, there may well be some issues on the existing platform that require my attention. I have already found a couple.

The Gallery2 software lasted 12 years which is pretty good, more than most other picture platforms. It was chosen because there was no support for images within the forum software at the time.

Some may think that Google and Facebook are the way to go. But I think using those facilities to store our knowledge of Jowetts is risky. I also think the conversations on Facebook and Twitter are of little use in building a repository of knowledge. As a referenceable resource I find Facebook of no use. Also once information gets into Google's clutches deciding what is authoritative and what is not, is difficult.

[Forumadmin]

Over the next couple of days I will be building and testing the new software platform that allegedly is faster. The upgrade will mean that hopefully another upgrade is not forced on the site for the next 10 years, although the versions of the platform and JowettTalk software will regularly be updated.

This change will move from version 5.6 to 7 of the php preprocessor. Jowett Talk uses phpbb which shortly will require version 7 and would not run on the old version of php. JowettGallery will not run on version 7 without a lot of modification so it is now no longer used.

Since I am attempting not to modify phpbb from the core set of software (with some additional supported extensions such as the language, imagemagick and topic sort enhancements), future migration to new versions should be less problematic. Although modification does create perhaps a better site, it means that maintenance is harder. Even supported extensions can get out of step with the core phpbb software so care is needed in selecting them and checks need to be made after a core upgrade.

I have already reprogrammed the rest of jowett.net (the member services) and jowett.org to use version 7, but will retest again on the upgraded test system.

[Forumadmin]

I am currently investigating an issue reported by Tony that has required going back to a database backup done on Friday. I am now trying to restore posts made since then.

I restored all of yours , I hope, but not all the ones I did over the last two days. It appears the database suffered a corruption, perhaps due to the reorganisation I did.
The test server was OK so I was able to restore the database from there.

Although I did the major reorganisation on the test server I did some more minor ones on the live server moving topics around, which should have been fine, but I think sometimes a transaction gets interrupted because it is on a shared server. In future I will do this on the test server which will not suffer those problems.

Please let me know if you find ANYTHING WRONG.

Thanks again to Tony Fearn who informed me before too much was lost.

[Forumadmin]

Some more testing has shown that when uploading an attachment the image that is displayed is that of the previously uploaded attachment. If you right click on the image and then paste that link within an image BBCODE then you get the correct image displayed. Alternatively paste the link into the browser address and you will also get the correct image.

So I need to look at the module that displays the page....tomorrow. This is why I do not like using the latest software or software with mods.!!!!
The problem seems to be caused by the post display routine using an old thumbnail rather than the one associated with the attachment. If you go and edit the post again the correct thumbnail is displayed!

Has anyone else experienced this issue?

[Forumadmin]

But this time not with JowettTalk. Much time spent yesterday and today with the hosting and network support as I could not use the remote access system and I hit a file limit issue with the 30000 files in one folder.
Neither of these issues is solved but a laborious workaround has been found.
I suspect some of the issues on Jowett Talk are caused by the old platform software so will press ahead with the upgrade.
I also hope that time may heal the remote access issue.

[Forumadmin]

The Live platform has now been updated to the latest php version up from the very old 5.4 to version 7.2

Hopefully now the system will be more responsive and resilient.

I have also found the gremlin that crept in with an extension called imagemagick that was added to make picture inclusion better . I have disabled it as it caused issues displaying thumbnails of newly added attachments. I will review on the test system to see if I can fix it and then enable again.

All this was after a very frustrating 3 days battling with two support organisations trying to fix an issue with me getting into the server. I eventually found that because I was doing so much work on the server on Friday my IP address had been blacklisted. I was able to get in with my phone that used a different IP and with the cable connection when I disabled the firewall on the router. I eventually found out that doing the latter also gave a different IP address which allowed me to get in that way. But having initially blamed the issue on a software mod Virgin had made to the Firewall I had to eat my words and blame the hosting company for blacklisting my connection. It took a lot of debug logs to prove to them they were at fault. Never mind I learnt a lot more about my cable connection and secure connections.

[Forumadmin]

As you hopefully have noticed a lot has moved around within JowettTalk with the hope that things become more accessible.
The main areas are the consolidation of topics :

1. under the subforum 'The clubs and what is on? ' where each club has an 'Events' subforum, I have pulled sub-forums and topics from the Public Albums and other areas together. Some more local events I have put under the relevant SECTION subforum.

2. under the subforum 'Technical. Recipients should check with relevant experts before relying on information provided here' where each subforum has a 'Best Bits ' subforum where the major projects are preserved for posterity. Hopefully those visiting the site can gain the most useful information more quickly.

3. under the subforum 'Library' any useful stuff in my own and others' Personal and Public Albums has been moved into the relevant section. Some that was considered of sensitivity public and not just for club members has been moved to the 'Technical' or other subforum. The Library is really only for club members. My intention is to empty the 'Public Albums' subforum and to encourage others to put their personal contributions into a more relevant place by asking me to move the topics.

There is still a lot to do tidying up various subjects. There is also the rather more complex reorganisation of the Legacy sections. I have not fully thought through how to go about this. Contributions have come in over 20 years so many have been superseded by better items and so can be archived or deleted. Some exist in various formats such as pdf, tiff or html-- so do we keep them all or just chose one. There has already been much discussion on this but another review would still be valuable. Some items are missing so we need a concerted effort to find and scan them in.

[Keith Clements]

I think I have completed manually moving and reformatting all that had collected in the old jowettnet between 1999 and 2006. I found it nostalgic to review the contributions made by Jowett people from around the world. It emphasised to me how important JowettTalk has been bringing us together. It is great to be able to go to local and perhaps the occasional national meeting to talk to like minded people, but JowettTalk has brought together everyone from the far flung corners of the world. Of the 33 people who had contribution pages, 23 were not from the UK. You may ponder why that is, but I am convinced that those in the UK can satisfy their Jowett sociability more easily and traditionally, whereas those elsewhere are so spread out that the Internet provides a means of bringing them together. This international sociability continues, even after 20 years and the advent of other social networking tools.

What is left is the old jowetttalk as I cannot decide how to move this in or even whether it is worth the effort.

Well here is the link if you have any suggestions.
http://jowett.org/oldjowetttalk/

There are 903 postings in there from November 2001 till February 2006. The postings can be searched using the jowett.org search page accessed by the link on that page .

[Forumadmin]

These were in Jowett Internet Services and not JowettTalk and were related to the upgrade of php from 5.4 to 7.2. The Integrated Development Environment did not highlight them and I was not diligent enough to do all the testing and look at all the error logs. I do not know if the issues were visible to the user, but they are now all fixed I hope. 7.2 is much stricter so did not like some of the old code.

I have also updated the JIS help file and added a screenshot tutorial for login, just in case it is required.

I also checked that links to pictures in the new JowettTalk could be added to the vehicle register. I do need to make the vehicle page more user friendly though, but today was sunny and the daffodils are starting to open so I spent the afternoon clearing up some beds in the garden.

The late Harry Brierley did a blog in the Jowetteer on the Life and Times of his car OTB. Why not do one on yours and link it in with the Register? Just start a topic either in your personal album or in 'My Jowett Experiences' forum. Put the link to the JowettTalk topic in to the Pictures field in the Vehicle Register.